Our guest blogger today is Femi Taiwo, lead developer for the OpenVR project. The OpenVR Application was used to register more than 70 million voters in the just concluded general elections in Nigeria. Read on as Femi takes us behind the scenes of an application that has possibly changed the face of elections in Nigeria for ever.
OpenVR – short for Open Voters Registration – is a free and completely open-source biometrics capture solution built on open standards.
OpenVR is the software that was used to register over 73million people during the voters registration exercise held earlier this year in Nigeria. And it was BUILT (from scratch) BY NIGERIANS – all members of the team are volunteers – software engineers born, bred, trained in Nigeria.
Nyimbi ODERO – System Architect
Ronke DOMINGO – Project Manager
Femi TAIWO – Lead Software Developer
Sola AJAYI – Software Developer
Fisayo ORIMOLOYE – Software Developer
Doyin KASSEM – UI Designer
CHUKWUEMEKA Ujam – Biometrics Expert
To its credit, OpenVR can boast of
– Aggregating over 85 Tera-bytes of Data
– Printing 73M+ Voter’s Cards
– Generating over 4,600,000 Pages of Reports in PDF
– Collecting over 500,000,000 Fingerprints
– Installation on over 132,000 machines and hundreds of servers
– An effective one-click patching system
– Full Integrated AFIS
Built from ground up to be very secure and unobtrusive at the same time, OpenVR software runs on Ubuntu – a Linux Operating System and supports a variety of fingerprint scanners, external hard drives, printers and web-cameras. It also features an in-built Automatic Fingerprint Identification System (AFIS) to reject multiple registration attempts. It comes as a bundled solution that works right out of the box, without any configuration required on most hardware.
We created it as a one-click-to-install solution – it automatically installs and configures the video drivers, sound drivers, network and wireless card drivers, scanners, printers and other devices. We chose Ubuntu for several reasons including – ease of customization, rigorous quality assurance, hardware compatibility and a renowned security track record.
So, why did INEC choose to go Open-source on a very important, critical and failure-intolerant project? And why did we volunteer to build it?
For me to answer the first question, I’d like to explain what open-source really means.
Free Software and Open-Source Software
These terms are easily misunderstood.
Open source describes practices in production and development that promote access to the end product’s source materials – in this case – the sourcecode for OpenVR. Phrased from the free sharing of technological information, Open source software is software whose source code is published and made available to the public, enabling anyone to copy, modify and redistribute the source code without paying royalties or fees. Open source code evolves through community cooperation. These communities are composed of individual programmers as well as very large companies. The Linux operating system is a great example of opensource software – and in this case, it’s the complete operating system that is opensource.
Free software is a matter of the users’ freedom to run, copy, distribute, study, change and improve the software. More precisely, it means that the program’s users have the four essential freedoms:
The freedom to run the program, for any purpose.
The freedom to study how the program works, and change it to make it do what you wish. Access to the source code is a precondition for this.
The freedom to redistribute copies so you can help your neighbor.
The freedom to distribute copies of your modified versions to others. By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.
This essentially means that anyone with the publicly available source-code of the OpenVR software can make modifications to the software, and release such changes, without payment to the developers of the original version.
Essentially, for INEC, this meant complete freedom from proprietary software and vendor lock-ins,It which is something that has left many institutions and other government bodies with a bitter after-taste in their mouth. It also meant that there were no hidden backdoors, no quick hacks “just make it work mentality” and that they could get independent resources at any point in time to run a peer-review of the entire software – architecture, source code and implementation. Now that’s complete control.
At the onset of this project in August (did I already mention it took under 3 months from start to finish?), there were 9 core questions
1. Will it work?
2. Will it be ready in time?
3. Will it be secure?
4. Will it be easy to use?
5. Will it be fast?
6. Will it be stable?
7. Will it be scalable?
8. Will it be re-usable?
9. Will it be extend-able?
Will it work? Yes!
By using Linux, and a baseline API for accessing all peripherals we were able to support 8 different types of fingerprint scanners, any video camera supported by Linux (for the Photo capture), and over 3500 printers (we had at least 2000 HP printer drivers installed!)
Will it be ready in time?
Good programmers thrive when working with impossible deadlines. We had a ‘healthy’ set of milestones and an incredible amount of pressure, not to mention the negative articles in the media.
Nights turned into days and the only other times we were not in front of our laptops or white-board, we were crashed out. We drew up feature quadrant – Must Have, Should Have, Could Have, Good to Have. With lots of feedback, criticism and insight from people who were around during the previous exercises we got a pretty clear idea of where each ‘brilliant’ feature went and the direction.
Will it be secure?
128 Bit Encryption. Built with fraud in mind. Used a white-list approach. Restrictive, but works like magic. Every bit (or byte if you choose) of data was encrypted – inside and outside the machine. Since there was no possibility of network coverage, the only other option was to use a removable drive to backup and transfer data – using the encrypted file-system support in the Linux kernel to protect the data and additional hashing algorithms to detect any unauthorized modifications during transfer.
Will it be easy to use?
After one of our demonstrations, a security guard sat in front of the machine and registered himself without supervision. In programmers lingo, there’s this thing called KISS. Keep It Short and Simple (there are other less polite versions). The KISS principle states that simplicity should be a key goal in design, and that unnecessary complexity should be avoided.
Effectively and intensely training 240,000 operators to learn how to use a software within a short space of time is a really difficult task.
With this understanding and knowing that the operators would have varying degrees of computer-literacy, anything that was NOT downright obvious and intuitive was considered unusable.
Will it be fast?
Our target was a maximum of 8 minutes per individual. We ended up with 4minutes (less really). A lot of things were sent to the background and managed to pull 1260 comparisons per second.
Will it be stable?
Linux prizes itself in its stability. Even when operated under ‘hectic’ conditions for extended periods of time. We had our fare share of untrappable interrupts from hardware devices which kept crashing the software. There was also the expected hardware faults. Crashed hard drives, faulty backup removables, motherboards burnouts, broken screens, faulty keyboard, corrupt boot sectors – a mile-long list of things that could possibly happen to render a system unusable after it’s been on the field and been used to capture a few hundred people.
We had a two-pronged approach. First ensure the system itself is as stable as possible – a barrage of tests of extended use with a large amount of data. Two, in the event of a failure, use the Swiss Army Knife – the OpenVR Recovery DVD. We created a DVD containing tools specifically customized for
a purpose – recovering data and restoring broken machines without data loss. The ability to recover data when everything else has gone wrong – we think this was a critical singular component and success factor in the OpenVR deployment.
Will it be scalable?
What happens when you have 10 people registered? Or 250 or 500 or 2000? Will it slow down at all? Or slow down considerably or crash outright? Will it make use of the multitasking/multi-threading capabilities of the underlying hardware? Will it handle growing amounts of work and features in a graceful manner or not?
First we identified the bottlenecks in the system – Fingerprint – AFIS. When a fingerprint is captured, we need to compare that finger against the existing database of fingerprints to check for multiple registration. The larger the database, the longer that process will take. Initially we had a sequential process – the search ran in a straight sequence – like going through (potential) records from id 1 to id 300. Instead we break up the records into sets, with each set containing a number of records – 30 for example and run a threaded process against each set. The size of each set and comparison is determined by the underlying hardware – processor, memory (free memory) and very importantly – the true multitasking features of the Linux operating system.
Will it be reusable?
It needed to be – for continuous registration and for similar implementations.
Will it be extend-able?
We needed to be able to make changes to the software – out there on the field- on 120,000 machines. And we needed to update the machine, change the way it operated, fix bugs with minimal user interaction and without data loss. If we lived in a connected environment – Internet, that would have been a walk in a park.
However, there’s no Internet. No network coverage. The GSM networks effectively cover less than 60% of the country, not to talk about how much of that amount can carry data consistently. Beyond software updates, we also needed to transfer all registration data from each machine down to the central database at the state level. Therefore using a network would have been prohibitively expensive. So we made the entire software – including the operating system itself PATCHABLE. We used Sneaker-net – the transfer of electronic information, especially computer files, by physically couriering removable media such as magnetic tape, floppy disks, compact discs, USB flash drives, or external hard drives from one computer to another. We created patches (encrypted archives as well) that work just like a virus – except that it needed users to click the Update OpenVR. This came in especially handy after the first four days of registration that was the most trying period in the entire exercise. Nothing worked as expected.
What helped, what worked and what didn’t
Immense feedback and ideas from the ICT department in the commission, and technical advisory from IFES (International Foundation for Electoral Systems), and UNDP (United Nations Development Programme). This helped in clearing up the gray areas and think through the best approach in each scenario.
The most used phrase? : It’s not working.
Anticipate problems. Because they will come. Every time we were reminded of Murphy’s law: “Anything that can go wrong will go wrong”.
If you perceive that there are four possible ways in which something can go wrong, and circumvent these, then a fifth way, unprepared for, will promptly develop.
All of 3 months – that’s all it took for 3 young and determined programmers, 1 brilliant architect, 1 persistent project manager, 1 intuitive designer and 1 knowledgeable biometric expert to build the entire OpenVR from scratch and deploy for the voters registration exercise. And we did it for free.
This is the biggest Open source effort (and Linux Based effort too) yet to come out of Nigeria – and Africa as a whole. Disruptive technology. That’s what this is all about.
We strongly desire the growth of the software industry here in Nigeria and Africa. It is our passion and love for our country that was the fuel that drove us to succeed and even exceed expectations in many aspects.
Partner at INITS Limited, a firm specialized in creating custom desktop solutions and web apps.
Ardent C++ and PHP Programmer.
Over 9 years of experience creating applications.
Heavy Linux user.