Explaining the IEBC misnomer to my cat and a 2yr old

Mar 17, 2013 34 Comments by

A system let down by humans

IEBC this. IEBC that. let us put some things into perspective.

In my article in the #140Friday blog, I talked about a few issues in the government systems. I have delayed this blog post to today, from the promised 7th of March, after I was released from prison in the 5th of March.

I have to reiterate that I was not jailed for ANY connections or suspected connections with any IT crimes. I am too sleek for that.

The delay was due to the fact that alot of cooks and watchmen, sensing the public need for an explanation, had started filling the blogosphere with articles and “technical explanations” as to what happened, to get reads and some ad-clicks.

For the patient one, here is an article (from the master) explaining the following:

  • What happened.
  • How it happened.
  • How do we make sure it NEVER happens again.

“What happened, Salim?” You ask!

There are 2 theories.

One talking about SQL Injection in the server, where un-sanitized data entered the Database, thus multiplying the rejected votes by 8. A possibility. IEBC had hired at least 3 of the sickest hackers I know in Kenya to monitor the system. They are masters in the Network and Transport level security. I cannot think of one hardened application-level and server-level hacker/admin who was hired.

How it happened.

Possible fail-points

Server Specs

The other one was more credible. Talked about what I thought the problem was. I will address this in more detail. But let us get some perspective.

  1. The IEBC hardware arsenal had 339 servers.
  2. They had 33, 000 mobile phones.
  3. There were 26, 000 users logged on (Laptops, mobile etc)
  4. The server has over 20 CPUS, had 128 GB Ram and 18 TB Hard Disk space.

Disk space

With a voting population of just 12 Million people, if EVERYONE had voted, the approximate mathematics would be like this:

  • Data for the 6 candidates (Names, Designation (President, Governor etc), Voter details) per voter would not possibly exceed 240 bytes.
  • 12 Million people (if everyone voted) would generate uncompressed plain-text data of 2.682209015 GB
  • This data would just fill 0.000145519 % of the hard-disks that IEBC had.
  • If EIBC was to get fancier and SCAN all the voter documents for digital storage at at least 600DPI, the 12, 000, 000 voters’ documents would eat up an extra 20.6 TB of space, since the average size of each PNG would be 300KB. 2 servers would comfortably handle this data.

Network IO

Safaricom guaranteed 99.9% uptime but recorded  100% uptime in all VPN links. The average data per transmission could not possibly exceed 240 bytes, so with the 3, 000 MBPS pipes that safaricom had, the MOST they recorded from IEBC was 3.5 MBPS. A 0.001166667% load. So there was NO congestion.

So, hardware was not an issue. Space was not the issue. 3 of these super-servers could comfortable run an election of 20M people. Transmission was NOT the issue. Safaricom gave a 10-lane high-way, but only 3 bicycles passed.

With the above, I will, therefore, stand infront of intelligent men and women and say that the IEBC failure should NOT condemn the Kenyan tech scene. What we witnessed is a clear case of a boy being given a man’s job.

IEBC opted to use MySQL, a database that has been tried and tested. But this proved to be the Achilles heel. The database can be configured to create log files (a report of everything that is happening). There are various log levels, the noisiest of which is the “debug level”. This means that once you create a database setup whose log-level is set to “debug”, the directory/folder you have specified as the “location” of the log files has the possibility of getting filled up quickly.

The IEBC team setup the MySQL server to store logs in a special folder called “/var” in Linux. Loosely think of this as creating a drive “v:\” in windows. Now, the knowledge that you will generate a possible MAX of 2 GB of data from the entire country, would have told any sysadmin worth his weight in salt, that the “/var” needed at least 20 GB of space (approximates). IEBC allocated LESS space. Tones of space remains idle on the server disks.

So the “/var” was full. A new record would come in to the Db, the DB tries to log it, chokes and does something we call a “server panic”. Crashes. Record lost. Restarts. Record comes in. Maybe 2000 records per second. MySQL Crashes. And the cycle continues.

Confidence is lost. People panic! All because the “living room” of the server could ONLY accommodate a stool. While the other rooms, where no living furniture is kept, have LOTS of space.

Think of it as going out with KSHS 10k, and you get a bill of KSHS 12k. And you have no ATM, Mpesa or someone to call. You chonga-viazi, painfully knowing that you have 2M in your mattress at home. Sad.

Way forward?

It is very simple. At least to me. First IEBC should sell the 336 servers. They are a waste of space and power. We just need 3. Better still, donate one server per county for a national-wide super-computer grid that I will happily setup. Sweet. IEBC will still have 46 servers to operate on.

The real solution is the same solution I always propose to other problems. Acceptance. If the IEBC Sysadmin had accepted the fact that they have NEVER configured a MySQL server to server a system as big as this one, we would have had a president by now. No doubt would have been created. Just because you have driven a Vitz, don’t assume you can drive a 18-wheeler. Database si database!

IEBC should hire the seasoned minds. Most of who, sadly, would NEVER send their CVs to IEBC. it is upto them to head-hunt. I could recommend a few.

CREDITS

140 Friday Article

IEBC Tech Blog

IEBC tech blog article on failure

Close Sources**

Back to code.

Wazi

 

Personal, Xema Labs

About the author

Coder, hacker, inventor, pool guru.

Switch to our mobile site