Thus Spaketh Idd Salim

Mpesa downtimes – Safaricom NOT to blame

by on Apr.19, 2010, under Personal, Symbiotic, Zunguka

Mpesa

A troubled but genius innivation

Last week, I called my landlord to ask her why there was no water in my apartment. She told me,  “pigia city council uwaulize. I just connect you to water. Not provide it.” So I packed my stuff and moved to the leafy suburbs where the taps never run dry.

Also, Last week [what an eventful week. Arsenal match included], I Mathematically demonstrated that it was IMPOSSIBLE for Mpesa to go down due to user-load. But, the downtime issue occured again last week!  This downtime lasted so long that I, for just some seconds, assumed that the eye of the Nebula had finally opened and the finger of god was about to start poking us all. I could not send money home and I had to cancel my Friday night Pool Hustling to take the money back home, by hand. Yuck! 2002 all over again.

But now, Mpesa is back up. We are all smiling. Long live Safaricom. Until the next downtime. They we can all switch back to Safaricom-ni-madogi mode.

After a response by Kaduki and a blog posting by Kachwanya (both very learned, incisive and non-partisan friends of mine and former Stacherians) about a non-safaricom-controllable element of the downtime, I decided to do my research and what I found out was interesting.

The Mpesa Architecture

Minimalistic Mpesa WAN Overview

Note/Disclaimer : The map above is my own sketch of how the Mpesa system would hypothetically work. It is by no way endorsed by Vodafone or Safcom. Ok.. Safcom wouldn’t -ofcourse; So let me stop at Vodafone.

From the WAN-map above, we see that Mpesa has 3 primary points-of-failure.

Point 1 : Data Path-1-to-2 Request Path

If the link between Saf and Voda fails (cut, rained on, power issues of just the plain fear of Makmende), your Mpesa will fail. Shared responsibility – Saf-Voda

Point 2 : Data Path-2-to-3 Auth path

If for some reason Voda does not get a full hand-shake and ACK from the bank, then your Mpesa will fails. Shared responsibility – Bank-Voda

Point 2 : Data Path-3-to-2-to-1 Response path

If for some reason Saf does not get a full and timely response from Voda, then your Mpesa will fails. Shared responsibility – Saf-Voda

So, clearly, Safaricom might, and I suspect, always does her part.. and VERY fast at that, but the multiple-points-of-failure make them look bad in the eyes of the public.

Solution

Many come to mind:

  1. Develop a Kenyan Mpesa. Locally hosted and run. No downtime.
  2. Take and work on daily data snapshots. Reconcile with Voda at end of day ala the ATM Model.
  3. Work on a Store-and-forward modus operandi where there is a system-trust threshold based on the last-known-user-balance so that the client ALWAYS gets served and reconciliation is delayed abit. This could also be made more secure by placing repeat requests by this client on queue-2 is reconciliaton is not yet down.

That’s all, folks!

Back to code.

:, ,

  • http://johnkaranja.com John Karanja

    I think links 1 2 and 3 should be redundant and use a Virtual Private Network/Extranet.

    I manage servers for a daily living and this is not rocket science.

    You should see my Mpesa hack for my payment API.

  • http://www.facebook.com/norman.ondego Norman Ondego

    I hate the safaricom-ni-madogi mode am not sure if they would accept your proposals but tis worth a try

  • iddsalim

    That is why Mpesa is just like my girlfriend. When she is online, it don't get no better… but when she decides to be a bitch… Murder comes first to my mind…

    Yes, we should see the API. Any links?

  • iddsalim

    Wazi bro normz… Safaricom never accepts any kenyan IT proposals. Will not even bother trying…

  • http://wapichapaa.blogspot.com edwinabuga

    Very insightful, as always!

  • http://wapichapaa.blogspot.com edwinabuga

    When you say “Develop a kenyan mpesa, do you mean mpesa is not kenyan?!

  • iddsalim

    Mpesa – the name = Kenyan
    Mpesa – the idea = Kenyan
    Mpesa – the rest – not Kenyan…

    no wonder the failings…

  • eebrah

    Wazi bro, explaining this MPESA sh*t to me, To the best of my understanding, such a system can be implemented on a carrier independent manner( ala Mauritius, Maldives ). In the Maldives it is run as a Parastatal type deal where anyone on any network can have a mobile bank account and all the banks in the country are connected to the system.

    Why we are still sticking to such a flaky, carrier dependant system with the banks refusing to link to( #wankers! ), I could never understand, but there are alternatives and as Kenyans we have refused to vote with our feet/wallets, we really are sheep!

    Now to shut myeyes and dream that them Brown( Bharti & Essar ) people will do the right thing and finally show up this dogi we call safcom.

  • http://www.gmeltdown.com gmeltdown

    Good answer indeed. However you might need to clarify that not all idea owners get to implement them – leave alone make economic gains out of them. Consider Mpesa – the profits = MJ and Vodaphone (Government and Kenyans combined own less than 50% of Safaricom)

  • john john

    Of course the above diagram is just a load of cr@p. I used to be in the MPESA development team and you have no idea how much more complex MPESA is. There a lot of other external components contributing to downtimes like for example Safaricom’s SMSC

    Point 1 is void because the link is over satellite, which has its latency but also minimum rate of failures.
    Point 2 is void because the bank only comes into play when agents/stores, and not end customers like yourself, need to withdraw/deposit cash.
    Point 3 (which to you is also Point 2 somehow) is also void because, as mentioned before, your ‘sketch’ does not in the least bit relate to reality.

    A bit of background on MPESA. The architecture in place now is actually a prototype that was deployed to production as another competitor was trying to steal the idea and land in the market first. A new version of the system will be deployed very soon which purportedly focuses on performance issues.

    http://en.wikipedia.org/wiki/M-Pesa

    Having your own blog does not mean you can push utter nonsense down people’s throats.

    What’s FORCING you to use it, I can only wonder!

  • Anonymous

    Damn.. I must REALLY have annoyed you. My apologies.

    if you worked at saf/mpesa as someone in a technical capacity (not a sweeper or a cook as your rage suggests), then you would know a thing or 2 about information disclosure ethics. Google that. We *dont* share *everything*.

  • http://www.gmeltdown.com/ gmeltdown

    Hi there John John – thanks for your version of the story. Any explanation as to why the service has to be hosted outside Kenya(via your point 1 satelite) whereas 99% of its users are in Kenya. Insufficient local capacity?

    Perhaps you are right Kenyans are not supposed to make direct economic gains from their own peculiar habits (embracing M-PESA).

  • Anonymous

    GM, this is a classic case of a ‘hormonal response’.

    This post is pro-mpesa, but johnny here is all rubbery and has taken it as ‘another attack to our beloved mpesa’…

  • Mynewemailacct2009

    Bless you all, thanks for taking an interest, mpesa keeps me awake at night, looking after 14 million peoples money tends to do that. It can literally be a life and death decision if mpesa fails.
    mpesa has some of the brightest and most experienced minds in the industry working on it, combined experience of thousands of years. Working for life critical systems.
    We care a huge huge HUGE amount for the Kenyan people, you just wouldnt believe how much.

  • Man behind the scene

    The system was put in place and developed and expanded upon as the ideas increased in terms of where it could help the people of Kenya. Taking an application and then stretching it beyond it’s initial functionality is not easy when it’s online. Demand could not be anticipated at any point in time – all projections provided were exceeded. The development and operational team were extremely dedicated. I as the Operations Manager for M-PESA spent many nights looking after the system. The capacity utilisation and performance had to be matched with demand. The system had to be expanded rapidly – it was the first of its kind and there was no benchmark available.

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Kenyan Blogs Webring Member

Possibly related


Blogroll

A few highly recommended websites...

Archives

All entries, chronologically...