Over the past two weeks our Payment Gateway provider, Braintree, has had a couple of outages. This impacted some users who were trying to pay.
The first outage was on November 6 from 4:45-5:02 PM Eastern time. This outage was caused by network maintenance that impacted their database cluster. During this time, no registrations were able to be processed.
The second outage was on November 17 from 11:15-11:51 AM Eastern. This was a less troublesome outage which delayed processing for people registering on RunSignup. The larger issue was 13 duplicated transactions. The good news is we were able to find those and process Voids on those transactions that afternoon, so there was no financial impact for customers.
We are working with Braintree on creating a backup if errors like this occur again. We are investigating several solutions, including queuing transactions on our infrastructure if Braintree is not processing as well as having a failover mechanism that would use either a secondary Braintree environment or a different party altogether.
As we have said in previous posts, no one has a truly 100% uptime environment. But we strive to be as available as possible. So far this year we have had one incident int he first half of 2013 that impacted 5 registrants. In the second half so far we have had these two incidents with Braintree and the approximately 5 minutes we were down this fall for the big Financial Upgrade where we had too many software changes to make them live as we usually do.
Overall, less than one hour this year so far, which translates into 99.987% uptime.