Unfortunately, our 5 year, 9 month and 1 day of 100% uptime with zero time down even for maintenance of the GiveSignup | RunSignup system came to an end at 4:16 Eastern time on June 28, 2021. It lasted for 4 minutes before we were fully operational again. The last time the system was unavailable was October 27, 2015.
As reviewed in each year’s availability report, we spend a lot of time and money and talent to try to provide a continuously updated and available system for our customers. But we are only human and make errors occasionally. We have deployed over 10,000 releases of the system – improvements for our customers. Yesterday in one of those deployments, it impacted our Session Servers. In essence the system logged people out and customers were not able to reach their dashboards and participants were not able to signup for events. We quickly realized the error and reverted the change. That combination took 4 minutes before the system was back and functioning again.
From the details of the number of transactions (people paying with their credit card on the checkout page) happening per minute, it seems the couple of minutes after the outage, those customers came back to complete their transactions as there is a bit of a peak for a few minutes after 4:21. So it looks like we did not have a material impact on anyone’s fundraising or business.
We of course apologize for this and will do our best to start a new record over the next 6 years we will need to keep clean to match the previous 5 years and 9 months. But we are a bunch of humans here at our company and occasionally we make mistakes. But hopefully we learn from them and will come up with ways to insure this one does not have a repeat.