At around 2:00 AM on July 31 our Cloud provider, Amazon AWS, had trouble with three of the services that we use (SQS, SNS, and SES) simultaneously. The outage lasted until about 3:40 AM. This impacted about 50 transactions. We have been able to recover…
We have been asked by a customer to document our disaster recovery plan, and we figured that it might be useful to document this for other customers as well as provide an example to other websites who might want to use a…
In 2014 we had outages totaling 16-21 minutes (depending on how you count an outage) for an availability of 99.996%. This is down from 99.998% in 2013. As usual, we do a year end (well, almost – there are still 7 days…
Our site was down from 6:31 PM Eastern for about 8 minutes. This was due to a deployment we did, which ironically enough was to help prevent issues if the Amazon Content Delivery Network was unavailable, we would be able to quickly…
We had a 5 minute outage early this morning for 5 minutes at 12:53AM Eastern time. We were doing a deployment and it triggered errors. That error was corrected as quickly as possible. We apologize for the outage. This deployment we did…
We had a partial problem that affected about 1/8 of our users on Thursday at 3:24PM until 3:28 PM. It disrupted one person who was in the process of signing up for a race – meaning they were redirected back to the…
As always, we report any availability issues in the spirit of transparency. We did a database migration early this morning to better support International customers. The migration took 3 minutes and 29 seconds to complete. During this time, the site functioned fully with two…
We had about 2 minutes at 7:25 AM Eastern this morning where about half of requests were not being serviced properly. No transactions were lost, but users would have had to click refresh on their browsers. There was an unusual bug on…
Our 2013 Availability was 99.9981% for the year. We screwed up twice, and had one planned outage for a total “downtime” of less than 7 minutes for the year and partial registration impact affecting about 15% of customers for about 115 minutes….
One of the Amazon servers went into a hang mode yesterday (always on a Sunday ;-). As we have written before, all of our servers are set up in a redundant fashion, but there are always “gotchas” that you can only learn…