We had registration delays that occurred today from 3:52PM – 4:32PM. All registrations, ticket purchases and membership purchases during that period were processed at 4:32, with the registration timestamp occurring when the person registered.
Users saw a spinner on the final confirmation page that let them know we had their registration and not to signup again.
This could have a been a bit confusing for some customers, and we had several that did register again. We have cleaned those up.
The cause of the delay was a database migration on the waiver tables that was done to prepare for some future growth and features. There was an error in the database migration. That has also been resolved.
All transactions completed because of how our system makes use of queues for processing. So all transactions are held if there is something holding completion of the transactions up, like completing the waiver saving. This is part of our high availability infrastructure setup.
So the bad news is we had some customers see a spinner on the checkout page. The good news is all transactions completed, and our high availability infrastructure proved to be a solid investment. It allows us to make 10-20 updates of our software a day.
The other good news is that we have a great technical team that hopped on a Google Meet and coordinated our response and solution. Thank you to Stephen, Ryan, Jeff and Matt!
Also, for the record, we have had 18,000 transactions totaling $1.5 Million as of 5PM today. We had 1,074 transactions during the 40 minutes totaling $98,172.