Unfortunately, our payment processor, Adyen, had an outage with processing VISA and Mastercard credit cards (American Express and Discover continued to work) this morning 10//31/2023 from 10:10:00 AM until 10:35:51 AM. It is now resolved.
Adyen provided their RCA (Root Cause Analysis) to us. It was a misconfiguration of a new feature that they are in the process of testing. It is kind of interesting that the root cause was trying to add a feature that would eliminate issues like this in the future where their is automatic failover between regions for VISA and MasterCard transactions. Which is good news for the future.
We did have a message on all event websites about the problem as it was happening to try to reduce customer frustration.
From Midnight until 10:51 AM we have processed 5,984 transactions successfully. It appears there were several hundred customers trying to sign up for events that were frustrated and tried multiple times to process their credit cards unsuccessfully. This is frustrating, and we will report on Adyen’s final report of the problem.
One of the advantages of using Adyen is that they process about 20 Billion transactions per year and they have duplicate infrastructure in the US and Europe and our systems are set up to failover, which they have in the past successfully. Unfortunately, this outage was both on the US and European infrastructure.