We are in the midst of doing a major database upgrade from Amazon Aurora V2 to V3 as part of our recurring infrastructure maintenance and improvements. This is a fairly major upgrade as it also includes an upgrade from MySQL 5.7 to 8.0. We also have 3 major databases – our Main, our Shard and our Analytics database. Each database also has a Read Replica for scalability and high availability. The Main database has over 1,700 database tables and over 4 Billion rows of data. You can read more about our infrastructure here.
We have been doing pre-work over the past couple of weeks doing things like upgrading all of our temporal tables in preparation for new features in MySQL 8. We have upgraded the database used by our 9 test servers. We discovered some small issues and had to write some error handling code during that exercise.
Yesterday we upgraded the Analytics database with Aurora’s Blue-Green upgrade mechanism. This afternoon we did an upgrade of the Shard at 2:02:09 PM Eastern. Both of these upgrades too less than 40 seconds. Most interactions were unaffected. Some page loads that involved making a query to the database (most do not since we cache data in our architecture) took 40-45 seconds, but those were in the vast minority.
This graph shows the average response time went up to a bit under 200 milliseconds (0.2 seconds), about the time it takes for the human brain to recognize a face.
However a few people did see more lengthy delays in response time (60 people over 5 seconds and 99 total over 2 seconds):
Next week we will upgrade the main database and hope for similar results. We are likely to put a notice out beforehand since that will delay transactions and there is a bit more risk.
12/17/2024 Update: The Main database update was also very successful. We did the upgrade between 9:31-9:33 AM today. There were no lost transactions (transactions actually completed every minute). There was only minor slowness on the website as shown my a similar set of numbers from the earlier upgrade:
Summary
We strive to provide our customers with a stable platform for their events. This major upgrade will allow us to continue to provide a stable environment. The upgrade process itself proves out the quality of our infrastructure by achieving it with no downtime and minimum impact to users.