Some services are down due to Database
Resolved
Dec 28 at 06:35am EST
All services operational as of 11:06 UTC.
Affected services
Updated
Dec 28 at 06:01am EST
We have just finished spinning up a new database instance from backup and pointing traffic to it. We are seeing recovery for users, and are actively keeping a close eye on metrics.
Affected services
Updated
Dec 28 at 05:24am EST
We have escalated this issue to AWS, and also are working on restoring a copy of the database in parallel.
Affected services
Updated
Dec 28 at 05:08am EST
We are currently in the process of figuring out the best mitigation plan that will minimize risk and downtime once it is complete.
Affected services
Updated
Dec 28 at 04:15am EST
We are investigating and believe we have diagnosed the problem: we are having an active incident with our database and are actively working on mitigating.
Affected services
Created
Dec 28 at 04:12am EST
CPU functions, GPU functions, Snapshot restores (beta), and 2 other resources went down.
Affected services