Back to overview
Downtime

Some services are down due to Database

Dec 28 at 04:12am EST
Affected services
CPU functions
GPU functions
Snapshot restores (beta)
Sandboxes
Image builds

Resolved
Dec 28 at 06:35am EST

All services operational as of 11:06 UTC.

Updated
Dec 28 at 06:01am EST

We have just finished spinning up a new database instance from backup and pointing traffic to it. We are seeing recovery for users, and are actively keeping a close eye on metrics.

Updated
Dec 28 at 05:24am EST

We have escalated this issue to AWS, and also are working on restoring a copy of the database in parallel.

Updated
Dec 28 at 05:08am EST

We are currently in the process of figuring out the best mitigation plan that will minimize risk and downtime once it is complete.

Updated
Dec 28 at 04:15am EST

We are investigating and believe we have diagnosed the problem: we are having an active incident with our database and are actively working on mitigating.

Created
Dec 28 at 04:12am EST

CPU functions, GPU functions, Snapshot restores (beta), and 2 other resources went down.