Back to overview
Degraded

A fraction of new containers are experiencing errors

Feb 20 at 08:30am EST
Affected services
CPU functions
GPU functions
Web endpoints
Sandboxes
Image builds

Resolved
Feb 20 at 01:43pm EST

All issues have been resolved.

Updated
Feb 20 at 01:01pm EST

Functions are fully restored. We are still investigating possible lingering effects for a small number of Sandboxes that were launched between 13:30-14:44 UTC.

Updated
Feb 20 at 12:01pm EST

New containers are working normally. We are continuing to investigate possible lingering effects for recently started containers.

At 13:30 UTC a bad change went out causing new containers to potentially get stuck or experience errors. At 14:44 UTC a the bad change was fully reverted and the disruption mitigated.

We experienced a secondary, lesser problem with similar symptoms between approximately 16:00 and 16:15 UTC as part of the mitigations put in place during the response.

Updated
Feb 20 at 09:44am EST

Issues with some image builds getting stuck have been resolved.

Created
Feb 20 at 08:30am EST

We have identified the root cause and are taking action to mitigate.