Previous incidents

Due to network degradation in GCP, Modal is compromised in its ability to maintain Worker capacity and container health.

If you have GCP-pinned (cloud="gcp") Functions or Sandboxes, unpinning will result in avoiding this issue.

Update: The incident is now resolved.

Cloudflare incident: https://www.cloudflarestatus.com/incidents/t8mq7d1yjqm4

We are remediating this by shifting Modal Workers away from us-east-1.

We have scaled up capacity successfully and redirected traffic to work around the Oracle Cloud regional networking outage in Ashburn, VA.

Newly created sandboxes are fully operational. Some sandboxes created during the incident may not have been able to spin up.

Volumes and Image builds recovered.

Image builds recovered.

Everything is back up.

CPU functions, GPU functions, and Image builds recovered.

Web endpoints, Frontend, Web API, and 1 other resource recovered.

Modal systems are back up. We're monitoring closely.