Three outages in two days, and why we missed one for 15 hours
AgentRaaS Cloud was unavailable three times on 23 and 24 September. The first outage had a real bug behind it, which we found and fixed. The cause of the other two is still unknown. The longest one lasted about 15 hours before anyone noticed, because nothing was watching. A product that sells reliability should not find out about its own downtime by accident, so this is the full account, including what is still open.
What was affected
The dashboard and the API (webhook, SDK-style REST and MCP routes) on AgentRaaS Cloud. Self-hosted installations were not affected: they don't depend on our servers at all, and license checks are offline.
AgentRaaS is designed to fail closed. During these windows, calls routed through AgentRaaS Cloud returned an error; none of them could run against Stripe, Twilio or any other service without deduplication. That is deliberate: silently routing around an outage would recreate the exact double-execution problem the product exists to prevent.
Timeline
| When | What happened |
|---|---|
| 23 Sep | After a power cut, the production server came back, but the API container crashed on boot with Temporary failure in name resolution while connecting to the database: DNS wasn't ready yet, about 20 seconds after the network reported itself online. The container was configured not to restart automatically, so it stayed down. The public tunnel stayed up and returned a 502 error to every visitor until someone restarted the container by hand. This had happened three or four times before we traced it. |
| 23 Sep | Fixed: the container now restarts on failure, so a slow network at boot costs seconds instead of an outage. Deployed and verified the same day. |
| 23 Sep 23:40 | The server booted. Its last log entry was at 00:02. There is no record of a shutdown. |
| 24 Sep 14:54 | The server booted again. It had been down for about 15 hours, overnight, and nobody noticed. |
| 24 Sep 15:18 | Services stopped in order, the way they do when a shutdown starts, but the shutdown never recorded as complete. |
| 24 Sep 17:34 | Back up after manual intervention, about 2 hours 16 minutes later. By this point the homepage was already being served from Cloudflare's edge and stayed up; everything else returned an error. |
What went wrong
- A restart policy chosen for the wrong reason. Automatic restarts had been turned off to avoid an unrelated bug in how our container tooling orders dependent services. That trade-off was never revisited, so a transient DNS delay at boot became a full outage that needed a person to fix.
- No external monitoring. Nothing outside the server checks whether it is up. The 15-hour outage ran overnight with no alert. We found it afterwards, by reading the boot history.
- The server didn't come back on its own. It is set to power on when mains power returns, and that did not bring it back either time on 24 September. Whether these were power losses, a power button, or power management is still unknown.
- One server, one point of failure. The shared AgentRaaS Cloud environment runs on a single production server, so when it was down, the API was down. When a customer comes on, we move their workload to cloud infrastructure sized to their availability needs, and self-hosting removes the dependency on our servers entirely.
What we changed
- The API container restarts automatically on failure (23 September).
- The homepage is served from Cloudflare's edge instead of our server, so it stays up during an outage (24 September). On 26 September the rest of the public site followed: docs, guides and comparison pages.
What is still open
| Item | Status |
|---|---|
| Root cause of the two 24 Sep outages | Open The system logs from those boots haven't been reviewed yet. |
| External uptime alerting | Open Not in place yet. Until it is, an outage like the 15-hour one could go unnoticed again. |
| Restart fix under a real power cut | Open The configuration is verified; it hasn't been through a real outage since. |
| Infrastructure per customer | Planned Each customer's workload runs on cloud infrastructure sized to their availability needs, set up when they come on. |
| Automatic restart on failure | Done |
| Public site served from the edge | Done |
We will update this page as the open items close. If you run agents through AgentRaaS Cloud and this matters to you, self-hosting removes the dependency on our servers entirely and is free on every tier. Questions: hello@agentraas.io.