Search results
People also ask
What happened to Roblox?
Why did Roblox go down?
What happened to Roblox on October 31?
Why did Roblox crash?
Nov 1, 2021 · The game-creating platform was restored on Sunday afternoon, after being dark for more than two days. In a blog post, Roblox founder/CEO David Baszucki apologized for the lengthy delay in...
- Outage Summary
- Preamble: Our Cluster Environment and Hashistack
- Initial Detection
- Early Triage
- Return to Service Attempt #1
- Return to Service Attempt #2
- Research Into Contention
- Root Causes Found
- Restoring Caching Service
- The Return of Players
The outage was unique in both duration and complexity. The team had to address a number of challenges in sequence to understand the root cause and bring the service back up. 1. The outage lasted 73 hours. 2. The root cause was due to two issues. Enabling a relatively new streaming feature on Consul under unusually high read and write load led to ex...
Roblox’s core infrastructure runs in Roblox data centers. We deploy and manage our own hardware, as well as our own compute, storage, and networking systems on top of that hardware. The scale of our deployment is significant, with over 18,000 servers and 170,000 containers. In order to run thousands of servers across multiple sites, we leverage a t...
On the afternoon of October 28th, Vault performance was degraded and a single Consul server had high CPU load. Roblox engineers began to investigate. At this point players were not impacted.
The initial investigation suggested that the Consul cluster that Vault and many other services depend on was unhealthy. Specifically, the Consul cluster metrics showed elevated write latency for the underlying KV store in which Consul stores data. The 50th percentile latency on these operations was typically under 300ms but was now 2 seconds. Hardw...
The first two attempts to return the Consul cluster to a healthy state were unsuccessful. We could still see elevated KV write latency as well as a new inexplicable symptom that we could not explain: the Consul leader was regularly out of sync with the other voters. The team decided to shut down the entire Consul cluster and reset its state using a...
We had ruled out hardware failure. Faster hardware hadn’t helped and, as we learned later, potentially hurt stability. Resetting Consul’s internal state hadn’t helped either. There was no user traffic coming in, yet Consul was still slow. We had leveragediptablesto let traffic back into the cluster slowly. Was the cluster simply getting pushed back...
Over the next 10 hours, the engineering team dug deeper into debug logs and operating system-level metrics. This data showed Consul KV writes getting blocked for long periods of time. In other words, “contention.”The cause of the contention was not immediately obvious, but one theory was that the shift from 64 to 128 CPU Core servers early in the o...
Several months ago, we enabled a new Consul streaming feature on a subset of our services. This feature, designed to lower the CPU usage and network bandwidth of the Consul cluster, worked as expected, so over the next few months we incrementally enabled the feature on more of our backend services. On October 27th at 14:00, one day before the outag...
It had been 54 hours since the start of the outage. With streaming disabled and a process in place to prevent slow leaders from staying elected, Consul was now consistently stable. The team was ready to focus on a return to service. Roblox uses a typical microservices pattern for its backend. At the bottom of the microservices “stack” are databases...
The final return to service phase began officially at 05:00 on the 31st. Similar to the caching system, a significant portion of running services had been shut down during the initial outage or the troubleshooting phases. The team needed to restart these services at correct capacity levels and verify that they were functioning correctly. This went ...
Oct 31, 2021 · Roblox, the gaming platform that is immensely popular amongst young players, said on Twitter Sunday evening that it is back online worldwide. Roblox is back online everywhere! Thank you for...
- Rita Liao
Sep 8, 2023 · Updated Apr 4, 2024, 1:34 PM PDT. Roblox: all the news about the popular social and gaming platform. By Jay Peters, a news editor who writes about technology, video games, and virtual worlds....
- 52 sec
- Jay Peters
Nov 2, 2021 · 2 November 2021. Comments. Roblox. The site is back up again after a weekend-long outage, but what actually happened? If you were having trouble logging into Roblox this weekend, you weren't...
Dec 21, 2023 · October 31, 2021: The website started to come back at 8 a.m. Roblox had announced a few hours earlier that they had encountered the root cause of the issue and figured out how to solve it. By 5:45 p.m., about 70 to 72 hours after the incident started, the Roblox website was completely functional and players were able to return to their hobby.
