GitHub Was Down for 8 Hours. Retries Made It Worse.

In August, GitHub went down for almost eight hours.

It started when one part of its system got overloaded. Then everything that couldn’t connect kept trying again, all at the same moment. All that extra traffic made it even harder for GitHub to recover. In one case, the retries pushed traffic up to about 10 times its normal level.

Retrying sounds like the safe thing to do. If something fails, try again. But most systems retry straight away, over and over, with no break in between. When thousands of apps do that together, the problem just gets bigger.

Retries that wait work better. That’s how we handle them in Unmeshed. For each step in a workflow, you decide:

• how many times it should try again
• how long it waits between tries
• whether the wait grows each time, like one second, then two, then four

If the step still fails after that, Unmeshed stops it and can start a backup plan, like alerting your team. No pile-up of requests, and the system gets room to recover.

Want to see how it works? Take a look at API Retry and Failover | Unmeshed