Microsoft says a software bug introduced in July caused last week’s worldwide Microsoft 365 outage, triggering a cascading failure that left more than 80% of its networking gateway servers unresponsive.
The almost 24-hour outage on November 12 stopped Microsoft 365 services around the world. This affected Outlook, Exchange Online, Teams, and Copilot, making it hard for users to access their email and slowing down message delivery.
In a preliminary post-incident report shared in the Microsoft 365 Admin Center, the company said the outage was caused by three problems: a previously undetected software bug, increased traffic after a Microsoft Teams failover, and reduced server capacity.
The event started at 11:20 AM ET on Thursday, November 12, and went on until 11:10 AM ET the next day, when Microsoft said that services were fully restored.
According to Microsoft, the software bug was introduced in July through an update developed for Microsoft 365 networking gateway servers.
While the update had been deployed for more than three months without issues, it contained a bug that could overwhelm servers under certain traffic conditions, causing them to stop responding.
Microsoft says its tests of the update did not detect the bug as it used less powerful testing hardware than the production servers.
“Our normal load testing did not detect the latent issue because the traffic mix and SKUs used in our labs,” Microsoft explained in its preliminary report.
“The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores.”
The bug happened after Microsoft Teams switched to the network used by Exchange Online about 15 hours before the outage, which made those servers busier.
Microsoft Teams had another outage earlier that day, but it’s not clear if the two problems are connected.
Microsoft says almost 14% of the gateway’s server capacity was already down because of a different problem. The system had enough capacity to handle the drop, but the mix of more traffic and less capacity caused the software bug.
As servers stopped working, Microsoft’s systems sent users to other servers. Those servers also got too much traffic and failed.
“In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive,” explained Microsoft.
Microsoft says safety measures meant to stop problems from spreading between areas also did not work, letting the outages spread globally.
Other Microsoft 365 services, such as Teams, OneDrive, and SharePoint, could change to different systems and recover faster. However, Exchange Online could not do this, so users couldn’t access their mail.
Microsoft says recovery was difficult because restarting affected servers caused them to be quickly overwhelmed again by backed-up traffic.
After identifying the bug, Microsoft engineers developed a fix and deployed it across affected servers, allowing services to stabilize throughout Friday.
Microsoft confirmed that active customer impact had ended by 11:10 AM ET on November 13.
The company says it has increased its capacity, improved its monitoring, and added protections to stop traffic redirection from causing similar failures. In another statement on X, Charles Lamanna, Microsoft’s executive vice president of Copilot, Agents, and Platform, said sorry for the disruption.
“We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations,” Lamanna said.
Microsoft says its investigation is continuing, with a final post-incident report expected later this week.
InfoSecBulletin Cybersecurity for mankind
