Facebook publishes more details on the technical error that led to a massive 6-hour outage that left billions of users out of the world’s largest social network and its other apps.
The outage was “caused not by malicious activity, but by error on our part,” Facebook vice president of infrastructure Santosh Janardhan said in a blog post on Tuesday.
The incident occurred when an engineer performing routine maintenance work issued an order “which unintentionally severed all connections to our backbone network, effectively disconnecting Facebook data centers around the world,” wrote Janardhan .
“Our systems are designed to audit orders like these to avoid such errors, but a bug in this auditing tool did not properly stop the order,” Janardhan said. “This change caused a complete disconnection of our server connections between our data centers and the Internet. And this total loss of connection caused a second problem that made matters worse.”
The new details come a day after the blackout that not only left users without access to Facebook, Instagram photo-sharing app and WhatsApp, but also cost them and those who operate their businesses or broadcast. advertisements on these platforms, potentially millions of dollars, if not Continued.
►Facebook whistleblower spills tea:Frances Haugen tells Senate Facebook is making “dire” choices, prioritizing profit
►Just chill:The Facebook breakdown, has Instagram strained you? 6 signs you need to unplug
Facebook, which also includes Instagram, dominates all social ad spend in the United States with an 82% share, according to research firm eMarketer. In comparison, the second closest is LinkedIn with 4.3%, Twitter with 3.8%, Pinterest with 3.4% and Snapchat with 3.1%. Others, including TikTok, are at 2.2%
For example, Facebook’s digital ad revenue in the United States is estimated at over $ 48 billion this year and $ 57 billion in 2022, eMarketer said. And, Instagram’s digital ad revenue is expected to reach $ 25 billion this year and more than $ 32 billion next year, reports eMarketer.
Social media has been the most popular marketing tool among small businesses, said Patrick Gillooly, Marketing Director at Constant Contact, a digital marketer. He said a recent Constant Contact poll found that 63% of small businesses use social media, primarily Facebook, to promote their businesses.
“A lot of businesses, small businesses in particular, rely on social media to start, grow and scale and (Monday’s blackout) could be a watershed moment for some of them,” said Gillooly.
Advertisers and businesses using Facebook are probably breathing a sigh of relief. The tech giant said Monday evening that there was “no evidence that user data was compromised as a result” of the outage.
However, one cybersecurity expert believes Facebook wouldn’t be so open had it faced an attack.
“Would they even admit to being attacked, given the current environment they find themselves in?” Said Tamara Schwartz, assistant professor of cybersecurity and business administration at York College in Pennsylvania. “Because a lot of people don’t already trust them due to recent accusations by the whistleblower about their unethical choices and Congress accusing them of being irresponsible to its users.”
Schwartz added: “It is not unreasonable to think that something else has happened.”
When the outage happened, Facebook suffered a double because “it all happened very quickly,” Janardhan said. when the failure occurred.
In the race to figure out what was wrong, the engineering team quickly learned that Facebook and its litany of apps had apparently been removed from the Domain Name System (DNS) servers that essentially make up “the address book. Internet, ”Janardhan said.
“(The engineers) faced two big hurdles: First, it was not possible to access our data centers through our usual means because their networks were down,” Janardhan said. “And second, the complete loss of DNS broke many internal tools that we normally use to investigate and resolve failures like this.”
Therefore, Janardhan said, the end result was that “our DNS servers became inaccessible even though they were still operational. This made it impossible for the rest of the Internet to find our servers.”
Since “Facebook’s out-of-band network access was down,” Janardhan said engineers had to go to the on-site data centers to “debug the problem” and restart the systems. But it took a while, he said, as these on-site locations have high levels of physical and system security.
“Once inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. So it took longer to activate the necessary secure access protocols for people to be on site and able to work on the servers. , Janardhan said. “Only then can we confirm the problem and get our backbone back online.”
While Facebook is back up and running, Janardhan has not ruled out that the platform and its properties will experience another outage, albeit less impactful, from happening again.
“I think a compromise like this is worth it – dramatically increased daily safety versus slower recovery after a hopefully rare event like this,” he said. “From now on, our job is to strengthen our tests, exercises, and overall resilience to make sure events like this happen as infrequently as possible.”