Zayo

Zayo is the leading global communications infrastructure platform, delivering a range of solutions including fiber and transport, packet, and managed edge services. Zayo’s strong, AIOps-driven platform empowers IT teams and lines of business with complete edge-to-cloud solutions.

Challenge

• Zayo’s managed services business is increasing 30% annually, adding 20,000 devices to oversee—well beyond team capacity.
• Instead of only examining proper warnings, the network operations center (NOC) wasted time chasing false positives, duplicates, and benign alarms across various tools and terminals. The amount of unique notifications quickly outgrew manual human efforts.
• The company had to scale through technology, not through headcount.

Zayo’s infrastructure was in the midst of rapid business growth, which directly translated to increased responsibility to monitor tens of thousands more devices each year. An unviable tsunami of IT noise was subsequently generated, quickly surpassing what human incident response teams could realistically manage across various tools and terminals. The inability to scale IT operations through technology hindered Zayo’s ability to adequately prevent and resolve outages.

Jon Moss, head of edge software engineering at Zayo, recognized that this escalating IT cacophony was unsustainable. “Due to the volume, my team could not manually filter for root cause or prioritize alerts. Before BigPanda, events went directly to our NOC without processing and were converted into ServiceNow incidents. The NOC lacked context for those events. The NOC was drowning in noise, unable to scale and unable to keep pace with the growth of the business. You can’t employ enough people to match the deployment and business needs from the large number of events we were generating.”

Increased mean time to resolve (MTTR) was decreasing customer satisfaction and threatening important relationships. “We spent too much time chasing ghosts, which caused client discontent. Our end-users and customers complained that we were missing events, handling the incorrect events in the wrong order, and not resolving events fast enough. It was a general state of chaos and something we felt like we couldn’t scale unless we made a pretty major shift,” says Moss.

Solution

AIOps helped Zayo address these issues. Zayo can now automatically process huge event noise using BigPanda Alert Intelligence.

Alert Intelligence dramatically reduces IT noise by de-duplicating, filtering out false positives and benign events, and aggregating events into alerts. Customers can enrich alerts with context buried within alerts’ own payloads (i.e. location, host, or affected service) to deliver a stream of high-quality, actionable alerts so ITOps have a first pane of glass to quickly understand what is happening in their IT environments.

Zayo’s NOC team is now able to identify and judge the significance of alerts and determine next steps accordingly. “Being able to take highly enriched, highly contextualized alerts and turn them into much fewer and much more actionable incidents was really transformational for us,” adds Moss.

Benefits

Figure 1. Zayo’s alert flow with BigPanda over a three month time period. 52,734,759 events were correlated into 1,300,987 incidents (98% suppression). From that, 5% were auto-shared, and 95% were non-actioned.

Zayo reduced incidents drastically. BigPanda Alert Intelligence transforms noisy, low-quality events into fewer, high-quality alerts that accurately point to issues that need attention, no matter where they originate.

• BigPanda’s Alert Intelligence service ingested and filtered 25M Zayo events in three months, reducing them by 90% to 2.52M alerts.
• 1.04M BigPanda incidents (correlated group of alerts) were further compressed by 59%.
• 25,023,583 events were turned into just 203,050 ServiceNow tickets
• 20% reduction in ServiceNow tickets.
• BigPanda Alert Intelligence filtered out 99.991% of events.

This refined level of transparency is also an advantage to their customer experience. “We can now route [alerts] to the appropriate teams. We get them to that team faster and reduce MTTR, which makes the customers really happy. They get really good visibility into what’s happening in their network, and they know that we’re working on the right things with the right people as quickly as possible,” says Moss.

Zayo now has a clean and manageable data flow that equips Zayo to scale their IT and grow their business like never before thanks to BigPanda and Alert Intelligence.

“We are getting ready to hit a scale that is unparalleled, something that we couldn’t do with our previous tooling and manual processes. Automating our entire technology stack with BigPanda at its core is allowing us to drive a better MTTR, deliver better customer experience, and better scale using technology, not headcount. All of these things that BigPanda has now unlocked for us is going to drive a much much better experience for us in the end. There is no limit to what we can do now that we have BigPanda in place.”