How Gamma reduced alert noise by 93%

“Through the Integration Hub, BigPanda exposed [our] experts and their knowledge in the platform, allowing us to identify the root cause of an incident and reduce the overall time.”
Dan Bartram
Head of Automation and Monitoring, Gamma Communications

93%
reduction in alert noise
60%
less noise across the IT stack
40%
automated alert closure

Gamma Communications plc is a leading supplier of communication services in the U.K., German, Spanish, and Dutch business markets. Gamma is admitted to trading on AIM and employs approximately 1,900 people. With a range of UCaaS, mobile, and connectivity services, Gamma provides robust and secure solutions that enable organizations to communicate, collaborate, and offer a better customer experience.

Challenges

Gamma grappled with significant alert-management challenges from its email-alerting system. The IT team embarked on a project called “Inbox Zero,” a mission to reduce its reliance on email alerts. Despite having a low overall incident volume — Gamma maintains six-nines uptime — the sheer volume of alerts caused significant operator fatigue. The situation was unsustainable, requiring more personnel and higher costs to manage.

With email as the primary alerting mechanism, Gamma could review only 3% of alerts it received due to highly manual identification and correlation processes. It was challenging to identify what was broken with so many alerts.

“It was simply unsustainable,” says Dan Bartram, head of automation and monitoring at Gamma. During incidents, the entire team would join calls without a clear context of the issue they needed to solve. They spent significant time identifying what was broken instead of fixing it — a time-consuming and costly process.

Dan and the team knew there was a smarter, more efficient way to enrich, correlate, and automate alerts, leading to significant team productivity improvements and cost savings.

Solution

Gamma wanted a solution that would reduce alert noise and enhance the context of each alert. Doing so would minimize inefficient bridge calls and translate directly to lower operational costs. The ultimate goal was to achieve a closed-loop system.

BigPanda emerged as the ideal solution through a proof-of-value project that showcased the platform’s benefits using specific data sources within Gamma’s infrastructure. The demonstration was pivotal in securing internal buy-in for the purchase of BigPanda.

BigPanda addressed Gamma’s biggest pain point by significantly reducing alert volume and improving alert quality almost immediately.

“Within two weeks, we had a substantial reduction in alerts — and better alerts,” shared Bartram. Gamma integrated many of its alert sources into BigPanda. A simple, straightforward process, the integration provided the ability to see where alerts originated, resulting in an “instant bang for the buck.” This immediate impact led to further investment in and expansion of the BigPanda implementation.

In addition to creating visibility into alert sources and reducing volume, Gamma gained new capabilities to predict the impact of alerts and incidents using the Incident 360 Console. Previously, that knowledge existed within dedicated teams but wasn’t accessible across the business.

“The expertise was in the company already. BigPanda gave us a platform that allowed us to share that expertise across the business,” shared Bartram. “Through the Open Integration Hub, BigPanda exposed our experts and their knowledge of the platform, allowing us to leverage it to identify the root cause of an incident and reduce the overall time of an incident.”

Benefits

Implementing BigPanda led to a dramatic 93% reduction in alert noise for Gamma, greatly improving operational efficiency and reducing the valuable employee resources required to manage these alerts.

“The majority of what the network management team was looking at is gone now, simply by onboarding those alerts into the BigPanda platform,” Bartram explained. “Some platforms are heavily integrated with a need for professional services support. BigPanda was not like that; it was simple for us to configure.”

Gamma achieved 60% noise reduction across its entire tech stack. They did it by integrating 80% of its monitoring systems into BigPanda within six months, with plans to eradicate email alerts within seven months of onboarding. The Gamma team accomplished the project in- house without needing extensive professional services support.

Additional benefits included reducing incident volume and creating the ability to automate the closure of 40% of alerts without manual support.

With the help of BigPanda, Gamma increased the efficiency of its teams and incident management process. Alert volume initially increased during onboarding because they were looking at all alert sources. Using tags and correlation, however, they removed alerts and reduced noise significantly.

“Not only can we see the alerts, but we can evaluate them using correlation that recognized patterns, connected alerts, and led to fewer incidents,” according to Bartram. He attributes the implementation success to a “don’t boil the ocean” approach. The team prioritized their top challenge and focused solely on fixing it.

“BigPanda has been a breath of fresh air into a minefield of alerts that left scale feeling unachievable,” Bartram said. “The platform is easy to work with and get started, providing flexibility to integrate with existing platforms.”

Going forward, Gamma aims to use the BigPanda platform to move toward a closed-loop system. The goal is to selectively monitor critical alerts, push relevant ones to AWS, and automate responses wherever possible. Gamma intends to create a more intelligent and automated alert-management system, further reducing manual intervention and enhancing operational efficiency.

“We want to identify the scenarios where we don’t even need a human engineer, then automate them,” said Bartram. “BigPanda is fundamental for that.”