Sam Kendall’s noisy alert problem
Sam’s a father of two boys living in the bucolic LA suburb of West Covina. He’s a family first guy who paints model military cargo planes for fun, makes award-winning paella, hates his commute, and loathes his phone between the hours of midnight and 4:00 AM.
Sam was a kid when he joined News Corp as a help desk analyst in 2000. More than 15 years later and he’s now Sr. Director of IT managing a growing team of 30 NOC engineers, sys admins, and DBAs. Over the years, he has received more promotions than Trump on his own Twitter feed by delivering results and never wavering from two core beliefs that influence everything he does:
- Technology serves customers. Not the other way around.
- Great tools are no substitute for great process.
Like most BigPanda customers, Sam’s NOC uses more monitoring tools than they should, but they’re too dependent on them to consolidate and too invested in years of configuration to know where to start if it was a priority. They monitor the network with SolarWinds, servers with Nagios, AWS workloads with CloudWatch, uptime with Pingdom, synthetic transactions with Catchpoint, logs with Splunk, and performance with New Relic.
As a result, routine tasks like upgrading the firmware on a router often generate hundreds of redundant alerts. Sam tried everything to suppress noise without missing real issues. He failed. Rosary beads didn’t work. Rabbit feet gave his kids nightmares. Years into his cage match with 2:00 AM wake-up calls he called us.
With BigPanda, he sees one incident per actual issue with a live timeline that provides details about each related alert, including when it occurred and why. His team integrated seven monitoring systems in about a day, and now all event correlation happens in the cloud while he sleeps.
When asked how he responded to a recent attempt to replace BigPanda, he went all Heston with a heartfelt “…they’ll need to pry it from my cold, dead hands.” Before BigPanda, everything in Sam’s world was structured and predictable…except his noisy alerts. Now the chaos is gone. His wife and kids say he’s less grumpy and his team says he has more time for them.
Site visits like this are the best part of my job. They’re a reminder that what we do is bigger than event correlation or monitoring or even technology. Sam, thanks for an excellent day. No need to thank us…just send paella (and hold the clams).