How 83% noise suppression saved Vlad a million dollars so far this year
I met Vlad in the bar in Vegas after a long day of telco NOC drudgery. He was enjoying his whisky and clearly didn’t want to be interrupted by me asking about his datacenter. I could tell he’d rather I had asked about anything else… Cat Stevens, Greek myths, Faberge eggs. Anything. I interrupted him anyway and asked what’s required to go from the three nines he referenced in his keynote to the five nines his customers demand. He winced in pain. I thought he swallowed an ice cube or his Johnnie Walker was laced with cyanide. Turns out he was deep in thought. He proceeded to share wisdom that inspired me… to drink whisky and grow facial hair.
Vlad told a tale about his NOC in 2013 humming with thirty-five fresh-faced IT sycophants huddled together like immigrants nursing blinking lights from red to green and playing digital whack-a-mole with performance alarms. He then said the journey from two nines to three was painful but he’s got a strategy for going from three to five by end of year.
My eyes got wide.
He said what makes life tough for his young NOC team is that as the services they deliver – voice, video, and internet – become more critical to their subscribers and the competition gets better the apps themselves and the infrastructure required to support them has become more complex. There are more tools to manage, more developers pushing more code more frequently, and more executives scrutinizing over every second of downtime.
Vlad said he can’t afford to solve the problem like he did in 2013. He said it would take two years and another thirty-five operators to brute force even one more nine. He said the answer is making the blinking lights stop, isolating the root cause of issues faster, and extracting more knowledge from every incident.
He then put down his whisky. I feared it was to drop a haymaker on my left jaw or find someone interested in discussing Faberge eggs. I was wrong.
Instead, Vlad took out his phone and gave me a tour of BigPanda. He explained how in three months they’ve used it to reduce incidents by 83% and they’re already pushing four nines of uptime. I grinned wide, introduced myself as a panda, and we hugged it out like soul mates.
It was just Vlad and a snifter in a dirty bar but I wouldn’t have been more inspired listening to Dr. King on the courthouse steps in Montgomery. It takes more than people to fix a problem created by machine data and he gets that better than anyone.
Vlad, buddy, I expect you to demo BigPanda instead of me at the next event. Then I’ll buy you a whisky.