Solarwinds is a 21-year old publicly traded monitoring and network management vendor with 300,000+ customers across the world. It’s familiar to IT operations and monitoring teams across enterprises big and small. And this week, it found itself in the news for all the wrong reasons.

The Wall Street Journal summarized what happened,

In the latest incident, hackers appear to have gained a foothold in their victims’ networks by adding “back door” code to SolarWinds Orion software, according to an analysis of the event by Microsoft Corp. Once installed, this software connected to a server controlled by the hackers that allowed them to launch further attacks against the SolarWinds customer and to steal data. The vulnerable updates were delivered to customers between March and June, SolarWinds said.

while The Economist framed the hack in stark terms:

Hackers have vaulted into the heart of America’s government.

The hack’s blast radius

This “supply chain” attack began months ago when a highly resourceful and determined hacking operation infiltrated Solarwinds’ update build system, and then used it to compromise a routine patch to the highly popular Orion observability tool, hosted on Solarwinds’ own website.

While the specifics of how hackers were able to hijack a routine patch are still murky, up to 18,000 Solarwinds customers are believed to have downloaded the infected update.

For now, the attack seems localized to roughly 425 of those 18,000 Solarwinds customers, but this number seems sure to grow. For example, the US Department of Homeland Security isn’t wasting time – and has issued an emergency directive to government organizations to check their networks for the presence of the trojanized component and report back.

How did it go undetected for so long?

As details begin to emerge, CSO has thoroughly detailed the attack for IT Ops leaders, and why the attack went undetected for months:

The attackers managed to modify an Orion platform plug-in that is distributed as part of Orion platform updates. “After an initial dormant period of up to two weeks, it retrieves and executes commands, called ‘Jobs,’ that include the ability to transfer files, execute files, profile the system, reboot the machine, and disable system services,” the FireEye analysts said.

To avoid detection, attackers used temporary file replacement techniques to remotely execute their tools. This means they modified a legitimate utility on the targeted system with their malicious one, executed it, and then replaced it back with the legitimate one. A similar technique involved the temporary modification of system scheduled tasks by updating a legitimate task to execute a malicious tool and then reverting the task back to its original configuration.

It may take a long time before we know what was compromised / stolen (in some cases we may never find out, at least publicly), how that information will be used, and by whom – but we can be sure that the fallout will be massive, expensive and long-lasting.

How can IT operations leaders avoid this fate in the future?

If you’re an IT operations leader or even a CIO at one of the affected customers, the news can’t get worse than this. If you’re a customer that hasn’t been affected, you’re thanking your stars and vigorously assessing the other tools in your environment for similar risks.

So how can you keep this from happening in your organization?

It’s critical to note that Solarwinds Orion is an on-premises (or on-prem) product, requiring local resources to install and manage. So in addition to the extra resources (hardware and people) on-prem products require to keep them working, these products carry a whole host of security risks that you and your IT leadership should carefully evaluate.

Here are four key considerations for IT operations leaders and stakeholders when evaluating the security of their monitoring, observability and IT operations tooling investments.

1. “Supply chain” infiltration risks: When you select a SaaS solution, you retain control by deciding what data you send to the SaaS provider – you don’t need to install complex software locally that could potentially access other systems and data within your corporate network.
   
   That’s not the case with on-prem tools. And this was a major problem with the Solarwinds “supply chain” hack. The attackers were able to use the compromised patch to infiltrate other systems, and it’s likely that they used those systems to infiltrate even more systems.
2. Elevated permissions and privileged accounts raise risk: With SaaS-based software, you don’t need to install complex third party software within your on-prem network.
   
   With on-prem software however, you often have to grant elevated permissions or highly-privileged accounts for the software to run, which creates risk.
3. Compromised patches: With SaaS-based software tools, you don’t have to review vendor patches or hotfixes being pushed to your SaaS-based deployment. That appears to have opened the exposure for Solarwinds Orion, when the software build system was compromised starting with the build for version 2019.4 HF 5. Ironically, the most exposed Solarwinds customers were the ones that were actually diligent about installing Orion patches. Anyone running a downlevel version of Orion wasn’t impacted by this hack. Sadly, this is an example of IT shops choosing an on-prem solution, “doing everything right” in terms of staying up-to-date on patches, and belatedly finding out that those actions actively put them in greater danger.
   
   
4. Safe harbor for malicious code: With a SaaS solution, you don’t have to exclude directories or policies from antivirus and antimalware scans.Yet that is how the Solarwinds exploit seems to have avoided detection, since the malicious code had a safe harbor to help it avoid detection.

A recent eWeek article by editor-in-chief Chris Preimesberger summarizes these lessons learned, and why IT should rethink on-premises tooling.

The security benefits of SaaS-based monitoring, observability and IT operations tools

Well-designed SaaS-based tools provide a superior level of security in almost all cases. Operations leaders and stakeholders considering IT monitoring, observability and IT operations tools should consider the following:

1. SaaS-based architectures can be modern and secure: Most SaaS providers use a modern, secure architecture that compartmentalizes data, security, and identity and access management into different cloud accounts. Access to critical systems is restricted to staff based on the “principle of least privilege”, and user access often requires multi-factor authentication using a code generator. It’s also important to note that most enterprise SaaS solutions are designed to be secure and must comply with independent security standards, such as annual SOC 2 Type II security audits and independent application “penetration tests” to ensure that known application vulnerabilities are remediated. Enterprise SaaS solutions are also hosted in secure data centers aligned with the ISO 27001 certification standard.
   
2. SaaS-based data is encrypted: Data at rest is generally encrypted by default. Encryption keys are secured in a Key Management System (KMS), where keys are encrypted. This helps secure customer data in the rare event of a rogue application gaining access to the SaaS solution.
3. SaaS-based tools use information compartmentalization: As discussed above, with on-prem software, when bad actors gain access into
   your network environment, they can potentially read any of the data flowing on that network. They often move laterally, from one device, host, application, or service, to another, and extract everything they can. By design, that’s just not possible with SaaS-based tools.

Combating phishing and social engineering: what’s the plan?

Last but not the least, while we still don’t know for certain how Solarwinds’ build process was compromised in the first place, a filing by the company suggests that an employee’s Office 365 account was possibly compromised.

This is a common attack vector by hackers for data exfiltration and ransomware, which is conducted through a social engineering or phishing attack via email.

While anyone in the organization can be compromised, your most critical staff – think CEO, CIO, head of engineering – are often targeted by hackers since they are likely to be the busiest and have the least amount of time to inspect emails for phishing attack indicators; and they have access to the most information and critical systems.

An ongoing security training program for all staff, including realistic phishing simulation tests, is key to raising awareness to reduce the risk from social engineering attack vectors.

But in this instance, such a program would not have helped affected organizations because their exposure came from a 3rd party vendor. That’s why today, every single organization and every group within every organization must have a robust plan in place to combat the risk of phishing attacks and social engineering – both internally AND with their vendors. Untrained users are often the weakest link in an otherwise strong environment.

Moving From On-Prem to SaaS Is Easier Than Ever Before

Before cloud computing and SaaS solutions became prominent, on-prem software was the standard for large enterprises and institutions. On-prem applications were considered reliable, secure, and provided customers a level of control over the infrastructure and data. Often, those customers built dedicated, tightly-controlled data centers to host those applications.

Over the last few years, that has started to change.

Here are some of the ways enterprises that historically preferred on-prem software solutions are adapting to a SaaS-first world as part of their digital transformation initiatives:

1. Requirements due to industry regulations: Some companies were required to deploy on-prem solutions because they were in highly regulated industries or subject to additional privacy constraints or compliance reporting, such as healthcare and utilities or when working with the federal government.
   
   But that is changing: large healthcare, utility and financial customers have started to embrace cloud- and SaaS-based solutions. Even the federal government has recognized the benefits of SaaS solutions through their FedRAMP certification program.
2. Software licensing preferences: Other companies preferred the licensing model of on-premises software, where they could purchase a perpetual license, deploy the software in-house, and then optimize for performance by provisioning systems and network resources in their datacenter.With SaaS-based tools, customers can typically scale their licenses up or down based on the number of users, the amount of data consumed or processed, the number of API calls, etc., which affords significantly higher operational and cost flexibility. On top of that, with SaaS-based tools, as long as users have access to a web browser and a standard high-speed network connection, the IT department doesn’t need to waste resources and time on optimizing for performance, with load balancers, HA/DR clusters, etc.
3. Uniqueness of the business: Many large enterprises believed their business was so unique that only on-prem solutions allowed for the extensive customization required to support their business. They also put in place expert teams who could be responsive to internal customers that were reliant on their heavily-customized on-prem solutions.
   
   But as IT applications and systems have become increasingly complex, having a large bench of specialists on staff, with in-depth knowledge and expertise at all levels of the technology stack, has become more difficult and expensive. Additionally, heavily-customized applications can become difficult to manage, support and update – because updates and patches can potentially interfere with those customizations.Today, many enterprise solutions used by IT departments and different business functions (e.g. Marketing, Sales, Operations, etc.) across most industries have moved towards standard operating procedures and away from customized business processes. This movement to standards and best practices makes it easier for companies to make the switch from custom on-prem solutions to SaaS-based tools.

In Summary

Solarwinds is a ubiquitous monitoring / network management tool. Per Gartner, as quoted in this article, Solarwinds is the #3 provider of IT operations software, behind only Splunk and IBM. That, combined with Solarwinds’ low corporate profile, likely made it an attractive target for the hackers.

Customers compromised their systems unwittingly by following standard best practices – downloading and installing updates and patches based on their vendors’ recommendations…which highlights the unacceptably high security risks associated with on-prem software.

Victims of compromised on-prem software pay the price both publicly and behind the scenes.

IT operations leaders and executives should consider adopting modern SaaS-based tools for monitoring, observability, event correlation and automation, and collaboration. Such SaaS-based tools significantly mitigate the security risks that come with on-prem software, while delivering on the benefits associated with SaaS-based products – such as elastic scaling, lower TCO and rapid time to value.