Protecting the Business, not Just the Data

“We are a very conservative organization” he told us. Many times. Throughout the day-long customer briefing.  Who is he? He is the CTO of a large healthcare organization.  We are a group of sales people and technical specialists covering technology solutions from modern application design and cloud to data center operations and data protection.

When it came time to talk about data protection, though, he shocked us. “We aren’t doing backups anymore for some of our most critical systems. Backups are useless to me.  By the time we get data back it’s too late.  Instead we replicate.”

I had a decade-old flashback to a CIO telling me “We don’t do backups because we use RAID-5.”

Introducing Business Risk

This could have devolved into a technical discussion comparing the spectrum of solutions all along the data protection continuum. But instead we took another course. I replied “Ok, you changed your data protection strategy based upon service level.  Did you also perform a business risk analysis to compare the different scenarios for which each strategy protects?”

This question got the attention and generated the well-balanced discussion I hoped it would. Data protection is often characterized by terms like Service Level Agreement, Recovery Point Objective, Recovery Time Objective, Retention, etc.  These are all important topics, and most people familiar with data protection know them well.  But the concept of business risk is equally important. After that discussion we concluded asking the same question.

What kind of events does any particular data protection solution defend against?

While there are many ways to lose data (and some days I feel I could start a Darwin Awards just for data loss), the main vehicles for losing data are malice, ineptitude, systems, and location.  Therefore, if you look at a few worst case scenarios, you’ll have a pretty good idea whether your data protection strategy holds muster.  For the record, this is a shorthand approach.  There is a longer methodology in performing a business risk assessment, but assuming you haven’t gone down that path, at least consider the following scenarios:

  • Disgruntled Sysadmin
  • Erroneous decommission of systems
  • System-level data corruption (bugs)
  • Ransomware and viruses
  • Untimely detection of data loss or corruption

I could write ten thousand words telling amusing stories about each of these scenarios; instead, I will pose the important question:

What can you do to mitigate the business impact of data loss incidents?

Protecting the Business Against Data Loss

In no particular order, here are the factors that help companies protect against these worst-case scenarios:

  • Security Officer Role
  • Segregation of Duties
  • Heterogeneity
  • Retention Policies

A security officer role is built into various types of information systems. The general idea is to require two independent people to authorize exceptionally high level activities such as initializing a file system or making changes to root level access.  If you have a system that has a security officer role, enable it and make the security officer someone who is not also a system administrator.

Segregation of duties is the principle that people should have access rights only to perform their essential job functions. In data protection, it means that a person with root access to the primary system should not also have root access to the backups, and vice versa.

Heterogeneity means that your data is stored on at least two different kinds of systems that share nothing. This makes segregation of duties easier to accomplish, and also protects against data loss due to system wide bugs and malware.  This could mean different storage arrays, different public clouds, or anything in-between.  Several NIST working groups embrace the idea of heterogeneity for various reasons, and data protection certainly on that list.

Retention policies are nothing new, and often are viewed together with the earlier stated list of service levels. However, retention policies are important to consider in the context of silent or undetected data loss or corruption.  While everyone would prefer to recover the most recent copy of data that they have, it’s not always going to fix the problem.  Many people use technological considerations to establish retention policy rather than business risk.  Just remember that data doesn’t always raise a red flag when it is deleted or becomes corrupt.  My view is retention policies for most data must be a minimum of five weeks for daily copies, and longer depending on data value, compliance requirements, and data access frequency.  Most importantly, this should be an “eyes wide open” business decision, and not a choice determined by the convenience of the Sysadmin and his or her favorite technology.

A Successful Business Conclusion

Going back to our healthcare CTO… He summarized the discussion with, “I had no idea that we had changed all of that.” The customer had cut retention times to a fraction of their former values, given the keys to the kingdom to a single individual, and exposed themselves to malware and ransomware unknowingly – all for the sake of improving their service levels.  Fortunately, with a new data center build in progress, the customer decided that all the data protection in the new facility would be built from scratch, and that a business risk assessment would form the baseline for determining the data protection policy.  In the meantime, backups that had previously been cancelled would be reinstated.

Remember:

What kind of events does any particular data protection solution defend against?

What can you do to mitigate the business impact of data loss incidents?

Without that, protection is just a set of tactical technical steps, not part of a business strategy.

This organization caught it early. It could have been much worse. I could tell you some stories…

Rich Colbert

One Reply to “Protecting the Business, not Just the Data”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s