Big Data. I sat with my head in my hands, buried by budgets, purchase orders, escalations with IT, and visionary presentations that now mocked me. “How did I end up here? I’m supposed to build innovative products that others struggle to deploy (that’s not really my goal, but I was despondent), not the other way around!” That was the nadir of my Big Data journey. In the past year, it has become a moderately successful, revenue generating, innovation enabling, and constantly aggravating solution. How did we get here and what recommendations do we have?
First, properly set expectations for Big Data. One executive asserted– “If you get enough data together in one place, it automatically generates answers to questions you didn’t know to ask.” He believed that a primordial soup of data would spring to life. As you invest in Big Data, emphasize that the goal is to help answer questions that have been too complex or expensive to answer. Otherwise, you’ll spend hours explaining that Hadoop, Cassandra, and CouchDB are tools, not sentient beings.
Second, avoid “Data Scientists” at the beginning. They will be useful later, but not at the outset. There are two types of “Data Scientists”. The former know how to (or want to learn how to) set up a huge Hadoop or Elasticsearch cluster or a big NoSQL database. The latter know statistics, the ‘R’ programming language, and graph theory. In both cases, they’re a solution in search of a problem. People who follow “If you build it, they will come” find themselves asked to go elsewhere.
Third, listen to parts of the company that lack a voice. Big Data reduces the cost or complexity of solving a problem. The business has already found a way to meet its needs in business critical areas, usually with a large investment in traditional technologies. Ultimately, Big Data can streamline those operations, but it’s not the place to start. Instead, look for areas where the business has been unwilling to invest. Our journey to success began when a support engineer observed that we could use Big Data to predict within 90 days that a Data Domain would run out of capacity. He was tired of taking support calls about “failed backups” because the backup teams were not trained to monitor storage capacities. He knew we could do better, and Big Data allowed us to do it without a huge investment.
Once you have properly set expectations, avoided the pitfalls of gratuitous investment, and found a critical, but underappreciated problem to solve, there are three success factors:
- Be Open – Too often organizations will create a Big Data Lake, only to prevent anyone from accessing the data. Innovation comes from bringing creative people and data together. Governance is important, but don’t let IT lock everyone out of the Lake.
- Revenue vs. Optimization – Many people want to optimize a process (e.g. fewer support calls, faster bug triage), but “optimization” is difficult to quantify and even harder to justify investment. Instead, focus on ways that Big Data can augment your revenue. At first, we futilely tried to get funding by demonstrating “reduced support case load.” Interest and funding expanded when we tracked the revenue generated by selling additional Data Domain storage and systems to customers who were about to run out of capacity.
- Generalist vs. Specialist – At the beginning, you don’t need a hyper-optimized Big Data infrastructure. You need somebody who understands the business problem, what data they need, how to access the data, and how to deploy basic Big Data tools. In short, you need a problem solving generalist who can learn quickly. As the solution expands, you can hire specialists to optimize each part of the process. At the beginning, though, generalists win.
As with most business/technology transformation, the challenge with Big Data is not one of technology. To succeed with Big Data, manage business expectations, avoid technology hype, and embrace revenue-generating ideas from underfunded areas. If you keep your Big Data Lake open and accessible, you’ll unlock the innovative passion of parts of the company that have been desperate to do more. And with all that, you can join me in feeling that mix of satisfaction and dull irritation that comes with knowing you should be doing 100 times more with your Big Data.
Stephen Manley @makitadremel