From Enterprise Resource Planning (ERP) to online video and everything in between, there’s a challenge that every technology once considered “emerging” has faced: the dreaded Trough of Disillusionment, made famous by Gartner’s influential Emerging Technology Hype Cycle reports. There’s a certain inevitability to the Hype Cycle – no matter how many headlines or how much fanfare a technology receives at the onset, there’s a universal dip into that Trough of Disillusionment as market realities set in and the world adapts to and adopts (or doesn’t adopt) the latest, greatest “next big thing.”
For the first time last year, big data found its way into the Trough of Disillusionment, which makes sense. We’re just starting to see organizations across the globe make good on the promise of harnessing big data – and I have no doubt that it will soon graduate into the Plateau of Productivity. Based on some of the chatter I hear across the industry, it would appear that Hadoop may be on the heels of big data, approaching its own trough.
This may be because we seem to be stuck in a bit of a tricky phase of experimentation and innovation when what we really need is to see operational customer data management solutions with real-time scoring, supporting frameworks for actionability. Isolated wins are one thing, but staying power means creating those frameworks for long-term, sustained success.
So if that’s where Hadoop stands now, what can we do to make sure it finds its way up the Slope of Enlightenment and into the coveted Plateau of Productivity? Here are a few tips:
- Focus on Data, not on the Tools: This may seem obvious, but I regularly see data scientists obsess over and pour their time into understanding the tools at their disposal. This over-focus on tools – a desire from data scientists for their own playground – can come at the expense of focus on the data itself and why it matters. Data scientists must also involve business stakeholders to ensure they’re focused on the data that moves the needle.
- Don’t Be Afraid of Incremental Improvements: So often we want to bite off more than we can chew and opt for a “boil the ocean” approach to improvement. But the reality is that this approach is hard to sell to internal stakeholders and extremely risky. Instead, focus on realistic, bite-sized goals. Improve the performance of existing predictive models by supplying them with fresher data and newer kinds of data like behavioral data to add context. You’ll see real, credible improvements, and that’s progress that will grow your company’s confidence over time.
- Lead with the Use Case: This one is simple – don’t go “lake first.” Rather than building a data lake and assume that use cases will create themselves, start the process by thinking about what specific benefits you want to see in the business – and design your processes from there.
The Trough of Disillusionment is a reality for any emerging technology. Many make it out and go on to be part of the fabric of modern technology, and others fizzle out. Hadoop and big data have the staying power to be part of that first group – but it’s up to our community to take the steps necessary to help it get there.
