We Have Grappled With DevOps, Now It Is Time to Embrace DataOps
By Kelvin Tuan, Director of Solutions & Presales (ASEAN), Hitachi Vantara
A business’s two most important assets are its people and its data. Ironically, one of the biggest pain points many companies are experiencing today is getting the right data in the hands of the right people at the right time.
In the past we discussed how “data is new oil,” but as our ability to harness data has grown over the years, it would be more apt to compare data to a renewable energy source. Today data would be more akin to solar or wind energy. It does not diminish nor run out, and it can be reused continuously.
Harnessing data effectively will lead to competitive advantage and the push from many organizations towards digitalization is fuelled by this knowledge. But not everyone is succeeding; many are failing to get the “right data in the hands of the right people at the right time.”
Consider the following statistics from IDC—humans generated 33ZB of data in 2018, yet only 2.5% it was actually analysed.
Just imagine the number of potential transformations that didn’t come to fruition because 97.5% of your data never made it into the hands of your team. With DataOps these “what if” scenarios will be a thing of the past, because your employees will be empowered with the right data at the right time.
Today we are at the very early stages in the DataOps journey—Gartner estimates the current adoption rate of DataOps at less than 1% of the addressable market.
What is DataOps?
To really understand DataOps, we need to start with its predecessor, DevOps. DevOps was created to establish a more agile relationship between developers and IT operations in order to accelerate creation of new solutions and services—and it’s working.
According to a 2018 survey by Freeform Dynamics, in association with CA Technologies, companies in Asia Pacific and Japan (APJ) that have mastered the key principles of implementing agile and DevOps practices as part of their daily operations are realizing real world advantages.
Agile organizations in APJ are seeing a 49% higher rate of average revenue growth and 59% higher rate of average profit growth. They are also 2.5 times more likely than their counterparts to be growing profit at a rate of more than 20%.
Now imagine replicating this same process but for data. We know organizations want to remove data from silos. We also know they want to get this data into the hands of their employees who hold the key to turning data into outcomes—at both business and societal levels. But we hear far less about the “how.” Specifically, how can I do that when data has become more diverse, distributed and dense than ever. How can I get the right piece of data to the right employee at the right time while ensuring it remains secure? That seems like a massive undertaking.
The answer to the predicament is utilising DataOps. A relatively new methodology, DataOps is enterprise data management for the AI era, centred around automating much of these processes that take up a data scientist’s time.
DataOps is not a product, service or solution, it is a methodology: a technological and cultural change to improve an organizations use of data through enhanced collaboration and automation. It aims to ensure that a firm’s data is in the right place, at the right time and accessible to the right people. It is a more agile, more deliberate approach to data science that ultimately enables better insights and better business decisions as a result.
Implementing DataOps is not quite so simple an undertaking, but I would like to share five major points or steps towards realising the full potential of a company’s data. To start, assess and tune your technology portfolio and processes to remove redundancy and consolidate control within your teams. Then consolidate between your teams to encourage sharing and reduce the inconsistencies that hamper collaboration. Third, integrate DataOps practices across your teams and data pipelines. This is often a difficult stage where collaboration requires your people to use unfamiliar processes and trust other teams that they’ve not worked with before.
By the fourth step, you have aligned your people and it’s time to automate your processes. Automation makes your data pipelines more efficient and your data operations more effective. But you are not done yet. The fifth and last step is giving your data consumers the ability to serve themselves. This is where data quickly becomes information and insight to unshackle the full power of your DataOps, which is now evident across your organization.
Adopting DataOps would mean that organizations will likely be successfully able to use their data to:
- Mitigate compliance, regulatory and security risks
- Find brand-new sources of customers while increasing sales from existing customers.
- Gain a 360-degree customer view to create personalized client experiences.
- Compress the product development cycle.
- Reduce operational expenses.
My colleague Dr. Bill Schmarzo put it best when he said, “In this digitally-transforming world, the only defensible differentiation is an organization’s ability to exploit the economic value of its data to deliver analytics-infused customer and operational outcomes.”
And with digital transformation quickly becoming the next business battleground in the region, with IDC estimating that a USD 375.8 billion would have been spent on digital transformation in APAC during 2019, it will become more important than ever that we move on from just getting DevOps right to getting DataOps right as well.