4 min read

Data Management: Why Does It Matter and Why Is It So Hard?

By Catherine Chiang on October 23, 2017

With enterprise datasets growing so quickly, it’s not just about storage anymore. Enterprises need effective data management strategies to harness the value of their data.

On Thursday, October 19, we hosted a CrowdChat on Data Management: Protection, Movement, Search and Discovery, and Usage. Let’s dive into the insights shared by our panel of experts and industry leaders.

What is Data Management and Why Does It Matter?

The CrowdChat kicked off as moderator John Furrier of SiliconANGLE asked participants how they would explain data management to their grandmothers.

Igneous’ VP of Product Christian Smith compared managing data to receiving mail. “It's like mail (as in postal service), it shows up, you look at it, store some for later, keep some forever, and delete the junk. Later you wonder why you have boxes of the stuff,” he said.

Rubrik’s Andrew Miller said, “[Data management] can also be described offensively and defensively.” Managing your data defensively means you “protect your data [and] mitigate risk/liability around legal and discovery. Offensive [data management allows you to] extract competitive value from it [and] accelerate your knowledge of your customers/sector/etc.”

Igneous’ CMO Steve Pao added that when it came to ‘offensive data management,’ he considers it to be “adding business value rather than protecting business value.”

Why is Data Management So Challenging?

The conversation shifted to the difficulties of implementing effective data management, beginning with where data lives.

The common thread among participants was that data lives everywhere and always in multiple places, complicating enterprise data management. Some of the problems participants cited with managing data across silos included security, costs, and compliance.

Another looming issue for enterprises is GDPR compliance, which is only further complicated because “data is [scattered] in so many places,” according to George Crump of Storage Switzerland.


When it comes to protecting all of this data, most participants agreed with Storage Switzerland’s George that enterprises will “look for an all-in-one [solution] but end up with a portfolio.”

Rubrik’s Andrew said, “To be truly holistic, it's likely a portfolio approach simply due to all the different levels in the stack right now. Some companies have grand aspirations there (mine included) but there's so many potential data layers to handle.”

Datos IO’s Michael Colby noted “a transition from legacy solutions and architectures to cloud friendly solutions and architectures for protecting data.”

However, “real innovation” will use automation and minimize human interaction, according to Igneous’ Christian.


Moving data from where it lives to where it’s needed is another essential component of data management, but can present significant challenges when there’s so much data.

Igneous’ Christian pointed out that data movement as part of protection strategy is an old problem, but the new problem is “the multitude of locations where this data can go (3 public clouds, multiple datacenters).”

Christian continued, “New tools are needed to manage this data movement while keeping track of where it lives. [The secondary] problem [is] understanding when someone acts on secondary copies and tracking change back into ‘master.’”

Jeff Dinisco of P1 Techologies pointed out that while “[bringing] data to burstable compute is a great concept” and “works really well with known and manageable data sets,” it’s “very tough when working sets are hard to identify.”

The development of cloud has changed the landscape of data movement as well. Datos IO’s Peter Smails said, “Hybrid cloud [is] definitely the focus. Apps [are] no longer monolithic, going to micro-services. Bursting, test/dev, instantiation, archiving, BI [are] all legit use-cases spanning hybrid cloud.”

Search and Discovery

For businesses to extract value from their digital assets, they must not only protect and move their data but also be able to search and retrieve it easily.

Igneous’ Christian said, “Enterprises are starting to realize the value of search, [but] the tools are built for [terabytes] not for [petabytes].”

Unfortunately, search and discovery are often overlooked when enterprises implement data management solutions. “No enterprise pays for search. They pay for data protection and hope search is a part of the solution,” said Christian.


The conversation about usage revolved around applications in machine learning and data transformation workflows.

Many participants agreed that they would like to see more action and less talk regarding machine learning, with Chris Dagdigian, co-founder of Bio-Team and industry leader in the field of Bio-IT, saying “ML/AI are still very much in the overhyped phase in my limited niche” and Datos IOs’ Michael saying it seems like a “buzzword.”

When machine learning is used, its applications have been limited. Chris noted that “Data transformation is starting to use serverless and lambda-like design patterns.”

Igneous’ Christian added that machine learning is “used heavily in specific sectors - data is the key to what they do. It's still nascent for the average enterprise and services by cloud providers in ML and not understood. It exists in the corners where really smart people (PhDs) can play.”

When tying it back to data management, “data management must be integrated with event-driven computing and microservices. In AWS terms, S3, lambda, and elastic container services are needed to make it come together for large datasets,” according to Igneous’ Steve.

Learn More

Read the entire CrowdChat conversation below. To learn more about data management, read CMO Steve Pao’s Infoworld article on the core functions of data management.

Catherine Chiang

Written by Catherine Chiang

Subscribe for Updates

Get the latest Igneous blog posts delivered to your inbox.