The impact of the pandemic on digital transformation and data access

Data access has become more critical for 53% of survey respondents throughout the pandemic as analytics workloads and demands increase significantly, according to a survey conducted by Enterprise Management Associates.

The imperative for faster data access is about driving business outcomes, with 35% of survey respondents looking to analyze real-time changes to risk and 36% wanting to improve growth and revenue generation through more intelligent customer engagements.

37% of survey respondents aren’t confident in their ability to access timely, relevant data for critical analytics and decision-making.

Data is significantly distributed and this complexity isn’t going away

The survey found that half (52%) of respondents have data in five or more data storage platforms. And more enterprises are expected to follow this trend, with 56% of respondents claiming they will have data on more than five platforms in the next year. It appears distributed data is here to stay.

Moving data across sources is riddled with challenges

Many enterprises are finding the task of building and deploying data pipelines difficult. Some of the biggest obstacles they face are combining data in motion with data at rest (32%), the excessive time it takes to address break and fix (30%), data pipeline complexity (26%) and the manual coding lift for deploying error-free data pipelines (25%).

Developing a data pipeline is time intensive

Building the right infrastructure for accessing data takes significant time and manpower, and enterprises can’t afford to wait. For 45% of survey respondents, it currently takes more than a day to develop a data pipeline, with 27% saying it can take anywhere from three days to two months. Not to mention making that data pipeline operational takes more time – 52% of respondents said that it adds another day or more, with 24% saying it adds another week to the process.

Every second between querying data and gaining insight counts

With the rapidly changing landscape of almost every industry due to the pandemic, today’s enterprises have significantly less time to gain insight on their data before it’s outdated. Viability of business decisions comes down to a matter of milliseconds (according to 17% of respondents), with 39% of business decisions requiring latency of one second or less.

“Data is the lifeblood of any business trying to navigate today’s digital economy. What we’re seeing in these survey results is that organizations have clear demands for faster and more comprehensive data access, but technical challenges still exist,” said Justin Borgman, CEO of Starburst. “It’s imperative that companies overcome these challenges because their customer experience, competitive advantage and growth depend on it. By empowering our customers to access all of their data, regardless of location, Starburst enables better and faster business decisions.”

The survey shows that enterprises are turning to specific practices and product capabilities to meet these challenges:

• Multi-cloud flexibility. Among survey respondents, 56% of their data is in the cloud and 44% is on-premises. The move to the cloud is quickly progressing, however, with respondents expecting to have 62% of their data in the cloud and 38% on-premises by the end of 2021. And this migration is not to one specific cloud provider: the number one criteria respondents looked for in cloud data storage was the flexibility to access data from multiple clouds (47%).
• Automation, search & cataloging. When asked, almost half (44%) of respondents identified automating IT and data operations as the most important practice to improve their organization’s data strategy. Survey respondents also identified implementing search (32%) and cataloguing data (30%) among the most important practices, pointing toward the necessity for organizations to make finding and using the right data a quick and easy process for users across the enterprise.
• Modern analytical capabilities. Legacy data solutions are no longer meeting the needs of the modern enterprise – data analysts need future-facing tools that enable the processes they are engaging in most. For 42% of respondents, that is running a single query across relational databases, file systems and object storage. Data analysts are also looking for the most support in analyzing streaming data or real-time events (38%) and running a single query across structured and semi-structured data (37%).

“The opportunity for every organization is to convert data into action,” said Mike Piech, VP and GM of Cloud Storage and Data Services at Red Hat. “But the challenge we see organizations facing is a veritable tsunami of diverse data in their hybrid cloud environments, with a range of users needing access to it, including application developer and DevOps teams, data science and data engineering teams, and cloud infrastructure teams. The imperative is to create a consistent foundation for storage that can handle a wider range of workload types, from structured databases and semi-structured data warehouses to unstructured data lakes. With this foundation, organizations can be better positioned to harness insights from their data, no matter where it comes from in their hybrid cloud environment.”