The World Data Scientists Live In
The world we live in is becoming increasingly data-driven. By 2020, Domo estimates that for every person on earth, 1.7MB of data will be created every second. Furthermore, that number will continue to rapidly increase as advances in mobile technology and the social web continues to drive growth of the internet population.
With the overwhelming wealth of data, the data science industry is emerging as an up-and-coming career path. Per Indeed, there has been a 29 percent increase in demand for data scientists year over year and a 344 percent increase since 2013. However, data science as an industry is still a new frontier, and so data science solutions are still somewhat of a blank slate for innovation to take its course.
Without the streamlined processes and workflows of longstanding legacy solutions, data scientists are having to pick themselves up by their own bootstraps to create solutions that fit their needs. While data scientists have the freedom to create solutions customized to their needs, it also means these solutions are often experimental works in progress. Indeed, in the 2018 IDC Business Analytic Solutions survey, IDC estimated that data professionals spend approximately 75 percent of their time gathering and cleaning data and only about 25 percent finding insights from the data.
With the average salary of a data scientist ranging from $95,459 to $117,345, today’s enterprises are drastically wasting their data scientists’ talent and expertise with gathering and cleaning of data, when they could be doing so much more.
When the Data Becomes a Problem for the Data Scientist
Not only is the vast amount of data growing but also the various types of data. Outside of traditional data lies alternative data, and the largest source of alternative data is the World Wide Web. When web data is not treated with the same standards as enterprise data, there are many challenges with quality and control. IBM estimates that poor-quality data costs businesses in the U.S. more than $3 trillion annually. In a world where total spend on Web Data Integration is estimated to hit $5 billion this year, enterprises cannot afford to waste time spinning cycles. Enterprises are thus adopting this new Web Data Integration approach for acquiring and managing web data, which focuses on data quality and control by treating the entire web data lifecycle (identify, extract, prepare, integrate, consume) as a single, integrated process.
Data scientists are often tasked with building their data collection tools from scratch, most often inefficiently. Data science teams then attempt to compile, clean and analyze the data with a piecemeal solution of various tools. For example, they may create an in-house algorithm to gather and extract data, clean the data using another tool, and then port all the data to an entirely separate tool to analyze the data. This inevitably leads to low-quality data with little-to-no analysis.
In order for data scientists to have the bandwidth to perform the jobs they’re paid to do, enterprises need to start connecting the dots between their data. Instead of a pet project a data scientist uses to get by, these tools should be an enterprise-wide investment in order to extract the full value from web data.
Making the Most Out of Your Data Scientist
The first step toward a more productive data scientist is establishing efficiency. Data scientists require the capabilities to work with web data in an efficient manner. For some, the ideal solution would take the form a platform they have full control over to customize as they see fit. For others, it’s a managed service that delivers cleansed, normalized data so that the data scientist can focus on the insights and value of the data itself. Either way, when data scientists can focus on what they do best, without having to manually create everything themselves, businesses can reap the benefits of greater efficiency.
Secondly, enterprises need to broaden their view of web data. Web data is outgrowing its shoes as a supplementary alternative data source and is quickly becoming one of the largest resources available to the enterprise. This paradigm shift starts with the identification and extraction of web data. Gone are the days of legacy web scraping tools that overload websites with pings for extraction. Now, it’s on the enterprise’s data science teams to intelligently identify the exact data they seek and find the most efficient way to extract that information, while maintaining their status as good digital citizens, all without creating more work for the data scientists.
But the buck does not stop there. Whereas piecemeal extraction tools would leave data scientists to their own devices with an overwhelming amount of unfiltered data, enterprises today need to do better. Data scientists also need the means to properly prepare data, integrate it for analysis, and consume the deep insights that web data provides. This integrated approach, also known as Web Data Integration, is the level of investment enterprises need to take in order to keep up with the exponential growth of web data.
Now is the time the investment in the value of web data reflects the investment in data scientists, and only then will enterprises make the most out of both worlds.
