The Predictive Analytics Conundrum

Predictive analytics is on the cusp of widespread adoption. Many organizations are excited to make use of the power of predictive analytics (including machine learning) because they understand the value it can provide. However, although numerous organizations do use predictive analytics today, adoption remains elusive to many. In fact, TDWI research indicates that if users stuck to their plans for predictive analytics adoption, 75–80% of organizations would use the technology already although only 35–40% do so currently.

There are numerous factors contributing to this conundrum. Some of the key ones include skill development, issues with data, and the time needed to put models into production.

The top challenge cited by respondents in previous TDWI surveys regarding adoption is most often skills. Some organizations start their predictive analytics journey with one or two people in their organization who can build models. With these skills in demand in the industry, there is often turnover, leaving the organization back at square one. Other organizations have a hard time hiring the right skills in the first place. They may look to improve or expand the skills of business analysts performing less sophisticated analysis, but this can be a slow process if there is no money budgeted for training.

Data infrastructure can also stymie predictive analytics. Many organizations start by building predictive models from data in their data warehouse or data marts. However, the iterative nature of predictive analytics and the fact that organizations need to manage an ever-increasing amount of data may mean that the data warehouse is not a long-term practical solution. Additional data issues for predictive analytics include data quality and data integration. Organizations want to build predictive models using disparate data types. Some of these are “newer” kinds of data including unstructured data or real-time data from sensors, which means assembling and integrating multiple data pipelines.

Other organizations build models using statistical or machine learning approaches but struggle to put them into production, making it difficult to act on the insights and realize the benefits of the approach. Funding dries up, as does executive support. On the other hand, organizations are often trying to manage unrealistic expectations by executives. Sometimes issues arise because of a mandate to perform predictive analytics without addressing which business problems to solve first.

These issues swirling around predictive analytics can and should be overcome. Predictive analytics drives significant value. TDWI research consistently finds that organizations using advanced techniques such as predictive analytics are more likely to measure actual top- or bottom-line impact for their analytics program than those that do not.

Given predictive analytics’ value, enterprises must identify ways to address it practically. That may include utilizing some of the newer data and analytics technologies on the market today, developing model building and deployment processes, and adopting organizational best practices for moving predictive analytics forward and democratizing it in organizations.

For practical tips, see my new Best Practices Report:  Practical Predictive Analytics