Addressing the three most significant challenges to maximize career satisfaction and business impact

Data science, AI, and ML have been overhyped for at least the last decade, resulting in often unrealistic and misaligned expectations between data scientists and employers.

The above has been one of the fundamental drivers behind the often-cited phenomenon that ~85% of analytics and AI projects fail. Data scientists want to work on meaningful, intellectually satisfying, and challenging projects, hoping to finally capitalize on that hard-earned Ph.D. or Masters in a relevant domain. At the same time, companies often expect data science to be the panacea for foundational business problems, BHAG-type transformational challenges, or both. It often results in otherwise highly skilled and motivated data scientists, managers, and executives experiencing three primary obstacles early in their careers: 

The mismatch between aspirations and reality: 

Data scientists are doing lower-level and often rote work, more akin to what a data analyst or specialist tends to do. It tends to be the case at companies with a relatively low capability in managing their data assets and platforms, relegating data scientists to play the dual role of data engineer and BI specialist.  

Example: VP of Sales routinely asks the Data Science COE for an Excel pivot table that summarizes Year-over-Year sales and gross profit results, overlayed with competitive price indexes. It is helpful information, but the organization would be better served if the request is routed to a data analyst or BI team.

Focusing on solutions, not problems:

Data scientists are unable to drive measurable value for their companies. More junior data scientists or those in the early part of their careers may focus more on model sophistication or solution accuracy instead of understanding and solving real problems that their stakeholders care about.  

Example: VP of Marketing asks the Advanced Analytics team how competitive price changes and discounting practices impact the demand for the company’s anchor products. The analysis will be a key, albeit high-level, input to a price and promotional planning meeting that is taking place in three weeks. The data science team spends 2.6 weeks creating complex RNN models, using LIME for model interpretability, and developing a robust scenario analysis tool. They show the completed work to the VP of Marketing two days before the promotional planning session, which is left scratching her head after the meeting on how exactly to use this work.  

The CXO “AI/ML fallacy”:

Data scientists sometimes cannot deliver on unreasonably high and ill-defined expectations by senior leadership. The same senior leaders may not yet be ML-literate, or they overestimate the current boundaries of AI/ML and underestimate the human factor needed for enterprise-wide AI/solutions to add value. 

Industry buzz about AI and the Fourth Industrial Revolution, along with a proliferation of white papers and vendor solutions, have created welcome excitement about the power of analytics and data science amongst CEOs and CXOs. However, it has also resulted in misplaced corporate expectations about just what exactly AI/ML can do for their companies.  

Example: a data scientist is tasked with building a semi-autonomous digital control tower for a multi-billion-dollar enterprise that will optimize all pricing, procurement, and supply chain decisions with minimal human involvement. 

While possible in theory, it is nearly impossible to successfully execute as a holistic, integrated solution platform unless you are a company whose core product is AI/ML in the truest sense (as opposed to “AI/ML” companies that do stats). 

Over the years, I have seen, experienced, and led more analytics project failures than successes, which is the reality for most seasoned analytics practitioners. Experienced data scientists have learned by now that the challenges to productionalizing models, driving user adoption of analytics products, or driving measurable business impact are rarely solved through more robust technical or algorithmic capabilities. Solid data science foundations are a must-have, but traditionally softer skills like business operator empathy, stakeholder management, and being problem-oriented and results-focused are more critical success factors to succeeding in the analytics game in the long run. 

In a subsequent article, we will dive deeper into each of the three realities above, understand why these problems exist, and give specific and actionable career advice to aspiring or nascent data science and analytics practitioners on how best to navigate them for a meaningful and impactful career.