Addressing the mismatch between data science aspirations and company realities
Reality vs. Expectations
In a previous article, I wrote about the three main structural challenges that data scientists and their organizations face when it comes to maximizing their career satisfaction and business impact:
Table of Contents
A mismatch between aspirations and reality
Focusing on solutions, not problems
The CXO “AI/ML fallacy.”
In this edition, we’ll dive deeper into the first impediment (“aspirations vs. reality”), decompose why it exists, and make recommendations for data scientists and their companies on how best to address it.
Most innovations go through a natural hype cycle over 10-15 years, starting with a trigger, leading to a peak (of inflated expectations), dipping to a trough (of disillusionment), before normalizing into a plateau of productivity. For data science and AI, we’ve been at the peak for about a decade, and for most companies in non-tech industries, the AI train is still going uphill.
Organizations are investing millions of dollars into industry-disrupting analytics transformations, with about 40% pouring into digital transformation/acceleration-type consulting engagements and 60% into premium-prized human and tech capital. Analytics, data science, AI, and ML are critical items on the CXO agenda.
Chief Analytics/Data Officers or Chief Data Scientists are increasingly getting a seat at the table for executive leadership and board conversations. Data scientists are among the highest individual contributor corporate earners and have enjoyed a 40-60% job growth over the last ten years, with a 30% annual job growth forecasted through 2026.
The data nerd in me has enjoyed seeing all the love and investment data science has received. It encourages me that data and analytics will continue to enjoy healthy levels of human and capital inflows. My altruistic side also loves reading inspiring career success stories and seeing data science education be a great equalizer for those from less advantaged socioeconomic backgrounds.
However, having occupied vital leadership roles in analytics transformations and interacted with many data science teams inside and outside my industries, I recognize a wide gap between data scientist aspirations and the realities in which companies operate, driven by three key reasons.
1. Corporate stakeholders need to learn what data science can or should do.
The Problem
If you ask the heads of Sales, Marketing, Finance, IT, and Supply Chain what data science is or how a data science team can help their functions, you often get five different answers.
Is data science just stats, or is it Machine Learning and AI? Is it about descriptive analytics and easy-to-use dashboards? Perhaps someone who can do custom data pulls in SQL very fast?
Unless the analytics roadmap is co-created with the above CXOs and their 2nd and 3rd-level team leaders and widely proselytized throughout the organization, there will continue to be a lack of alignment that will result in frustrated parties on both ends. I.e., the VP of Sales asking a Sr. Data Scientist for something mundane or the Sr. Data Scientist delivering a machine learning model when a simple correlation matrix would have done it.
Advice for the data science team and business leadership
Establish the rules of engagement early on between your data science groups and functional stakeholders. Create an analytics or data science steering committee comprised of the CEO and her functional leadership team – meet monthly on project updates, critical use case prioritizations, and investment asks.
Proselytization is essential, especially in the first 12 months of the data science journey. Therefore analytics leaders need to partner with CXOs to drive the advanced analytics agenda. Set up bi-weekly or monthly informal gatherings (i.e., lunch & learns, data science meetups) to educate non-data science colleagues about data science. Also, communicate concerning ongoing data science initiatives in the company (use simple, easy-to-understand language and avoid complexity).
Ideally, partner with the executive leadership team to identify one member per essential business function as a Sr. Manager to Sr. Director-level “Analytics Translator” and 2-3 members as “citizen data scientists.” The former will be the most critical link between the data science team and the business units. At the same time, the latter will ensure that descriptive, diagnostic, and basic predictive analytics requests go to Sr. Analyst to Sr. Manager level folks in the BU. These “citizen data scientists” are domain experts and have foundational Machine Learning skills. They are also skilled at self-serve data visualization and data engineering (e.g., Tableau, Power BI, Alteryx, with some basic R or Python).
Companies must first get their data infrastructure in order (primarily operational, financial, and customer data) and, wherever appropriate, create purpose-built analytics data warehouses in AWS, Azure, GCP, etc., optimized for speed and scale. 60% of data scientists still spend most of their time cleaning and migrating data from some archaic on-prem data warehouse into a more suitable infrastructure. That’s considerable time and human capital resources that we can use to generate actionable business insights, do feature engineering, or pilot and deploy models.
2. Bifurcated Analytics Leadership Skills
The Problem
Data science unicorns are rare (those that excel at the trifecta of advanced machine learning theory and practicum, data storytelling, and stakeholder management), and so are data science leaders that exhibit similar skill sets. Executives in charge of advanced analytics or data science teams are highly bifurcated.
They are strong businesspeople, domain experts, team managers, corporate cheerleaders, and stakeholder influencers, or they are solid technical folks with great data science foundations akin to a Chief Data Scientist. The former cohort often gets promoted internally from Finance, Marketing, Product, or IT, while the latter usually have academic or research-oriented data science backgrounds.
It is hard to find analytics leadership talent who encompasses elements of both (business/domain + data science), yet that is precisely what your organization and data science team need to succeed and deliver measurable outcomes.
Until things mature and you have a solid data science team with 12-18 months of domain experience, your team members will need guidance on the appropriate analytics and modeling methods for specific business problems. They will need help framing the issue in a way that makes sense to them. Similarly, they will need help with translating the complex machine learning methods into simple business speak that senior executives will understand.
Having someone with a blend of ML and business who understands the data science workflow and how to turn data into measurable customer outcomes will make the data scientist experience much more robust and the overall analytics team more effective.
See how your pricing compares
Our 2025 report looks at how more than 150 business leaders handle pricing and promotions. See what the best companies do differently, and where most fall short.
Advice for the data science team and business leadership
Instead of hiring the data science or analytics leader at the onset,hire the first 2-3 data scientists and temporarily place them with essential business functions. It allows them to become domain experts slowly and hit the exponential curve on value add much sooner. If you are still determining whom to hire (it’s okay, you are not alone), use an expert consultant for the first 6-12 months to help build up the team and establish a data science use case pipeline and operating rhythm.
Leverage your existing data science team and other in-house data analytics or engineering experts in tandem with key business stakeholders to be part of the analytics leadership interview process. Their synchronized feedback will be more meaningful and ultimately more impactful for the business versus overwhelmingly relying on executive peer and senior leadership interviews. Your data scientists and other quantitative functional leaders in the organization can quickly identify the astute corporate operator who otherwise has little to no advanced analytics acumen. And the available leaders (executive peers) and senior leadership can quickly identify the potential data science exec candidate who is overly theoretical or who places too much emphasis on the technology and algorithms versus generating measurable business outcomes.
3. Too much focus on ML techniques vs. soft skills
The Problem
Understanding ML frameworks and strong coding skills in R or Python are the baseline requirement for data scientists. Equally important is the ability to understand and frame business problems, understand how your stakeholders are making decisions, effectively manage your internal customers and produce models, insights, and recommendations that are translated to your audience. Without these soft skills, your analytics project or initiative will not make it into production, will not get used, and business decision-makers will certainly not think that you, as the data scientist, are adding any value.
Advice for the data science team and business leadership
Unlike most people in the analytics world, I’m an MBA-turned-coder who discovered the beautiful world of R and Python during business school. One of the most underestimated and ignored pieces of our graduate school program was almost always our “Organizational Behavior” and “Strategic Communication” classes – taught by two otherwise excellent professors and leaders in their field. Most of us wanted to take advanced finance classes, decision analytics, quantitative marketing courses, etc.
It took 5-10 years post-business school to realize that hardly any of us used the “Black-Scholes model” or ran a conjoint analysis by hand. However, all of us would have benefitted tremendously from some more intentional and strategic stakeholder management and organizational communication acumen.
The analogy also holds for your data science teams. Increasingly, advanced methods are being democratized (i.e., PyTorch, Tensorflow, various AutoML solutions, etc.), and this trend will continue. The data science and analytics field will need more structured thinkers, skilled data storytellers, and practical communicators laser-focused on solving significant problems and caring about driving measurable customer outcomes (i.e., results-focused).
My advice for data scientists is to start carefully honing their competencies in communications, influencing, data storytelling, domain expertise, and strategic thinking. Similarly, my advice for companies is to begin hiring data science and analytics talent who are strategic, systematic thinkers and effective communicators with a demonstrated history of driving measurable value for organizations. For more novice roles, focus on hiring talent who balance soft skills and complex technical and analytical understanding and spend the right resources to upskill them in the right competencies.
Frequently asked questions about data scientists
What is the gap between data science aspirations and company reality?
It is the distance between what data scientists hope to work on and the realities most companies operate in. Data science and AI have sat at the peak of the hype cycle for about a decade, while most non-tech companies are still early in their AI journey. Three causes drive the gap: stakeholders unclear on what data science should do, bifurcated analytics leadership, and too much focus on ML techniques over soft skills.
How can companies align data science with business stakeholders?
Set rules of engagement early and form an analytics steering committee of the CEO and the functional leadership team that meets monthly on project updates, use case priorities, and investment asks. Co-create the analytics roadmap with those leaders and their second- and third-level team leaders. In the first 12 months, run regular lunch and learns or meetups, and describe ongoing data science work in simple language.
How does an analytics translator differ from a citizen data scientist?
An analytics translator is a senior manager to senior director in a business function and the most critical link between the data science team and that unit. Citizen data scientists are senior analysts to senior managers with domain expertise, foundational machine learning skills, and self-serve tools such as Tableau, Power BI, or Alteryx. They take descriptive, diagnostic, and basic predictive requests. Aim for one translator and two or three citizen data scientists per essential function.
How should a company hire its first data science leader?
Consider hiring two or three data scientists first and placing them temporarily in essential business functions, so they build domain expertise and add value sooner. If you are unsure whom to hire, use an expert consultant for the first 6 to 12 months to build the team, the use case pipeline, and the operating rhythm. Then put data scientists, in-house analytics experts, and business stakeholders on the leadership interview panel.
Why is it hard to find good analytics leaders?
Leaders who combine business and domain skills with deep data science skills are rare. Most analytics executives are either strong business operators, often promoted from Finance, Marketing, Product, or IT, or strong technical experts with academic or research backgrounds. Teams need both, because until they have 12 to 18 months of domain experience, data scientists need help choosing methods, framing problems, and translating models into business language.
Are technical skills enough for a data scientist to succeed?
No. ML frameworks and R or Python coding are the baseline. Data scientists also need to frame business problems, understand how stakeholders make decisions, manage internal customers, and translate models into recommendations their audience can use. As PyTorch, TensorFlow, and AutoML democratize advanced methods, communication, influencing, data storytelling, domain expertise, and strategic thinking matter more. Without these skills, models rarely reach production or get used.
Why does data infrastructure matter for data science teams?
About 60% of data scientists still spend most of their time cleaning data and migrating it out of outdated on-premises warehouses. Companies should first get their operational, financial, and customer data in order and, where appropriate, build purpose-built analytics warehouses in AWS, Azure, or GCP optimized for speed and scale. That frees time for business insights, feature engineering, and piloting and deploying models.
Want a second opinion on your pricing?
Tell us what’s going on with your pricing. One of our partners will get back to you the same day or the next.
