Addressing the mismatch between data science aspirations and company realities

Reality vs. Expectations

In a previous article, I wrote about the three main structural challenges that data scientists and their organizations face when it comes to maximizing their career satisfaction and business impact:

1. A mismatch between aspirations and reality

2. Focusing on solutions, not problems

3. The CXO "AI/ML fallacy."

In this edition, we'll dive deeper into the first impediment ("aspirations vs. reality"), decompose why it exists, and make recommendations for data scientists and their companies on how best to address it.

Most innovations go through a natural hype cycle over 10-15 years, starting with a trigger, leading to a peak (of inflated expectations), dipping to a trough (of disillusionment), before normalizing into a plateau of productivity. For data science and AI, we've been at the peak for about a decade, and for most companies in non-tech industries, the AI train is still going uphill.

Organizations are investing millions of dollars into industry-disrupting analytics transformations, with about 40% pouring into digital transformation/acceleration-type consulting engagements and 60% into premium-prized human and tech capital. Analytics, data science, AI, and ML are critical items on the CXO agenda. 

Chief Analytics/Data Officers or Chief Data Scientists are increasingly getting a seat at the table for executive leadership and board conversations. Data scientists are among the highest individual contributor corporate earners and have enjoyed a 40-60% job growth over the last ten years, with a 30% annual job growth forecasted through 2026.

The data nerd in me has enjoyed seeing all the love and investment data science has received. It encourages me that data and analytics will continue to enjoy healthy levels of human and capital inflows. My altruistic side also loves reading inspiring career success stories and seeing data science education be a great equalizer for those from less advantaged socioeconomic backgrounds.

However, having occupied vital leadership roles in analytics transformations and interacted with many data science teams inside and outside my industries, I recognize a wide gap between data scientist aspirations and the realities in which companies operate, driven by three key reasons.

1. Corporate stakeholders need to learn what data science can or should do.

The Problem

If you ask the heads of Sales, Marketing, Finance, IT, and Supply Chain what data science is or how a data science team can help their functions, you often get five different answers. 

Is data science just stats, or is it Machine Learning and AI? Is it about descriptive analytics and easy-to-use dashboards? Perhaps someone who can do custom data pulls in SQL very fast?

Unless the analytics roadmap is co-created with the above CXOs and their 2nd and 3rd-level team leaders and widely proselytized throughout the organization, there will continue to be a lack of alignment that will result in frustrated parties on both ends. I.e., the VP of Sales asking a Sr. Data Scientist for something mundane or the Sr. Data Scientist delivering a machine learning model when a simple correlation matrix would have done it.

 

Advice for the data science team and business leadership

Establish the rules of engagement early on between your data science groups and functional stakeholders. Create an analytics or data science steering committee comprised of the CEO and her functional leadership team – meet monthly on project updates, critical use case prioritizations, and investment asks.

Proselytization is essential, especially in the first 12 months of the data science journey. Therefore analytics leaders need to partner with CXOs to drive the advanced analytics agenda. Set up bi-weekly or monthly informal gatherings (i.e., lunch & learns, data science meetups) to educate non-data science colleagues about data science. Also, communicate concerning ongoing data science initiatives in the company (use simple, easy-to-understand language and avoid complexity).

Ideally, partner with the executive leadership team to identify one member per essential business function as a Sr. Manager to Sr. Director-level "Analytics Translator" and 2-3 members as "citizen data scientists." The former will be the most critical link between the data science team and the business units. At the same time, the latter will ensure that descriptive, diagnostic, and basic predictive analytics requests go to Sr. Analyst to Sr. Manager level folks in the BU. These "citizen data scientists" are domain experts and have foundational Machine Learning skills. They are also skilled at self-serve data visualization and data engineering (e.g., Tableau, Power BI, Alteryx, with some basic R or Python).

Companies must first get their data infrastructure in order (primarily operational, financial, and customer data) and, wherever appropriate, create purpose-built analytics data warehouses in AWS, Azure, GCP, etc., optimized for speed and scale. 60% of data scientists still spend most of their time cleaning and migrating data from some archaic on-prem data warehouse into a more suitable infrastructure. That's considerable time and human capital resources that we can use to generate actionable business insights, do feature engineering, or pilot and deploy models.

 

2. Bifurcated Analytics Leadership Skills

The Problem

Data science unicorns are rare (those that excel at the trifecta of advanced machine learning theory and practicum, data storytelling, and stakeholder management), and so are data science leaders that exhibit similar skill sets. Executives in charge of advanced analytics or data science teams are highly bifurcated. 

They are strong businesspeople, domain experts, team managers, corporate cheerleaders, and stakeholder influencers, or they are solid technical folks with great data science foundations akin to a Chief Data Scientist. The former cohort often gets promoted internally from Finance, Marketing, Product, or IT, while the latter usually have academic or research-oriented data science backgrounds. 

It is hard to find analytics leadership talent who encompasses elements of both (business/domain + data science), yet that is precisely what your organization and data science team need to succeed and deliver measurable outcomes.

Until things mature and you have a solid data science team with 12-18 months of domain experience, your team members will need guidance on the appropriate analytics and modeling methods for specific business problems. They will need help framing the issue in a way that makes sense to them. Similarly, they will need help with translating the complex machine learning methods into simple business speak that senior executives will understand. 

Having someone with a blend of ML and business who understands the data science workflow and how to turn data into measurable customer outcomes will make the data scientist experience much more robust and the overall analytics team more effective.

 

Advice for the data science team and business leadership

Instead of hiring the data science or analytics leader at the onset,hire the first 2-3 data scientists and temporarily place them with essential business functions. It allows them to become domain experts slowly and hit the exponential curve on value add much sooner. If you are still determining whom to hire (it's okay, you are not alone), use an expert consultant for the first 6-12 months to help build up the team and establish a data science use case pipeline and operating rhythm.

Leverage your existing data science team and other in-house data analytics or engineering experts in tandem with key business stakeholders to be part of the analytics leadership interview process. Their synchronized feedback will be more meaningful and ultimately more impactful for the business versus overwhelmingly relying on executive peer and senior leadership interviews. Your data scientists and other quantitative functional leaders in the organization can quickly identify the astute corporate operator who otherwise has little to no advanced analytics acumen. And the available leaders (executive peers) and senior leadership can quickly identify the potential data science exec candidate who is overly theoretical or who places too much emphasis on the technology and algorithms versus generating measurable business outcomes.

 

 

3. Too much focus on ML techniques vs. soft skills

The Problem

Understanding ML frameworks and strong coding skills in R or Python are the baseline requirement for data scientists. Equally important is the ability to understand and frame business problems, understand how your stakeholders are making decisions, effectively manage your internal customers and produce models, insights, and recommendations that are translated to your audience. Without these soft skills, your analytics project or initiative will not make it into production, will not get used, and business decision-makers will certainly not think that you, as the data scientist, are adding any value.

 

Advice for the data science team and business leadership

Unlike most people in the analytics world, I'm an MBA-turned-coder who discovered the beautiful world of R and Python during business school. One of the most underestimated and ignored pieces of our graduate school program was almost always our "Organizational Behavior" and "Strategic Communication" classes – taught by two otherwise excellent professors and leaders in their field. Most of us wanted to take advanced finance classes, decision analytics, quantitative marketing courses, etc. 

It took 5-10 years post-business school to realize that hardly any of us used the "Black-Scholes model" or ran a conjoint analysis by hand. However, all of us would have benefitted tremendously from some more intentional and strategic stakeholder management and organizational communication acumen. 

The analogy also holds for your data science teams. Increasingly, advanced methods are being democratized (i.e., PyTorch, Tensorflow, various AutoML solutions, etc.), and this trend will continue. The data science and analytics field will need more structured thinkers, skilled data storytellers, and practical communicators laser-focused on solving significant problems and caring about driving measurable customer outcomes (i.e., results-focused).

My advice for data scientists is to start carefully honing their competencies in communications, influencing, data storytelling, domain expertise, and strategic thinking. Similarly, my advice for companies is to begin hiring data science and analytics talent who are strategic, systematic thinkers and effective communicators with a demonstrated history of driving measurable value for organizations. For more novice roles, focus on hiring talent who balance soft skills and complex technical and analytical understanding and spend the right resources to upskill them in the right competencies.