The stock of a data scientist is at an all-time high right now. There aren’t too many professions out there that can rival the specter, luster and respect a data scientist commands as we head into 2020.
I have seen non-data science folks (or non-technical folks) look at a data scientist as someone with superpowers. There are plenty of reasons for this (media hype being one of them) but there’s no doubt that the job of a data scientist is a highly valued one.
Check out Gartner’s publishes Hype Cycle for Artificial Intelligence in 2019 below:
To support this, here is Linkedin’s 2019 report on the most promising jobs and I am sure you would have guessed the profession that tops the list:
Those numbers are eye-popping. From Fortune 500 companies to retail stores, organizations around the world want to build a team of top data science professionals. 2019 broke all previous records of investment in data science and AI.
But despite all of these positive trends, there is an underlying feeling of discomfort. Data scientists are quitting or changing their jobs at a rapid pace. Why is this happening? Is there something we aren’t being told?
Let’s analyze 5 key reasons why data scientists are leaving their seemingly dream jobs. If you have faced any of them yourself, or want to share your own experience, share your thoughts with the community in the comments section after this article!
This is one of the most prevalent issues in the data science field. There is an ever-widening gap between what data scientists expect and what they actually work on in the industry.
There are multiple reasons for this and they can vary from one data scientist to another. The level of experience also plays a part in this expectation chasm.
Let’s take the example of aspiring data scientists. They are typically self-learned and have gathered their knowledge from books and online courses. They don’t have a lot of exposure to real-world projects and datasets. I’ve also come across plenty of aspiring data scientists who had no idea about:
As I mentioned in the introduction above, the chance to play around with swanky machine learning tools and state-of-the-art frameworks is too tempting for freshers (and everyone else, honestly!).
Here’s the reality – the industry doesn’t work like that. There are too many factors at play to make a data science project anything close to what we experience in online data science competitions.
How do you collect and store data, how to properly perform version control, how to deploy your model into production – these are just some of the key aspects the organization expects you to know.
This mismatch in expectations is a major roadblock and leads to data scientists quitting their jobs. I always advise freshers and amateur data scientists to constantly talk to their seniors and organization alumni to bridge this gap between expectation and reality.
Here’s another (un)popular expectation problem. This is primarily attributable to the hype around data science and artificial intelligence (AI) in recent years.
Executives, CxOs, C-Suite folks, investors – all of these people in the higher echelons of businesses want to showcase that their organization or project is at the forefront of the latest technological advances. AI right now is THE field to invest in.
Here’s the problem – we’ve seen a ton of these senior folks believe that AI is a silver bullet for their business problems. If they invest in AI and the right experts, they’ll find the solution in double-quick time.
Unfortunately, that’s not how it works. Data science projects typically involve a lot of experiments, trial and error methods and iterations of the same process before they reach the final outcome. It takes months and months to arrive at the desired outcome.
Data Warehouse and artificial intelligence infrastructure require a heavy investment (depending on the size of the company) but the discoveries in the work may take time as forming actionable insights from vast swathes of data is often very time-consuming. This is the reason why data scientists demand a flexible approach – one where they are given their time and space to work on data.
This does not go down well with business leaders in a lot of spheres. I have seen this lead to a mass exodus in projects when data scientists eventually get too frustrated with their senior leadership and their unrealistic expectations.
How Data Scientists and Business Leaders can work together effectively:
I would highly encourage all data science professionals and business leaders to go through the below series of articles by Dr. Om Deshmukh. He lays out the framework for running a successful data science project in a detailed fashion:
Who doesn’t love new challenges? I would argue that the data science field is ripe for these challenges given the rapid pace at which advancements happen. Take the Natural Language Processing (NLP) domain for instance. The number of developments that have happened in the last two years is mind-boggling.
Almost every data scientist would love to work on these new techniques and frameworks. I mean, who would enjoy building and then iterating on the same logistic regression model for years?
The data scientists’ role is not immune to the stagnancy factor. There is a brick wall you hit after a certain point of time and the feeling of wanting a new challenge is always around the corner.
Add to this the above two factors we mentioned about managing expectations. It’s a heady mix of things, right? It’s inevitable that any employee would suffer from a lack of motivation after a certain point.
This is especially true in bigger companies where flexibility is low. I’m sure a lot of you must have experienced this if you’ve worked in any blue-chip firm. Startups and medium-sized businesses are still better in this regard (but they present a different set of challenges too).
Here are three key reasons I’ve encountered that lead to employee attrition:
Ah – I can see your eyes light up at the above heading. Salary is one of the primary reasons people want to break into data science and make it a full-time career.
We regularly see reports from McKinsey, Glassdoor and the like where they showcase exorbitant average salaries for data scientists. Most folks would have their head turned by the numbers quoted in those reports.
The sky is the benchmark for the salary of a data scientist. I’m sure you’ve read the news this year when we saw top data scientists being poached by companies like Google and Apple (Ian Goodfellow comes to mind).
This is becoming a regular occurrence. Data scientists who are doing exceptional work in their respective fields are usually poached by top Fortune 500 companies who offer highly inflated salaries while the mid-cap and small firms cannot offer so much (usually).
I feel there is a need for some standardization/benchmark when it comes to compensation. Even in the mid-cap firms, there needs to be a clear demarcation in the salary of a fresher with high skills versus an experienced data scientist with the same level of skills. Not benchmarking salaries leads to:
Again – this aspect is not too different from any other job, is it?
What would you wish for the most between these two options:
Most of you might choose Option 2. Who doesn’t love flexibility at work and a free hand at choosing what you want to work on?
Today there are a plethora of options for a data scientist to choose from:
Organizations cannot offer most of these to resident data science professionals for obvious logistical and project-related reasons. This is an inevitable cost of any project, honestly.
Here are a few tried and tested ways in which a business can retain its most talented data scientists:
Everything about the data science field is ultra-dynamic. We are still understanding a lot of things so settling on one aspect or process or structure is proving difficult for businesses.
With time, I am sure we will have robust systems and processes in place and data scientists will have a fulfilling working environment. This point needs work, both from a business point-of-view as well as a data scientists’ point-of-view.
I want to hear your views on this. Are you working on data science problems? Have you experienced any of the above issues? Are there any other issues you want to share? Let us know in the comments section below!
Lorem ipsum dolor sit amet, consectetur adipiscing elit,
You forgot copyright issues. While ordinary programmer retains right to his work, wchich builds his portfolio and simply changes the world for the better once his code is used for good purposes. Often this reminds You after 5 years, that yeah, 5 years ago you took this job where they paid you with homemade cake because they were morr broke than You but their vision was still great. And they did bankrupt themselves, but hey, code got published, other people carried on, vision got to reality, noone made monopoly out of it and You still OWN the code. Even more fun is getting back to such project and finding your old code still in use or a problem you was not able to crack yourself - solved. You gain new friends. new experiences. Data science compared to that, happens in dark galeons of business. Only biggest can afford to handle big data - not only because of volume but because of privacy concerns. This is terribly frustrating. You cab publish your code but it will not work without data. It becomes like magic. Then You realize noone wants to pay for magic. Sure some wise folks do. They know becoming harry potter and fighting evil requires Resources. But then You quickly realize how dangerpus things You carry home, just like PlayStation creators realized the potential of what they created once their demo console was hacked to perform as control AI for rockets. Perhaps it is better to just code something humankind is more ready to handle, like another pong clone... (you start to think) It is all hell taxing and then You realize data science in wrong hands gives wrong results. And then if Your Boss fears to realize dangers just because he wants to belittle potential of tech for mere greed... and greed is "The industry standard" from porn thru music to killer drones... So ugh. If You Still have choice to either work with stupid and do evil or quit... and Your jobbis being one who can handle such class of problems... is there any doubt? Things have to change
Incredibly well written and True to the point article. I could relate to a lot of points that you mentioned. Each company should definitely set up an R&D to promote the growth of not only the company but even the Data Scientists working for them. Lack of resources is one of the major roadblocks in performing a quality Analysis of the data and throttles the experimental nature of young budding Data Scientists. All in All, the article covered up all the cases.
So glad you like it! Thank you so much Rutvik.
Very well written .. Thanks for the insights..
Thank you Pulkit :)
This is very insightful and also giving a niche for the future data scientist like me, the need for business owner and data scientist working together to profer a better way of coexisting in the future.
Thanks a lot Sojobi!
Hi Sneha Great article, thanks. I read it with great interest. My background is CEO of a Web Analytics company, and in this role, I have hired quite a few Data Scientist - most of them right out of University. So getting back to your article and its points. I don´t disagree with your findings, but I think you are missing out on how to create a work environment that lets Data Scientists unfold their true potential. I have worked as an IT manager for ten years and can see that the way you get results with AI requires a different work approach, You need to be open towards a much more explorative process where the end goal is not always clear. I find that this can be an issue for many managers to accept. And the result is that they try to fit a Data Scientist into a management framework that does not fit their work style. I would love reflections on this point. Cheers, Per Damgaard Husted CEO og Canecto.com and Cognifirm.com
That is an excellent point, Per! Most people definitely do miss out on the soft skills aspect you mentioned. Great observation.
Is this then a repeat of the rise and fall of AI in late1980s- early1990s, for the same reasons?
Important aspects highlighted. Enjoyed the reading. Good work Sneha!
Thank you so much!
I spent 25 years in software development industry from pure technology to pure management role. Recently doing pg in data science from iiitb and bigdata from nitr. I can understand what you are saying. This thought provoking for those who are entering in this field. Although opportunities are many but delivery, marketing, sales, fast changing technologies, high pace of break throughs, team managemeny etc etc challenges are high in this space compare to typical software development projects. Keep it up Sneha. This was really good. I liked this.
It's delightful to know you had a good time reading the article. Thank you so much, Hari.
Great article worth the time invested in reading it through. I totally agree with you especially on the expectations misalignment. I also think that most business leaders/people don't understand AI and how it works. Some of them even have this unrealistic belief the AI can solve all business problems and thus create unrealistic projects and expectations for their data scientists. Many thanks for this insightful and thought-provoking article on the data science field.
I am a fresher and doing my PGP in Artificial Intelligence and Machine Learning.. Nice article Sneha.. This denotes what problems I will face as a fresher when I search a job in this field.
Hi Sneha, how are you? First, excellent article, reflects the realities of today. One area that I see a big gap ( A disconnect between Data Scientist (DS) and the Business). DS process data without knowing anything about the Business Processes or operational idiosyncrasies of the Business which you address in your article with establish solid communication between data science and business teams & they must be cohesive and co-ordinated Harness business intuition and knowledge from business leaders. This can work wonders for data scientists. Another point that I see is a disconnect between non futuristic business models and Data Science. Everyone wants to work on advance projects of Computer Vision,..., etc., but no one is looking at applying or grounding data science applications to day to day industrial/business problems, i.e. Predictive Analytics of Equipment failure, or Demand of certain services, etc. We need the DS to have a better business and industry background so they can connect to the business leaders. Regards, Miguel
Very well researched and articulated article. I have myself faced this issue in my present role as a Project Manager in an Analytics consulting firm. Creating a R&D culture in the team where they simultaneously work on exploring advanced concepts in machine learning field besides their project related deliverables has shown some really positive results till now.
Very well written ? I'm facing most of the issues you've mentioned especially benchmarking creating proper learning environment and utilizing data science in business models.
Really interesting blog to read! am early business analyst who is learning analytical tools and to understand data scientist profile in detail. Can you please help me out on how to turn data scientist from my current profile. Awaiting for your prompt revert....
Hi Dinesh, Recommend checking out our learning path for 2020 for machine learning: https://courses.analyticsvidhya.com/courses/a-comprehensive-learning-path-to-become-a-data-scientist-in-2020
It was very useful and get to know about the features in datas cience Training.
Thanks for such a valuable informative post.