Data science and politics: meet the # 1 reason of failure of AI projects

One of the first data science projects that I was involved in was aimed to improve the targeting process for a sales organization. When we entered the project, the sales reps were expected to choose their own target lists, while competing each other on a single pool of leads.

One of the reps, who I will call Danny, was accountable for almost 40% of the total sales (which is unbelievably high, in a team of 12 reps). It seemed that Danny knew something that the other reps simply did not. Danny was well paid for his contribution and was continuously collecting extremely high bonuses.

The management was hopping to scale the sales up, by revealing Danny’s secret sauce and use it to automatically allocate the most promising leads in some even way. The company that I was working for was hired as a contractor, to build a model for scoring leads according to their propensity to buy.

I clearly remember the kickoff meeting of that project. Danny and another rep were representing the sales team. The VP sales started his pitch by saying that there were enough good leads for everyone, and that they could make sure that all the reps, including Danny get better results. During the meeting, Danny seemed to be engaged, but when he took us out, after the meeting ended, and we were left alone with him, he smiled and advised us not to take the project, since he did not think it will succeed.

Three months later we implemented our model. It was tested, and then tested again and again and proved to produce a very good lead scoring. For a reason that I do not exactly know, the outcomes of the automated list (on terms of actual sales) were a complete failure, although we examined the produced list and it looked just fine. A couple of days after the implementation, the sales team started with an Italian strike, and very soon the VP sales let us know that although he recognized the strength of the model, they were going back to the previous method. Danny had such a strong incentive to preserve the existing situation, and such a strong impact on every aspect of operations, that no matter what we did, he could make sure it fails.

That failure kept me frustrated for a long time, but it also taught me an important lesson: it is the human intelligence that make data science projects succeed or fail. Not the artificial one.

Data science projects, as any other organizational project are intended to promote the business. We talk a lot about the risk of managing such project from a scientific or a technical perspective. However, even if we are focused on the business objectives and make sure that the models we build deliver them, we must bear in mind that AI solutions eventually affect the daily lives of humans. And different humans might have different opinions on how they want things to be done. It is called workplace politics or organizational politics, and it is the # 1 reason of failure of AI projects.

In a quick, non-scientific and non-balanced poll that I once run, most of the respondents referred to organizational politics with a strong negative sentiment. Organizational politics is often associated with deals that are made behind closed doors, personal interests that conflict with the organizational motivation and even dishonesty. However, you cannot avoid the fact that promoting meaningful endeavors within an organization require the cooperation of several (and often many) stakeholders. These stakeholders are human beings, with different set of believes, different perspectives and different opinions about the situation. We do not want everyone in an organization to think the same. An organization in which everyone thinks the same is unhealthy. Moreover, when different stakeholders in the organization have an opinion on something, it is desirable that they assert on their opinions and act to promote it. On the other hand, an organization cannot be an anarchy where everyone is acting based on his or her own interests. And an organization is not a democracy either. Healthy organizations have a structured way to balance between the ability of stakeholders to affect decision and their responsibility to commit with decisions, even if they disagree with them.

Yes, organizational politics is central in any aspect of the organization and is not unique to AI. However, when it comes to AI projects, there is another dimension of misunderstanding, emotions and misconceptions that tend to magnify the impact of organizational politics. In most of the cases there are only few stakeholders within the organization that really understand what AI is and how it can improve the business or the product. Many others may have opinions that are based of professional pride, fears, and other emotions (including for example curiosity). Emotions are powerful, but when not managed and aligned, often lead to an organizational conundrum.

Over the years, and the different organizations that I got to work with on AI, I collected the following tips that might help you use the organizational politics rather than get stopped by it.

Management commitment: in a healthy organization, management have mechanisms to align the organization with the decisions that are made. Strong managements closely listen and aware of the organizational situation and can spot objections and act carefully to resolve them. If you want to promote a serious change in an organization, you must first build management commitment.

Clear vision: management commitment is a must, but it is not enough. To make sure that management directs in the right way, you must verify that it also has a clear vision for the project. No matter how strong the management is, it cannot align the entire organization without such a vision.

Communication of the vision: the only way to align an organization towards a vision is to consistently communicate that vision and help everyone understand it.

Mapping all the stakeholders and listening to all the voices: as part of the learning process it is highly important to map every stakeholder who might be affected or might influence the AI solution. Do not narrow your discussion to decision makers. Listen to each stakeholder, be sensitive to different opinion and think how they might affect the project.

Takeaway:

organizational politics is the # 1 reason of failure of AI projects. At the same time, organizational politics is not necessarily a bad thing. It is an integral part of the organizational life. Having a strong management commitment, a clear vision that is well communicated within the organization and by listening to the voices of all the stakeholders you can navigate your projects within the organization and dramatically increase your chances to succeed.