Can an AI coach do the job of a human coach?

This article has been extracted from the journal article titled ‘Comparing artificial intelligence and human coaching goal attainment efficacy’, which was published in PLoS ONE. The purpose of this shorter article is to make the findings of research on human and AI coaches accessible to a wider audience.

On the one hand, artificial intelligence (AI) conjures up images of robots taking over the world with no regard for uniquely human qualities such as trust and empathy. On the other hand, there are interesting new applications of AI in healthcare and helping professions such as psychology and coaching. However, not much research has been done on how AI can support the fast-growing profession of coaching.

Various studies have shown that coaching can help people with, among others, performance and skills, wellbeing, coping, work attitudes, goal-directed self-regulation, improved work-life balance, self-awareness and assertiveness, increased confidence, better relationships, goal achievement, role clarity, and changing behaviours. Unfortunately, not everyone has access to a coach because of costs and the shortage of skilled coaches. 

There is a strong link between successful coaching outcomes and the relationship between coach and client, with convincing evidence that the coach-client relationship is the most significant factor in coaching success. Can this connection be replicated between a client and an AI coach? The answer is yes, to a certain extent. For example, researchers have shown that AI coaching can help college students and cancer patients cope with depression and anxiety. These findings suggest that, while AI lacks true human intelligence and emotions, positive outcomes are possible. 

One of the focal areas of coaching is supporting clients to reach their goals. This is also what distinguishes coaching from other helping professions. Understanding the efficacy of AI coaching compared to human coaching in the domain of goal attainment therefore seemed like a reasonable starting point for AI coaching research. Hence, the researchers of this study asked: How does AI coaching compare to human coaching efficacy in terms of client goal attainment? 

In this study, coaching is defined as a one-on-one structured conversation between a coach and client with the aim of facilitating sustainable change for the individual and potentially other stakeholders. Artificial intelligence is defined as the collection of technologies – such as computer vision, language processing, robotics, robotic process automation and virtual agents – that are able to mimic cognitive human functions.

“There is a strong link between successful coaching outcomes and the relationship between coach and client … Can this connection be replicated between a client and an AI coach?”

What can we currently expect from artificial intelligence?

AI has seen several false starts mostly because of exaggerated claims of ability which have led to disappointment, the withdrawal of funding and the collapse of interest in AI research and development. Now, there is a renewed interest in AI focused on specialist areas in line with current AI capabilities and it shows promise in areas such as decision-making processes. 

It is important to distinguish between three types of AI: (i) Artificial narrow intelligence (ANI), which refers to systems that can perform a specific task in a narrow context, such as a self-driving car; (ii) artificial general intelligence (AGI), which refers to systems with intelligence similar to human intelligence; and (iii) artificial super intelligence (ASI), which refers to systems that can outperform human intelligence. AGI and ASI do not currently exist. 

For the foreseeable future, AI entities will remain unconscious machines that can at best support humans in complex, specific tasks. This implies that ANI systems will be highly specialised and skilled in specific tasks and may even outperform humans in these narrow focus areas. 

In this study, the AI coach’s sole task was to interact with (i.e., coach) human clients. Of primary concern is the need for the AI coach to have social ability, demonstrate credibility and contextual awareness, and to be proactive in assisting clients. It is also important that the AI coach strives to embody the aspects that make human coaching effective – such as demonstrating trust, empathy, transparency, predictability, reliability, ability, benevolence, and integrity. To create a strong AI-human relationship, these aspects must be operationalised.

“One of the focal areas of coaching is helping clients to reach their goals. This is also what distinguishes coaching from other helping professions.

Meet Vici, the AI coach

The AI coach used in this study, Coach Vici, was a custom-developed text-based chatbot deployed on the Telegram instant messaging platform. Vici was designed to facilitate goal attainment based on goal theory. Vici had two types of text-based conversations with users. In the first type of conversation, the chatbot helped users to specify realistic goals by questioning them about the importance, feasibility and impact of their goals. Vici then helped users to commit to achievable actions. In the second type of conversation, users would check in with Vici to report on their goal and action progress, reflect on obstacles and course-correct where necessary.

“levels of coaching maturity range from a mechanistic models-based approach to a sensitive, intelligent approach where knowledge is integrated and learning is applied across domains.

How was the study conducted?

The researchers compared two equivalent longitudinal randomised control trial (RCT) studies that measured the increase in clients’ goal attainment after having received coaching for a period of 10 months. The first study involved human coaches while the replication study used an AI chatbot coach. The participants in both studies were recruited via email from a business school in the United Kingdom. 

All the participants who were coached by professional coaches had to participate in all six coaching sessions to remain part of the study. The AI coach, Coach Vici, was available 24/7 to the experimental group and they could use it as often as they wanted, but at least once a month. The participants conducted a survey over eight timepoints using an online survey platform (see Figure 1 below).

The participants’ goal attainment was measured through self-reported scores of how successful they thought they were in achieving their goals and how difficult they perceived their goals. The overall goal attainment score was calculated by multiplying the success score and the difficulty score for each goal separately and dividing the scores for the two different goals to create an average goal attainment score.

“the lowest level of coach maturity (models-based) is potentially within the ability of a well-designed narrow AI system

What did the study take into account?

• Goal theory: This study, which focused on finding out how AI coaching impacted goal attainment, is underpinned by goal theory as applied in coaching. In essence, goal theory is an approach explaining the need to establish goals as an intrinsic motivation where a relationship exists between goal difficulty, level of performance, and effort involved. Goal theory is supported by five goal-setting principles: clarity (specific and clear); challenge (sufficiently difficult); commitment (buy-in from onset); feedback (regular stock-taking on progress); and complexity (not too complex). Goal setting and attainment have been shown to have a positive effect on workplace performance, emotions and wellbeing. Certain actions can enhance goal attainment – such as writing down goals, measuring goals, specifying timeframes, and making a public commitment about the goal. Goal theory is therefore used extensively in coaching as an underlying mechanism to facilitate self-regulation.
• Coach maturity: Coaching is an unregulated industry, which implies that coaches enter the profession with various levels of training and experience. The result is that coaches practice at different levels of coach maturity. The notion of coach maturity is an important consideration given that AI has taken over the jobs of some people, suggesting that human coaches who operate at a low level of complexity may be rivalled by AI coaching. In addition, levels of coaching maturity range from a mechanistic models-based approach to a sensitive, intelligent approach where knowledge is integrated and learning is applied across domains. While AI is currently incapable of the latter, the fact that ANI can perform specific tasks on a level of human competency and beyond suggests that the lowest level of coach maturity (models-based) is potentially within the ability of a well-designed narrow AI system.

“Surprisingly, the AI coach was as effective as human coaches in helping the participants achieve their goals. This has major implications for the future of coaching.

So, how did Vici fare?

The researchers wanted to compare the performance of AI chatbot coach Vici with human coaches in terms of client goal attainment by comparing two longitudinal RCT studies. In both studies, the experimental groups who had received either human coaching (Study 1) or AI coaching (Study 2) for 10 months had significantly higher goal attainment than the control groups. 

Surprisingly, the AI coach was as effective as human coaches in helping the participants achieve their goals. This has major implications for the future of coaching.

Goal theory was used to explain the outcome: Goal theory says there is a higher level of goal attainment when goals are clarified, buy-in is obtained from the start, and regular feedback is provided on progress. Also, the goals must be sufficiently challenging and not too complex. Practically, this translates to writing down goals, specifying timeframes, measuring progress, and making a commitment to someone about completing the goal.

“Goal theory says there is a higher level of goal attainment when goals are clarified, buy-in is obtained from the start, and regular feedback is provided on progress.

AI coach Vici nudged the participants to write down their goals and asked questions to test how feasible and realistic these goals were. Vici even helped the participants to create action plans to reach their goals. The AI coach was programmed to always enquire about goal progress and to keep a record for reference to share with the participants.

Here, human coaches would typically be able to engage in a more complex and nuanced discussion about goals. At the same time, human coaches could decide which aspects of goal theory they implemented in each session. 

“In short, the rigour and mechanistic execution of goal theory by the AI coach and its inability to deviate from a set process (which could in fact detract human coaches) compensated for its lack of human intelligence.

Hence, one would expect the human coaches to outperform the AI coach who employed a scripted conversation. But this was not the case. Although the AI coach lacked nuanced intelligence, it had an advantage over the human coaches due to the rigorous and consistent way it executed goal theory. Also, some of the coaches may have not been well versed in goal theory or may have forgotten to ask about goal progress at some sessions.

In short, the rigour and mechanistic execution of goal theory by the AI coach and its inability to deviate from a set process (which could in fact detract human coaches) compensated for its lack of human intelligence.

“This study has shown that AI coaching … is an affordable and scalable alternative to certain aspects of human coaching. AI coaching can therefore make coaching accessible to more people.

These findings have three significant implications

• AI coaching could be scaled to democratise coaching: Various coaching efficacy meta-studies have shown that coaching can help people develop, grow and achieve their goals. However, coaching is costly and coaches are scarce, especially in low-income areas. In the corporate environment, individual coaching is usually reserved for managers and senior leaders. This study has shown that AI coaching, when implemented with a specific focus in line with the current capabilities of narrow AI, is an affordable and scalable alternative to certain aspects of human coaching. AI coaching can therefore make coaching accessible to more people.
• AI coaching could grow the demand for human coaching: There are two sides to this debate. Many people are concerned that AI – and AI coaches such as Vici – can threaten their job security. On the other hand, if AI can help democratise coaching, more first-time users of coaching services would be exposed to the benefits of coaching. Due to the limited abilities of AI, at some point users of AI coaching services may need more advanced and intelligent human coaching. The researchers believe that increased awareness of and exposure to coaching through AI could create more opportunities for human coaches. Human coaches should therefore view AI coaching as an opportunity, not a threat.
• AI could replace human coaches who use simplistic, model-based coaching approaches: This study produced a warning: Coaches who operate at a low level of coach maturity could be replaced by AI coaches. This means human coaches need to assess their coach maturity and upskill themselves in order to ensure that their coaching services remain relevant and meaningful. For now, human coaches will outperform AI coaches in terms of contextual awareness, transference of learning, and higher order complex sense-making. However, coaches need to make sure that they embody these uniquely human forms of intelligence in their coaching praxis.

“increased awareness of and exposure to coaching through AI could create more opportunities for human coaches. Human coaches should therefore view AI coaching as an opportunity, not a threat.

Human characteristics such as emotional intelligence and empathy allow human coaches to build rapport with their clients. AI coaches cannot do this at present. But the gap is closing. This study showed that AI coaches that focus on a narrow aspect of coaching and are based on fundamental theories may well rival human coaches. These specific applications of coaching could scale and democratise coaching and make its benefits accessible to a much wider audience. At the same time, AI coaching can potentially grow the demand for human coaches through exposing more people to the advantages of coaching.

• Find the original article here: Terblanche, N., Molyn, J., De Haan, E., & Nilsson, V.O. (2022). Comparing artificial intelligence and human coaching goal attainment efficacy. PLoS ONE, 17(6), e0270255. https://doi.org/ 10.1371/journal.pone.0270255
• Dr Nicky Terblanche is head of Stellenbosch Business School’s MPhil Management Coaching. His areas of specialisation are Management Coaching and Information Systems.
• Based on these research findings, a spin-out company, AI Coaching Pty, was created that now offers Coach Vici to organisations to help democratise coaching.

“This study produced a warning: Coaches who operate at a low level of coach maturity could be replaced by AI coaches.