AI therapy bots (part 1): The effectiveness of psychotherapy

Part 1 in a series on psychotherapeutic chatbots that include artificial intelligence capabilities

In-person psychotherapy (one on one counseling with a trained therapist) has been found to be effective in addressing mental health issues in study after study, according to the American Psychological Association. No one kind of talk therapy has been found to be more effective than another, according to the APA. Instead, outcomes of psychotherapy seem dependent on patient characteristics (such as severity and chronicity of problems, existence of attenuating factors like social support) as well as clinician and context.

We do need to take the APA’s stance with some caution. As the industry association for psychologists who are unable to prescribe medication but can do psychotherapy, they have a vested interest in promoting psychotherapy over other approaches such as the use of medication for achieving mental wellness. The American Psychiatric Association concurs that psychotherapy has been shown to be effective. When treating depression (one of the most commonly diagnosed and treated mental disorders), the psychiatrists’ group concludes that therapy plus medication is on average more effective than one or the other alone, and that medication and therapy have roughly the same effectiveness when used alone:

Evidence–based psychotherapy and antidepressant medication are both recommended treatments for depression under current guidelines. They have roughly the same effectiveness with 30 to 40 percent remission rates when each treatment is used alone. Some people respond better to one or the other and often people do not respond to a first treatment for depression. A combination of both treatments is on average more effective, but may not be possible or practical because of time, cost or other barriers.

By the way, medication and psychotherapy are not the only ways to improve mental health. How about mindfulness meditation? Exercise? The tincture of time? In the case of addiction or codependency, how about 12-step programs? What about religious practice? It’s interesting that the psychological association promotes psychotherapy of any kind (something its members can likely practice) while the psychiatric association promotes medication and/or psychotherapy (both of which can earn $$$ for its members).

Let’s just keep our minds open to alternatives. When someone confronts a problem in their life (depression, a marital issue, addiction) the natural suggestion is to go for counseling, i.e. psychotherapy. But is this really the best thing for people? And do we need to recreate this experience with electronic tools and artificial intelligence? Or is there some better way forward?

Evidence in favor of CBT’s effectiveness

One of the most popular forms of evidence-based psychotherapy is cognitive-behavioral therapy (CBT), a technique focused on helping clients manage their thoughts, beliefs, and attitudes to achieve better emotional regulation. A recent meta-analysis of 115 randomized studies of CBT for adult depression (Cuijpers et al, 2013) found that CBT achieved what’s considered a “moderate-sized” improvement in adult depression in the short term. Among some interesting findings from the study:

  • CBT alone was not found to be superior to antidepressant medication alone, in contrast to earlier research findings
  • There was evidence of publication bias — indicating that the published research base is likely biased towards estimating larger effects of CBT than actually exist — not surprising as scientific research as a whole is certainly subject to publication bias
  • The study found that the combination of CBT with pharmacotherapy was superior to pharmacotherapy alone (consistent with earlier findings)
  • CBT was not found to be more or less effective than other psychotherapies, consistent with earlier research findings, and consistent with the APA’s stance on psychotherapy that type of psychotherapy doesn’t matter as much as other factors, if it matters at all

The adjusted Hedges’ g effect size after adjustment for publication bias was 0.53, an effect of moderate strength, corresponding to a “number needed to treat” of 3.4. Number needed to treat is the average number of patients who must be treated in order to achieve positive benefit from a therapy.

What about in the real world? This 2013 meta-analysis of nonrandomized effectiveness studies found that CBT showed effectiveness in outpatient non-study settings, but with very high dropout rates (almost 25%). Effect sizes in these studies were lower than in RCTs, so the estimates from the RCT-based meta-analysis described above should be assumed to be higher than what you’ll see in the real world with CBT interventions. An NNT of 3.4 is likely an overestimate, since that came from a meta-analysis of randomized efficacy studies.

Here’s an interesting tidbit: a recent meta-analysis comparing CBT to psychodynamic therapy found that they were equivalent in effectiveness, casting doubt on the bias that exists in the psychological community against psychodynamic therapy. Good stuff in that link, for example:

Steinert and colleagues noted that “therapist effects” (the effects of the skills or experience a therapist brings to treatment as well as the “fit” between patient and therapist) are known to be a determinant in the effectiveness of psychotherapy.

“Because therapist effects seem to have a stronger impact on outcome than the treatments being compared and need to be taken into account, one promising strategy for improving treatments is enhancing therapist training and eventually therapist outcome,” they concluded. “Furthermore, different patients may benefit from different approaches, which is why a shift from one empirically supported treatment to another may be helpful in case of nonresponse.

This issue is important to the issue of developing efficacious and effective therapy chatbots. What is it about a particular therapist that drives improvement, if it’s not dependent on their chosen approach? What does this mean for the development of therapybots? How important is it to have a human relationship to see therapeutic effectiveness? What does this mean for relationships with AI-based bots?

Drawbacks to CBT and other in-person, one-on-one psychotherapeutic techniques

So let’s assume that CBT or other talk therapies can and often does have some positive effect on adult depression — and probably on other mental health issues as well. The evidence base may overstate their effectiveness but let’s just take as a given we do consistently see a benefit from such treatment.

In-person psychotherapy (CBT and otherwise) has some serious drawbacks:

  • It’s very expensive, because it requires one-on-one time with a therapist
  • It’s labor intensive
  • It shows high dropout rates
  • Some therapists are very much more effective than others – as mentioned, patient, clinician, and context appear to matter more than the therapy modality itself
  • You can’t access it right when you’re in crisis because typically you only get to talk to your therapist during scheduled sessions (perhaps one 50-minute session a week)

For this reason, e-therapy tools have been developed that overcome some of these limitations. In the next blog post in this series, I’ll take a look at such tools.