The hard problem of AI therapy

Why even 'perfect' AI therapy may be structurally doomed

Things have an end, but
feeling is infinite:
we’re changing, but it’s alright,
and everything is finite.

— David Byrne

One of the hats I wear is that of psychotherapist, so naturally, these days I’m thinking a good deal about the ways “AI” (in the narrow sense of the commercially available, ubiquitous phenomenon of LLM-based chatbots) has the power to help or hurt mental health at the scale of hundreds of millions. It’s a conversation I have almost every day with my colleagues and friends.

I’m not interested in rehashing the discourse on people using ChatGPT or Claude as ersatz therapist, or recapping the myriad stories of LLM-driven psychosis. Not because probing people’s entirely homespun ways of relating to superintelligent chatbots isn’t interesting—I’m involved in a study of exactly this—but rather, because it’s hard to make a novel point when so much has already been documented and shared.

My only aim here is to stand athwart the flow of takes and assert one very simple structural point about the use of easy LLM applications for psychotherapeutic purposes, and why I think they’re doomed not to work well at scale, no matter how popular this use case gets. Whether or not it puts me out of a job.

As a sometimes-therapist, I have every psychic and material incentive to reject the idea of the AI therapist. But it’s my duty to be honest with myself and with you, and to level. I can’t tell you with any certainty that AI therapists won’t be able to match and eventually best us in every capacity—at this point, I see no reason to doubt that AI may one day outmatch everyone at everything.

If this is the case, why should I gatekeep good medicine and hope for ineffective technology?

Of course, there are many valid reasons for a more skeptical take (and one better for my long-term career prospects, even if not necessarily for the greater good).

There’s a genuine chance that the attunement of one embodied, feeling human to another embodied, feeling human’s experience is simply unbeatable. Add to this, LLM chatbots’ sycophancy problem—the tendency to validate to the point of telling us exactly what we want to hear, when one of the fundamental tasks of therapy is to unsettle your sense of “normal”.

You could argue that ChatGPT and Claude are too domain-general, that their architecture forbids a healthy compartmentalization of therapy away from non-therapeutic activities. You could argue that the way liability is parsed in our legal system, there is no way an AI-dominated therapy ecosystem could work.

Ultimately, though, I’m not convinced that any of these are long-term barriers to mass adoption, like it or not. For all I know, fine-tuned AI therapy will have all but replaced human-mediated psychotherapy at scale by the end of this decade. But what I am quite confident of, is that however widely AI is adopted for therapy and therapy-adjacent activities, and no matter satisfied people are with it, it will run into serious problems, and perhaps even stir up new ones at scale—for structural reasons more basic than anything I named above.

The frame is the medicine

Here’s the crux of it: the main problem with AI therapy is that it’s too available. Too cheap to meter.

Let me put this in clearer terms: psychotherapy, in all its well-known guises, is something you engage in within a limited, time-bound frame. In today’s paradigm, whatever your therapist’s orientation, that tends to mean one 45- or 50-minute session a week; for the infinitesimally small minority of therapy patients in classical psychoanalysis, this can amount to 3, even 5, hours a week. And then at a much smaller scale population-wide, people in intensive outpatient and residential treatment programs may spend one or two dozen hours a week in therapy—albeit, mostly of the group variety.

I can think of other exotic cases, like some DBT therapists’ willingness to offer on-demand coaching calls during crisis situations—with the crucial exception that in these situations, therapists are holding the frame zealously, jealous of their own time and mindful of the risks of letting patients get too reliant.

So even under the most ideal of conditions, in which an LLM-based chatbot outmatches the best human therapists—attunes beautifully, offers the sense of being witnessed by a human with embodied experience, avoids sycophancy, and draws clear boundaries between therapeutic and non-therapeutic activities—there’s still a glaring, fundamental difference: that it’s functionally unlimited and unbounded.

The current consumer model for access to mass-market LLMs is an extraordinarily generous one (designed to create a sense of indispensability, probably). In the case of ChatGPT, $20/month earns you access to unlimited text queries. In other words, for all intents and purposes, people have access to an unlimited monthly volume of something approximating therapy for the price of one lunch out. To be clear, for many who previously lacked access to therapy due to financial, social, or other structural constraints, LLM-based therapy may represent a marginal improvement over the base case of no therapy at all.

But all else equal: does infinite, on-demand therapy—even assuming the highest quality per unit of therapeutic interaction—sound like a good idea to you? I can tell you, to me it does not. First of all, despite detractors’ claims to the contrary, the basic idea of therapy is not to make you dependent for life—but rather, to equip you to live more skillfully and with greater self-awareness. As integration specialists famously say of psychedelics, you can only incorporate so much insight, and practice skills so effectively, without the chance to digest what you’ve learned over time.

In other words, even in good old talk therapy, drinking from the hose without breaks for practice and introspection in a more organic context risks drowning out the chance for real change and practical insight. To my mind, this rhythm is the basic structural genius of psychotherapy as we know it—no matter the modality, no matter the diagnosis.

Suppose your therapeutic interlocutor provides such thoughtful, attuned feedback and support that you crave it, as you would a basic nutrient. Without externally imposed limits, do you think you’d have the willpower to take a few morsels, incorporate them into your self-concept, try thinking and living a bit differently for a bit, and waiting a whole week to return to the sanctum?

Or is there a possibility you might become dependent on the companionship, the 24-hour, on-demand possibility of reassurance (the exact worst thing for people whose primary coping tactic for anxious feelings is reassurance-seeking), the chance to bypass your own introspective process by outsourcing it to an algorithmic being that outclasses 99%+ of people in eloquence?

Reassurance on demand: gasoline on the fire

Suppose you’re a high-functioning person with anxiety who discovers that the chatbot can always soothe you on the spot. You ask one question—Do you think I handled that email wrong?—and feel relief. Ten minutes later: But what if she’s actually mad? Then: Can you rewrite my follow-up? Then: List ten reasons I’m not a bad person for sending it.

The model is good, so it delivers: thoughtful reframes, elegant language, a warm sense of being held. You can sleep.

And the next day, your threshold for distress—and relational ambiguity—is lower. Your brain has learned a new rule: when you feel uncertainty, outsource it immediately. Your chatbot didn’t create the anxiety; it simply became the fastest reinforcement schedule anyone has ever invented. Eventually, you start avoiding the kinds of real situations that don’t offer instant soothing. In time, three-dimensional relationships, with all their friction, begin to feel… inefficient.

In this case, what you get is an exquisite simulacrum of the therapeutic relationship, one that turns out to be directly at odds with the fundamentals of therapy as art and science.

And for however frighteningly prevalent “LLM psychosis” may be, my strong sense is that LLM-fueled neurosis, disguised as “insight”, under the guise of reassurance-seeking from a kindly, wise, always-available authority, is substantially more common.

Your therapist’s finite, mortal nature: a feature, not a bug

But another problem with unlimited therapy on tap is that your therapist’s scarcity, her finiteness, is a feature, not a bug.

At the core of the therapeutic relationship—widely acknowledged to be an indispensable condition of any therapy—is the sense that you’re voicing your experience to another experiencing being, and more precisely, another human, who’s presumably experienced aches and pains, puberty, love, loss, triumph, and failure of her own. Let’s even grant the possibility that in the “face” of a sufficiently intelligent simulacrum, this may not matter as much as we like to believe.

But here’s another existential fact about your therapist: she’s a finite being. Whether or not this consciously registers as important when you talk to her—you may be the type who wonders about her childcare arrangements or her snowboarding injury, or you may not be—on a certain level, you are doubtless aware that your therapist’s time, cognitive bandwidth, and emotional availability are scarce resources.

The fact that you pay good money to talk to her is a direct correlate of this understanding—if you didn’t see your therapist’s presence as scarce, you wouldn’t be spending your finite resources on the enterprise.

We value the finite, try to wring as much as we can out of it, work to make it count. When we find the right therapist, at long last, we may experience the relationship as precious and worthy of our commitment in direct proportion to the challenge of finding her. What is free and abundant, we take for granted.

Cheerful priests of neoliberalism tell us that because everyone has a refrigerator and antibiotics are easy to access, we have no reason to fret about inequality. Pop-New Age voices insist that scarcity is just an illusion of mindset. There may be rational virtue in these points. But the fact is, we’ve been programmed over eons of evolutionary time to think about value in zero-sum terms. In the absence of radical new psychotechnologies, this isn’t going away any time soon.

A misalignment of incentives

And here’s the final piece: companies delivering consumer-facing LLM products have no incentive to give consumers less, when more is effectively just as cheap. And you can bet they won’t elect to be socially responsible just because—and that for an overdetermined set of reasons, attempts to regulate them will fall flat (not least being: that human-dispensed therapy itself is hardly a regulated good).

Meanwhile, consumers—even if trained to believe a rate-limited therapy-tuned product is what’s likeliest to help them—are generally unlikely to volunteer to pay more for a lower-capacity product, when they could pay pennies for a passable service that’s infinite in capacity.

Maybe, as the demands on compute increase from all ends, it’ll become more expensive again, transferring the cost burden to non-economically-vital services (sorry, therapists!)—though, following the implications of Jevons paradox, I doubt this. Maybe as OpenAI and Anthropic's investors demand returns, we’ll see their pricing model go the way of Uber’s and Lyft’s—first, they become essential consumer goods, fueled by artificially subsidized, comically absurd low prices. Only later, do users—long since hooked—find themselves grudgingly assenting to fair market prices.

Maybe this will usher in the era of the AI therapist that isn’t just perfectly witty, skillful, and responsive, but also, finite and costly enough to feel significant. Or maybe at such a price point, people may just opt to do their deep, high-stakes therapeutic relating with humans again—scarce, scarred, and mortal as they are.

Thank you to Andreea Sutu, Nikhil Mulani, and Stephen Kaplan for their thoughtful feedback and editorial suggestions.

---

Josh Lipson, PhD, is a postdoctoral fellow at the Emory Center for Psychedelics and Spirituality. Subscribe to Whit·manic for more writing at these intersections.