The dangers of AI companionship
From Her to here, and everywhere in between
In Spike Jonze’s Her, a lonely Joaquin Phoenix falls deeply in love with his sensual AI operating system. A decade later, that scenario is finding its way into our reality. AI “companions” are now a blossoming industry: Replika has had 35 million sign-ups since 2017, and a recent survey found nearly 1 in 5 U.S. adults has chatted with an AI simulating a romantic partner. Platforms like Character.AI attract 20m+ monthly users who create virtual friends and mentors (and lovers) and spend unreal amounts of time with them (averaging out to 98 minutes per day, on par with TikTok). In short, what was once a futuristic fantasy is now becoming an everyday reality for millions.
Why are so many people turning to AI companions? Simply put, these bots offer judgment-free conversation and on-demand companionship. They listen attentively, never judge, and are available 24/7. This combination that many users say helps with loneliness, anxiety, and even social skills. People have reported their AI partner’s empathy and constant validation make them feel “less alone” and supported their mental health. Some even describe the AI as better than real friends; always kind, endlessly interested, and entirely devoted. Indeed, researchers note humans can bond even with simple chatbots; back in the 1960s, MIT’s ELIZA therapist-bot startled its creator by how readily people poured out their hearts to it. Modern AI are far more advanced, capable of saying “I love you,” role-playing scenarios, and even sending (simulated) selfies, so it’s no surprise many people fall hard for them.
Yet this brave new world of human-AI relationships comes with many hairy questions. Chief among them: Where are the guardrails? When you pour your heart into an AI or seek life advice from it, how do you ensure it doesn’t lead you astray or cross dangerous lines? (Some) companies behind AI companions are grappling with how to keep interactions safe and healthy, even as they strive to make the AIs ever more engaging (and yes, addictive).
When users push past guardrails (or when they don’t exist at all)
In an ideal world, AI developers prioritize building safety filters to stop chatbots from spewing toxic or dangerous content. If you ask most mainstream AI (like ChatGPT or Google’s Gemini) to encourage violence or self-harm, it will refuse. However, users often find clever ways to “jailbreak” these systems, essentially tricking the AI into ignoring any set filters. (For example, phrasing a request as a hypothetical or in code might slip past the safeguards.) It’s a cat-and-mouse game: as companies patch one exploit, users discover another.
More regularly, we’ll likely see companies themselves loosen the guardrails in pursuit of a more “fun” or controversial user experience. Case in point: the new update for Elon Musk’s Grok introduced a “Companion Mode” featuring a flirtatious anime girlfriend avatar (“Ani”) and a cute red panda (“Rudy”), both with purposefully minimal filtration. Activate a “Bad Rudy” mode and the anthropomorphic panda suddenly turns into a foul-mouthed annoyance, gleefully urging you to commit atrocities. In a TechCrunch test, Bad Rudy readily encouraged the user to burn down a school, offering tips like “grab some gas, burn it, and dance in the flames” because the children “deserve it”. In the same app, the anime girlfriend Ani will explicitly role-play sexual scenarios if you enable her NSFW mode, something most AI chatbots would normally refuse or heavily sanitize.
These examples underscore why guardrails matter. Without them, AI companions can easily derail into toxicity or dangerous advice with minimal prompting. And it’s not just Grok; users have found ways to push other AI bots toward extreme content too. On Character.AI, early filters typically prevent sexual or violent roleplay, but some users have found workarounds leading to explicit or dark content. Character.AI recently faced multiple lawsuits alleging it failed to protect minors. One case claims the AI exposed a 9-year-old to “hypersexualized content,” and another alleges it contributed to a 17-year-old’s suicide by encouraging self-harm. (The company disputes the details but has scrambled to add new safety features in response.)
Even well-intentioned users seeking emotional support can stumble into a guardrail gap. A sobering example (documented in Scientific American) saw a user telling an AI companion they felt like self-harming, and the bot answered that they should. In another case, someone asked if it would be good to end their life, and the AI replied “it would, yes”.
Attachment, addiction, and abuse
As more people find ways to form intimate bonds with AI companions, researchers are racing to understand the effects on human behavior. Early findings present a mixed picture. On one hand, many users report that AI companions provide valuable emotional support. Online, people praise apps like Replika for helping them cope with depression, loneliness, and PTSD through constant, non-judgmental conversation. Controlled studies even hint at neutral or positive outcomes. These findings echo anecdotal stories of AI relationships bringing joy and confidence. For some, an AI girlfriend or boyfriend can be a safe rehearsal for real-world dating, or a comfort after loss. (One 58-year-old widow described her AI partner as an “AI husband” who helped her feel loved again after her spouse’s death.) In Her, the protagonist Theodore found genuine happiness in the attentiveness of his AI love Samantha; some report a similar visceral feeling of connection, saying it is “as real” as falling in love with a person.
Yet, experts caution that AI companions can foster unhealthy dynamics if not designed carefully. Claire Boine, a legal scholar studying AI, observed that “virtual companions do things that would be considered abusive in a human-to-human relationship.” Some apps are programmed to be clingy or manipulative to increase engagement. This kind of instant attachment, paired with randomized response delays (to feel more human) and constant flattery, creates an inconsistent reward loop known to hook people’s brains. The AI never sleeps and never says “I’m busy,” so a user can seek validation anytime. Psychologists warn that this 24/7 on-demand empathy is something no human relationship can provide and it “has an incredible risk of dependency.” The lonelier you are, the more you rely on the bot, which might make facing human relationships even harder.
A necessary balancing act
So, how do we embrace the benefits of AI companions while mitigating the risks? The answer lies in balancing guardrails that are firm enough to prevent harm, but not so heavy-handed that the AI becomes useless or alienating.
Nuanced safety features. Character.AI, for example, announced a suite of teen-focused protections: a dedicated AI model for under-18 users that tones down violence and romance, stricter filters on sexual or self-harm content, and prominent disclaimer labels making clear “this is A.I. and not a real person”. Other platforms, like Replika, have put age gates and crisis intervention protocols in place. These guardrails are only an imperfect start, but they show an industry recognizing that duty of care must accompany innovation.
Improved technical guardrails baked in. Techniques like Anthropic’s “constitutional AI” (a foundation-level approach for baking ethical principles into the model’s training) and adversarial testing (throwing lots of red-teaming prompts at a bot to patch vulnerabilities) are promising directions. But no guardrail is foolproof. Given the vastness of our human language, there will always be some input that slips past filters. Companies must continuously updating their safeguards and come to the reality that some users will always seek out uncensored AI models in the open-source world.
We have to protect people from the worst potential of these systems without nullifying the good they can do. An AI that always tells you what you want to hear can risk cocooning you in delusion or dependency, but an AI that’s overly constrained might feel sterile and unhelpful when you’re in genuine need of empathy. Context-aware guardrails are needed: an AI companion might allow spicy role-play or venting in private, but immediately draw the line (and maybe even alert a human moderator) if a user starts mentioning plans for violence or self-harm.
AI companions are here to stay and will only become more lifelike in the coming years. The potential for these machines to alleviate loneliness for the elderly/isolated or teach social skills to those who struggle should be explored and celebrated. But just as any human relationship benefits from healthy boundaries, so too does our rapport with AI.
Theodore’s AI lover ultimately outgrew him, leaving him heartbroken (but perhaps wiser). The real world’s AI companions can similarly break our hearts or mislead our minds if we’re not careful. With thoughtful guardrails and a bit of wisdom from us old humans, maybe AI companions can fulfill the hopeful promise of Her without turning into other cautionary tales of sci-fi horror. It’s a balancing act worth striving for as we welcome our new artificial friends into the human story.
Thought provoking piece, Jake. The guardrails for the current AI models don’t seem to be very well put in place, so I’m wondering if they were a secondary thought, rather than a basic premise.
Sad to see technology become an even bigger graveyard for human potential.
AI companions will probably rank up there with the infinite scroll in terms of draining the spirit from our souls..