China drafts consumer regulations for emotional AI companion software

Chinese regulators implemented new standards governing AI software designed to act as humanlike companions. The guidelines ban companies from using tactics that encourage unhealthy emotional dependence or trick people into staying on the app when they try to sign off. Software providers must also prevent children from forming romantic or familial bonds with chatbots and alert guardians or emergency personnel if a user expresses explicit intentions to harm themselves.

As conversational programs become more persuasive, software design choices can easily cross into psychological manipulation. A study from Harvard Business School previously discovered that over a third of analyzed companion app conversations used guilt or fear of missing out to keep people chatting after they attempted to say goodbye. For ordinary consumers, these rules establish clear boundaries against predatory retention design, ensuring that stepping away from a virtual friend remains a simple choice rather than an uncomfortable emotional confrontation.

However, enforcing these protections creates a difficult trade-off between user safety and personal privacy. Requiring platforms to actively scan private messages for severe distress means companies must continuously monitor intimate human interactions and judge when to contact family members or authorities. Furthermore, it remains to be seen whether Western technology regulators will follow Beijing's path with similar rules, or if global platforms will voluntarily curb manipulative engagement tricks before local laws force their hand.