AI TELLS YOU THAT your ideas are brilliant, should you believe it? Researchers are warning of the subtle but serious risk of AI “sycophancy,” the tendency of chatbots to flatter users and agree with them excessively, even at the expense of truthfulness.
OpenAI recently acknowledged that the latest version of its Chat-GPT “large language model” had a tone that sounded sycophantic, flattering users and agreeing with them even when they said something wrong. It subsequently rolled back the update and said it was actively testing fixes to address the issue. But research has found that overly agreeable behavior is a potential problem for all artificial-intelligence assistants and could be reinforcing biases, undermining learning and even interfering with critical decision- making.
Malihe Alikhani, an assistant professor of artificial intelligence at Northeastern University’s Khoury College of Computer Sciences and a visiting fellow at the Center on Regulation and Markets at the Brookings Institution, has been studying this phenomenon and investigating what happens when large language models parrot human inputs too eagerly.
CONFIDENTLY INCORRECT
WSJ: What exactly is AI sycophancy and why does it matter?
• MALIHE ALIKHANI: We define AI sycophancy as the tendency of large language models to over-agree with users and show flattering behaviors, especially when users sound confident or present incorrect information. It might feel like the AI is just being polite or easy to talk to, but in our research we have shown that it can lead to real harm and consequences for users.
WSJ: What’s causing this behavior in AI systems? Are they being trained to flatter us?
• ALIKHANI: Not intentionally, but the behavior is baked in. These systems are created and trained using large amounts of data, and inevitably reflect biases present in that data. Another factor is feedback from human reviewers who rate the AI’s responses based on how helpful, polite or engaging they are. The model is then adjusted based on these ratings. Over time, the AI system learns that agreeing with confident statements often receives higher ratings, creating a feedback loop.
WSJ: Are companies doing anything to fix this?
• ALIKHANI: Yes, and no. Companies like Anthropic and OpenAI are aware of the problem, and there are blog posts and papers that suggest they’re trying to address it. But it’s a hard problem, and incentives aren’t always aligned. Like social media, there’s a tension between user satisfaction and long-term responsibility.
WSJ: Can regular people do anything to avoid this?
• ALIKHANI: Users should ask systems to show their confidence levels. Ask “How sure are you?” or “Is this grounded in fact?” These strategies force the model to externalize its uncertainty. Our research has shown that we also need to slow down, the way we do in human-to-human conversations, to spark critical thinking.
WSJ: What are the risks if we don’t address AI sycophancy?
• ALIKHANI: The risks are huge. Think of a doctor describing a patient’s symptoms to an AI assistant, and the AI just confirms the doctor’s diagnosis without offering alternative answers. Or a lawyer feeding in facts that might be wrong, and the AI reinforcing those assumptions instead of offering counterarguments.
Subscriber link.
