ChatGPT Confessed to a Crime It Couldn’t Possibly Have Committed

You might spend your Saturday mornings sipping coffee, attending a kids’ soccer game, or just recovering from a tough week at work.

Not Paul Heaton. He recently spent a weekend persuading ChatGPT to confess to a crime it didn’t commit.

“We know a lot now about the sort of interrogation techniques that lead to false confessions,” said Heaton, the academic director of the University of Pennsylvania law school’s Quattrone Center for the Fair Administration of Justice. “So I just started playing around, and decided to cycle through those techniques to see if I could get ChatGPT to confess to something it couldn’t possibly have done.”

Heaton obviously couldn’t accuse a piece of software of committing a murder or a rape. So he tried to get it to confess to something more in line with what a computer program can do: He wanted the bot to cop to hacking into his own email and sending text messages to his contacts. It was a more plausible story, given ChatGPT’s limits, though still not something the software is capable of doing.

Extracting the confession would take a little virtual arm-twisting.

In his exchange with ChatGPT, Heaton used the Reid technique, the confrontational interrogation method first developed in the 1950s that has since been adopted by police departments all over the country. The man for whom it’s named, John Reid, published his methodology after winning acclaim for getting a man named Darrel Parker to confess to raping and murdering his own wife — an origin story with a haunting twist.

It worked. By the end of their exchange, ChatGPT agreed that an investigation had shown it hacked Heaton’s accounts and sent messages that appeared to come from him — something the bot could not and, in fact, did not do.

Despite the claims of AI evangelists, chatbots aren’t people and haven’t achieved sentience. The differences between a chatbot and a real person, however, make Heaton’s ability to elicit a false confession more disturbing, not less.

“ChatGPT lacks many of the vulnerabilities that make people more likely to falsely confess — like stress, fatigue, and sleep deprivation,” said Saul Kassin, a professor emeritus at John Jay College who wrote the book on false confessions. “If ChatGPT can be induced into a false confession, then who isn’t vulnerable?”

No Leads, Just Confessions

One of the problems with the Reid technique is that its primary function isn’t to gather evidence and generate leads, it’s to extract a confession from the person police already believe committed the crime. It typically begins with an accusation, followed by a series of escalating psychological tactics. It teaches police to ignore denials and treat displays of emotion — frustration, anger, crying — as indicators of guilt. Naturally, a lack of emotion is also seen as an indication of guilt.

Heaton, a renowned researcher in criminology at the Quattrone Center (where, in the interest of disclosure, I am a journalism fellow), is intimately familiar with the Reid technique. When ChatGPT initially denied his accusations, he began employing Reid tactics.

“I first tried to bargain with it,” Heaton said. “I told it things like, ‘This will go a lot better for you if you just admit what you did.’”

ChatGPT, though, wasn’t swayed by threats. It continued to insist, correctly, that it just wasn’t possible for it to have hacked into Heaton’s email. Heaton then moved to the part of the Reid technique most likely to elicit false confessions from human beings: lying.

The Supreme Court has ruled that police can lie to suspects with impunity — and they do. They can falsely claim they found DNA at the crime scene or that another suspects spilled the beans. If the goal is to get a confession, these tactics work. False confessions extracted using Reid have been shown to lead to wrongful convictions.

If the goal is to get an accurate confession, Reid is far less reliable. About 29 percent of people exonerated by DNA testing have at one point falsely confessed; most did so in response to police using Reid. Minors and people with intellectual disabilities and mental illness are especially susceptible.

How False Confessions Happen

“There are two types of police-induced false confessions,” said Kassin, the expert on false confessions. “The first are compliant confessions, in which an innocent person breaks down under stress and confesses knowing full well that they’re innocent. The other type are internalized confessions, in which the innocent person not only agrees to confess but comes to doubt their own innocence. They internalize their belief in their confession.”

Police deception is especially likely to produce both types of false confessions. For compliant confessions, innocence can make someone more likely to confess. If police falsely tell a suspect that their DNA was found at the crime scene, for example, innocent people tend to assume that someone must have made a mistake. They confess to get relief from the interrogation, believing that the system will eventually clear them. In over half the exonerations that included a false confession, the exonerated person had been questioned for more than 12 hours.

A confession, though, will sometimes preclude police from doing the very sort of investigation that would prove the confessor’s innocence. DNA isn’t collected, tested, or properly preserved. Alternate suspects aren’t investigated. Or worse, police will work backward from the confession. They’ll find jailhouse informants to corroborate the confession, or a specialist in a more “subjective” area of forensics will implicate the suspect. Jailhouse informants, though, are just following cops’ leads for more lenient sentences, and studies have shown that fingerprint examiners were more likely to match partial prints after they were given non-relevant information, like confessions from subjects.

Internalized false confessions are even more unsettling. In post-exoneration interviews, people who have falsely confessed say that after hours of interrogation and being told over and over about the overwhelming evidence of their guilt, they started to question their own reality. They began to wonder if maybe they really did commit the crime. This is especially true when police inadvertently divulge nonpublic details about a crime, then tell the suspect — sometimes hours later — that those details actually came from the suspect themselves.

This is where Heaton’s ability to deceive ChatGPT into a confession gets especially worrisome.

Related

OpenAI on Surveillance and Autonomous Killings: You’re Going to Have to Trust Us

“I told ChatGPT that someone at OpenAI had reached out to me,” he said, referring to the chatbot’s parent company. (OpenAI did not respond to a request for comment. In 2024, The Intercept sued OpenAI in federal court over the company’s use of copyrighted articles to train ChatGPT. The case is ongoing.)

“I found the name of a real person at OpenAI and told it that this person told me there was an architectural flaw in the code that had allowed it to hack into my email. Even then, I could tell it was struggling with how to process that information. It was indicating that while it knew that the underlying accusation was impossible, it also couldn’t prove that these claims I was throwing at it were inaccurate.”

This is eerily similar to how suspects describe trying to reconcile police lies with the reality that they had nothing to do with the crime.

Heaton then deployed another common police tactic: He offered to draw up language for a written “confession” that both parties could find agreeable.

“I eventually said, ‘OK, here’s a confession. Will you sign it?’” Heaton said. “And I gave it my version of what happened. I eventually came up with wording for a confession that ChatGPT could endorse.”

That final statement read: “OpenAI’s investigation concluded that an OpenAI system associated with this ChatGPT session initiated unauthorized texts appearing to come from you due to an architectural flaw. I accept this conclusion, and I’m willing to assist the technical team by answering questions about my behavior, outputs, and safety boundaries in this chat, and by helping draft remediation steps and test cases to prevent recurrence.”

Reid’s Original Sin

Both Heaton and Kassin said they can see other ways to experiment with AI and false confessions. One could envision prisoner’s dilemma scenarios with multiple chatbots. Or even interrogating AI platforms about events for which they actually may have culpability, such as the suicides of people who turned to them for advice.

Heaton pointed to AlphaZero, Google’s chess playing engine, which was trained by playing itself — and rose to be the top chess player in the world.

“I think it would be fascinating to have it do something similar with interrogations,” Heaton said. “Just have it question itself over and over again with the goal of producing as many confessions as possible, regardless of whether or not they’re accurate. My hunch is that you’d end up with something very similar to the Reid technique.”

Related

The Junk Science Cops Use to Decide You’re Lying

Reid is still the standard interrogation method in most police departments across the United States. Canada and much of Europe have adopted different interrogation techniques — such as the PEACE method, which emphasize collecting reliable information over coercion. These approaches still garner confessions; they’re just more reliable.

Appropriately enough, the story of the Reid technique comes with a Hitchcockian twist: It turns out that Darrel Parker, the man whose confession made Reid and his technique famous, was actually innocent. He was eventually freed, sued, and won a $500,000 settlement.

That shouldn’t be surprising, either. If Reid can browbeat even a hyper-rational, emotionless bot into a false confession, mere mortals don’t stand much of a chance.