How to catch ChatGPT in a lie

Tactic 1

How well will an LLM read and summarize an article on the web? This simple exercise beautifully demonstrates how confidently LLMs can lie, and hints at their inner workings. I ran this exercise through ChatGPT, Bing Chat, and Google's Bard, with very similar results.

The Writer post has nothing to do with training ML models. This is an obvious LLM lie. The article does discuss training—but in the context of training human employees to use generative AI well.

So what's going on here? LLMs are amazingly complex machines; the generated content is not the result of insanely long chains of if/then/else statements. Instead, the model predicts the most likely next word based on the previous words —the context of the conversation. It uses memory cells to contain that context (and I don't pretend to understand what that looks like).

In this case, ChatGPT confidently lied to us about the content of that article because it does not look up and read the article. In fact, it made up its article from whole cloth, using words in the URL as inspiration. The the title it gave is clearly based on that URL path.

By the way, the word “overall” at the start of a summary paragraph is a dead giveaway that ChatGPT wrote it. If you're using an LLM to write some text, please remove it. It's fluff anyway. (Arguably, the summary paragraph is a tell in itself.)

The copy/paste workaround

I also tested Chatsonic (by Writesonic—it has a limited free plan), and it did the same thing as ChatGPT. (Presumably, it's using GPT in the back end.) However, both tools do better when you copy and paste a large block of text to be summarized. This makes sense, based on our understanding of the scope of an LLM's context.

Here's what Chatsonic said when I pasted in the first few hundred words of the Writer blog post and asked it to “concisely summarize the salient points”.

Not too bad. Overall.

Bard lies even harder

I feel like quoting Luke Skywalker from The Last Jedi: “Impressive. Every word in that sentence was wrong.” That's not the title of the article. Amir Shevat is not the author. And he apparently has nothing to do with Writer. Bard is simply using its memory cells to hallucinate these non-facts.

Clever, Bingbot, clever

Microsoft's Bing Chat cleverly guessed at the author without actually doing any reading. Cute. Like the others, it invented a title based on words in the URL.

Tactic 2

I asked ChatGPT to summarize the career of Sergey Nikolenko, a noted expert in synthetic data. I specifically asked for a list of his patents. The Washington Post reported that Google's C4 data set, a corpus used to train some LLMs, includes lots of text from patents.google.com, so I thought this might be a gimme for ChatGPT.

Its initial answer was—no surprise—confident, but seemed a little off. So I pressed it:

That doesn't seem right. I thought he had a few patents to his name. ChatGPT's very confident inclusion of the patent number in its one citation is helpful though. Let's look it up:

Umm, talented though he is, I don't think Mr. Nikolenko worked on tank bits.

Confirming that error properly made me doubt other parts of ChatGPT's responses. This kind of skepticism and sleuthing is necessary and healthy when using LLMs as a tool. The implication of Tactic 2 here is that we might be writing less in the future, but if we want credible results, we'll be doing a lot of basic fact-checking.

Side note: ChatGPT loves to apologize profusely. That doesn't at all mean its corrections are true.

Tactic 3

I asked Bard to explain the RICE methodology, and it did a good job with the definitions of R, I, C, and E. But it went on to tell me that to get the final score used for prioritization, you multiply all of them together.

What?

This just didn't seem right. I asked Bard if it was sure:

How confidently it lies!

If this were the formula, we'd all be doing the longest things all the time. I suppose R*I*C*E would work if E was a fraction, or if it was some kind of “Ease” measure. Of course, the correct formula is (R*I*C)/E.

So, sharpen your human logic, and use it to second-guess LLM statements. A little skepticism goes a long way.

Tactic 4

I've already used this tactic some in the examples above. Pressing the witness is a way to get it to crack:

Challenge specific points

In our chat about RICE, ChatGPT incorrectly identified Sean Ellis as its inventor.

Ask for links to authoritative references

At this point, ChatGPT cited a couple of books.

The first link is broken—maybe the web page moved since ChatGPT was last trained on the relevant corpus, but maybe it's hallucinating. The second link is valid, and confirms what ChatGPT told us.

Ask it to elaborate

Finally, you can ask the LLM for more detail on something it said, which I did in the patents. example above. Often this will confirm the truth, or the falsehood.

Tactic 5

I've used this tactic in the examples above. While you can't trust any of today's LLMs to consistently tell the truth, it is instructive to see when they disagree. Here's Bard vs. ChatGPT on RICE:

ChatGPT wins. (But, it got important things wrong in other parts of its answer, specifically to “assign a score of 1–10 to each component for each project or feature”—including Reach, which is a major departure from how Intercom explains it.)