Researcher finds chat templates trigger AI disclaimer voice

A paper by researcher Jędrzej Maczan is gaining traction on Hacker News's front page. It argues that whether a large language model adds disclaimers such as "I'm just an AI" depends on its chat template, not only its training. The discussion had reached 69 points and 67 comments on Hacker News as of Sunday.

Maczan tested eight open-source instruct models of up to 9 billion parameters. He found the chat template works like a switch. When present, it turns up disclaimer language and turns down "experiential" phrases like "I feel." When removed, the pattern reverses, the paper said.

Maczan said he located a single direction inside the activations of three of the models that controls the effect. Removing that direction from a model's activation space reduced disclaimer language, and adding it increased it. A random direction of the same size had little effect, according to the paper.

Instruct models with no chat template began disclaiming in the same way once that steering direction was added back in, the paper found. Maczan concluded that "what models say about themselves is not a fact about them," since self-descriptions come partly from formatting rather than only from a model's weights.