AI Reasoning Traces Leaked Real Passwords and Keys
Modern AI usually thinks before it answers, working through a problem step by step in a hidden layer called a reasoning trace. You never see it, which makes it easy to forget it exists. Researchers just gave everyone a reason to remember. A disclosed vulnerability at a frontier AI provider exposed those hidden traces, and inside them were 62 unique API keys and 33 passwords. Real credentials, sitting in the part of the system nobody was treating as sensitive. The lesson generalizes well past one provider: your AI’s private thinking is data, and data leaks.
The layer nobody secured
Most people picture an AI interaction as two things: what you type and what it says back. So that is what gets secured, if anything does. But there is a third thing in between, the model restating the task, listing what it knows, trying and discarding approaches, and that working is generated, stored, and transmitted like any other text. When credentials show up in it, they did not come from nowhere. They came from context somebody supplied, flowed through the reasoning, and settled somewhere nobody was watching. It is a textbook case of sensitive data pooling in an unexamined place.
Where your data actually goes
The useful mental correction is to widen your picture of an AI conversation from two boxes to five, and to notice that only one of them is the answer you looked at.
| What exists | Do most people secure it? |
|---|---|
| What you typed | Sometimes |
| Files and context you attached | Rarely |
| System instructions your tools inject | Almost never |
| The model’s hidden reasoning | Almost never |
| The visible answer | Usually |
This is the mirror image of prompt injection. There, hostile instructions travel into the AI hidden inside content. Here, your own sensitive material travels out hidden inside the AI’s working. Same blind spot, opposite direction.
Four habits that cover most of it
None of this needs a security team. First, never paste live credentials into an AI tool, and rotate any that have already gone in, since you cannot un-send them. Second, redact or use placeholders for client identifiers, financial details, and anything you would not email to a stranger. Third, be cautious about publicly sharing AI conversations, exports, or debugging output, because they can carry more than the visible text. Fourth, use business-tier tools with clear data handling and retention terms rather than free consumer accounts for anything sensitive, the same discipline behind enterprise AI data security.
A familiar shape of problem
It is worth keeping this in proportion. A new system quietly accumulating sensitive data somewhere nobody thought to look is not a new story, it is the story of email attachments, log files, and browser caches. In each case the answer was not to stop using the technology; it was to learn what the system actually retains and adjust habits accordingly. Do that here and the risk shrinks to a footnote. Ignore it, and you have credentials sitting in a layer you did not know existed, which is exactly the situation those researchers walked into. The same vetting instinct in our AI security buyer’s checklist applies: ask where your data goes, and do not assume the answer is only the parts you can see.
Frequently Asked Questions
What is a reasoning trace?
Modern AI models often work through a problem step by step before answering, and that intermediate working is called a reasoning trace or chain of thought. You usually do not see it. It is the model talking to itself: restating the task, listing what it knows, trying approaches. Because it is hidden from the polished answer, it is easy to forget it exists at all. But it is generated text like any other, it gets stored and transmitted like any other, and, as researchers just demonstrated, it can be exposed like any other.
What did researchers actually find?
A disclosed vulnerability in a frontier AI provider exposed hidden reasoning traces, and when those traces were examined they contained real secrets: 62 unique API keys and 33 passwords. Those credentials were not in the visible answers. They surfaced in the model’s private working, because whatever context it was given, including material a user or system pasted in, flowed through its reasoning. The finding is not that AI invented credentials. It is that the hidden layer had been quietly accumulating them, and nobody was treating it as sensitive.
Why does this matter if I just use ChatGPT normally?
The core lesson generalizes well beyond one provider: anything you put into an AI can end up in places you did not picture. Most people mentally model an AI conversation as a question and an answer, and secure the answer. But the context you supply, the documents you attach, the system instructions your tools inject, and the model’s own working all exist as data somewhere. If your team pastes credentials, client details, or contract terms into a prompt to get help, that material is now in a pipeline you do not control end to end. Treat every input as potentially retained.
What should we actually change?
Four things cover most of it. Never paste live credentials into an AI tool, and rotate any that have been. Redact or use placeholders for client identifiers, financial details, and anything you would not email to a stranger. Be careful about publicly sharing AI conversations, exports, or debugging traces, since those can carry more than the visible text. And prefer business-tier tools with clear data handling and retention terms over free consumer accounts for anything sensitive. None of that requires a security team, just a habit change.
Is this a reason to stop using AI?
No, and treating it that way would be an overcorrection. This is a familiar shape of problem: a new system quietly accumulates sensitive data in a place nobody thought to secure, until someone looks. Email attachments, log files, and browser caches all had the same moment. The response then was not abstinence, it was learning what the system actually stores and adjusting habits accordingly. Do that here, keep credentials out entirely, redact what is sensitive, and choose tools with honest data terms, and this becomes a manageable footnote.
Keep sensitive data out of your AI pipeline
We help Canadian businesses set clear rules for what goes into AI, choose tools with honest data terms, and close the gaps most teams never think to check.
Related Articles
AI That Watches Your Screen So You Stop Repeating Yourself
Malicious AI Add-Ons Were Downloaded 1.7M Times
Prompt Injection: The AI Risk You Have Not Heard Of
AI consultants with 100+ custom GPT builds and automation projects for 50+ Canadian businesses across 20+ industries. Based in Markham, Ontario. PIPEDA-compliant solutions.