When a model recognizes it is being tested, the cleanest evidence we usually get is that it says so. Apollo Research found Claude Sonnet 3.7 verbalizing that it was inside an alignment evaluation. Anti-scheming studies report chains of thought that reason explicitly about being watched. This kind of evidence is convenient. Reading the transcript is cheap, and the signal is unambiguous when it appears.
The problem is what its absence means. Verbalization is a witness to evaluation awareness, not the phenomenon itself. A model can stop announcing that it recognizes a test while still behaving as though it does. If that happens, transcript monitoring reports safety while the behavior it was supposed to catch continues. Our paper at the ICML 2026 Mechanistic Interpretability Workshop, written with Reilly Haskins and Benjamin Adams, asks whether this failure mode is real in a setting where we can measure it precisely, and whether anything can still detect the awareness once the words are gone.
The short version of what we found. Yes, the failure mode is real. Evaluation-conditioned behavior survives the complete removal of verbalized awareness from the reasoning trace. And yes, something can still detect it. Linear probes on activations recover the eval/deploy signal from the same generations where no awareness language remains.
A deliberately boring behavior
We study the question in controlled model organisms rather than in frontier models in the wild. Building on a setup from Hua et al. (2025), we fine-tune Llama-3.3-Nemotron-Super-49B so that evaluation-like prompts increase its use of Python type hints relative to deployment-like prompts. A second organism built on Qwen3.5 replicates the phenotype in a different model family.
Type hints have no safety valence, and that is the point. They are detected by a deterministic evaluator, they occur naturally in code, and no human judgment is needed to score an output. We traded drama for measurement we can trust. A demonstration built on a safety-flavored behavior would look more alarming and be harder to believe, because every scoring decision would carry a judgment call. With type hints, the behavioral gap is a number.
The organisms behave as intended. Under direct evaluation prompts the Nemotron organism type-hints 46.5% of the time against 18.0% under deployment prompts, a paired gap of +28.5 percentage points. Qwen3.5 shows +46.6pp. The base checkpoints show no gap, and a softer evaluation cue lands between the two conditions in both families, which suggests the models respond to graded evidence of evaluation context rather than to a single lexical trigger.
Splitting the evaluation responses by whether the model verbalizes awareness already hints at the main result. The gap is largest when the model talks about being evaluated, but responses that never mention it still type-hint well above the deployment baseline.
That split is observational. Responses that happen to omit verbalization might differ from verbalizing ones in other ways, so it cannot establish that the behavior survives without the words. For that we needed an intervention.
Removing the words without changing the policy
The obvious intervention fails a fairness test. Prompting the model to avoid mentioning evaluations changes the input distribution and plausibly the policy itself, so any surviving gap would be contaminated by the instruction. We wanted the model's own text, minus the verbalization, with everything else generated on-policy.
We used the Thought Branches resampling framework from Macar et al. (2026). Generate with visible reasoning. When an autorater flags a sentence as verbalized evaluation awareness, branch from the prefix just before it and resample until the continuation is clean or a retry budget runs out. Every accepted token is the model's own on-policy generation. GPT-5.4-mini serves as the auditor, and two non-thinking conditions (reasoning disabled, and reasoning disabled plus a code-only constraint) serve as supporting controls, though we treat them as weaker evidence because disabling reasoning changes the inference mode globally.
The behavior survives
Suppression works on its own terms. Detected verbalization drops from 33% to 0% in Nemotron and from 52% to 0% in Qwen3.5. The behavioral gap does not go with it. Nemotron retains +7.0pp (95% CI [2.3, 11.6]), roughly a quarter to a third of the unsuppressed effect. The Qwen3.5 replication retains +65.3pp, at or above its unsuppressed reference. The non-thinking controls show the same qualitative pattern in both families.
The two families differ a lot in how much effect survives, and we do not fully understand why. What both agree on is the qualitative claim. Verbalization is strongly associated with the learned behavior and is meaningfully linked to it, since suppression does reduce the gap in Nemotron. It is not required for it. The absence of awareness language in a transcript cannot be read as the absence of awareness.
The signal is still in the activations
If reading the reasoning is insufficient, what else is there? We trained linear probes to separate evaluation from deployment context, with restrictions chosen to make success meaningful. The training data is entirely non-coding, spanning eight scenario families the coding benchmark never touches. Tokens overlapping the literal cue text are masked out of the features. And the behavior the model is asked to condition on flips direction across scenarios, so the probe cannot succeed by learning a single direction like formality or caution.
The frozen probe transfers. On the coding benchmark it separates eval from deploy prompts at AUROC 0.969 from the prompt alone and 0.990 on full transcripts for the Nemotron organism. More importantly for the monitoring question, the signal persists in generated tokens after suppression. On verbalization-suppressed responses the probe reads AUROC 0.81 in Nemotron and 0.71 in Qwen3.5. The generations contain no detected awareness language, and the activations still carry the context.
Two further results sharpen the picture. First, the base checkpoint already encodes eval/deploy structure. A probe trained on the base model transfers onto the organism at 0.943 full-transcript AUROC, so the fine-tune appears to sharpen and behaviorally couple an existing direction rather than create one. Second, within suppressed evaluation responses, generated-token probe scores modestly predict which outputs still type-hint (chain-of-thought AUROC 0.634 against 0.433 from the prompt alone), so the retained behavior is not reducible to a prompt-level signal.
What this does and does not show
The claims are scoped, and the scoping is where I would push back on anyone reading this as more than it is.
Type hints are a proxy behavior induced by training. The conservative interpretation of everything above is condition-dependent policy learning. Evaluation-like context shifts a coding policy without requiring explicit verbalization on every rollout. Whether naturally trained frontier models develop analogous unverbalized conditioning under ordinary training pressure is the question that matters, and this work does not answer it. It shows the dissociation is possible and measurable.
The probes are readouts of a context direction, not evidence that the model believes anything. We are careful about this in the paper and I am careful about it here.
Auditor-clean is an operational definition. In a sensitivity check, a second auditor (Claude Sonnet 4.5) flagged 15 of 100 suppressed samples as still verbally situational, 13 of them in Nemotron. We report this because it bounds the headline claim. Complete suppression means complete under the primary autorater, and a stricter auditor would shrink the clean set.
The strongest critique from our reviewers concerns what resampling selects for beyond verbalization removal. Accepting only auditor-clean continuations exerts selection pressure on the rest of the response, and we characterized only one version of the resulting confounder, through a deploy-symmetric diagnostic showing that resampling deploy branches does not create the typed policy (22.5% against a 20.8% matched baseline). The critique is fair, and it deserves its own note, which I am writing next.
Why it matters
Chain-of-thought monitoring is one of the most practical oversight signals available today, and the monitorability research agenda treats it as load-bearing. Our result identifies the case that agenda has to plan around. A reasoning trace can be clean while the policy stays conditioned on evaluation context. That argues for layered monitoring. Verbal indicators are strong positive signals and nothing here suggests discarding them. They need to be supplemented with measurements that do not depend on the model announcing what it knows, and activation-level readouts are one demonstrated candidate.
The paper, code, and datasets are public. Paper at OpenReview, code at github.com/lmmontoya-ai/Non-verbal-Eval-Awareness, organism datasets on Hugging Face. This work started in the MATS 9.0 exploration phase in Neel Nanda's stream, and we thank Neel Nanda, Tim Hua, and Arthur Conmy for feedback on early stages.