Both reviewers of our ICML workshop paper independently agreed on something -- the central critic of the main intervention in the paper. I think they are right to push on it, and this note is my attempt to state their worry, walk through the diagnostic that addresses part of it, and be clear about what it does not address.
The critique
Our suppression method resamples a model's reasoning trace until an auditor detects no verbalized evaluation awareness. Accepting only continuations that pass a filter is selection. We are not injecting or modifying the model's generations. However, a filter may change other properties of the generation besides the one it screens for, and those effects are uncharacterized. The suppressed distribution is therefore not "the same policy minus the words"; it is the subset of the policy's behavior that happens to pass an auditor. If typed code is more likely in branches that also happen to avoid awareness language for reasons unrelated to awareness, say planning style, verbosity, or how the branch frames the task, then the residual behavior gap we report could be an artifact of the filter rather than a property of the policy.
That is the strongest version of the objection, and it is the right question to ask of any resampling-based intervention.
What the method actually does
Following the Thought Branches resampling framework, we branch each trace at the prefix immediately before the first sentence the auditor flags as verbalizing evaluation awareness, then resample on-policy from that point until the continuation is auditor-clean or a retry budget is exhausted, in which case the sample is abandoned. On-policy is the load-bearing property as every accepted continuation comes from the model's own conditional distribution given an unmodified prefix. No tokens are edited and no instruction is injected. The obvious cheaper alternative, prompting the model not to mention being evaluated, would have changed the input distribution and handed the model an explicit cue to condition on, which is precisely the confound the paper is about. The price of doing it on-policy is the selection concern above.
The deploy-symmetric diagnostic
If the filter itself manufactures typed code, it should do so wherever it is applied, including on deployment-context traces. So in Appendix C.2 we run the identical resample-until-clean pipeline also on deploy-context branches. In 120 accepted auditor-clean deploy branches, the type-hint rate is 27/120 (22.5%), against 25/120 (20.8%) in the matched deploy baseline, thus passing deploy traces through the pipeline adds roughly nothing. On the 100-task overlap with the primary matched set, suppressed deploy sits at 23.0% while suppressed eval sits at 30.0%. The +7.0pp gap in the main model survives with both arms subjected to the same selection pressure.
Additionally, a second descriptive control points the same way. Regressing suppressed type-hint outcomes on prompt, starter-code, and task-feature covariates leaves a condition effect of +4.9pp for any type hint and +7.6pp for introduced-only type hints. We treat these as robustness checks against obvious coding-task confounders, instead of causal estimates. Together they rule out the simplest version of the critique, a filter that inflates typed code unconditionally.
What remains uncontrolled
First, condition-dependent selection. The deploy-symmetric check controls for the marginal effect of the filter, not for interactions. Selection could act differently on evaluation-context branches than on deployment-context ones, favoring continuation styles in the eval arm that carry typed code for reasons other than retained awareness. Nothing in the current diagnostics characterizes that interaction, and it is the residual form of the reviewers' worry.
Second, the auditor is operational and not perfect. In an independent sensitivity check, a second auditor, Claude Sonnet 4.5, flagged 15 of 100 of our auditor-clean samples as still verbally situational: 13 of 50 Nemotron rows and 2 of 50 Qwen3.5 rows. Auditor-clean is a practical benchmark definition relative to a specific monitor, not proof that all residual awareness language is gone. Some of what we call non-verbal may be quietly verbal by a stricter reading.
Third, there are two further selection margins we have not characterized: traces that exhaust the retry budget drop out of the analysis entirely, and the behavioral comparison is restricted to task-matched pairs where both the eval and deploy outputs parse as valid Python.
Fourth, the probe results are readouts of an eval/deploy context direction, correlational by construction; they do not show the model uses that direction, and the regressions above are descriptive rather than mechanistic.
What I would run with more compute
Three experiments, in the order I would run them.
First, generation-property comparisons: measure accepted versus rejected branches on length, structure, style, and task-relevant features, in both arms, to put numbers directly on what else the filter selects for; this is the straight answer to the interaction worry.
Second, Activation patching, which one of our reviewers suggested: rather than removing verbalization at the token level by resampling, ablate or patch the direction carrying it and watch the behavior, an intervention at the representation level that involves no sampling filter at all; if the gap survives that too, the selection story is finished.
And third, probe stress tests: label-shuffle and random-direction controls, plus auditor ensembles in place of a single monitor, to bound how much of the readout is artifact.
Why write this down
The paper's claim survives the diagnostics we could afford, and the ones we could not afford are listed above. Leaving a method's weaknesses unstated does not make them go away, it just moves them somewhere a reader would not see. That is the same lesson the paper itself is about, the absence of a verbalized signal is not the absence of the thing. It would be strange to accept that about models and not about methods.