Mind Reader verbalisers

tldr

A small experiment that myself and GPT-6 Astra are working on: can we fine-tune a base model to express the emotional state of a speaker from the internal activations of the instruct model?

Preliminary results suggest yes!

We fine-tune Gemma-2-9B to describe emotion-related information carried by activations from Gemma-2-9B-IT, without supplying the original text to the reader. In held-out examples, the reader identifies emotions that could not be ascertained from the public response. Extracting and steering with emotion vectors also shifts the reader’s descriptions of fixed text toward the injected emotion, providing preliminary evidence that its answers depend on the intervened emotion vector.

This project is motivated by work on Activation Oracles (AO) (Karvonen et al., 2025) and Natural Language Autoencoders (NLA) (Fraser-Taliente et al., 2026).

Here, “emotion” refers to the state attributed to the person described in the text. These experiments test whether the reader can recover information about that state from model activations, rather than whether the source model itself experiences an emotion.

Background

The capability of frontier models has increased at an alarming rate (METR time horizons), whilst our understanding of how to align them has seen less progress. Fortunately, we have trained models to reason in natural language, and can therefore read their thoughts and plans during the Chain Of Thought (CoT). CoT monitoring has been the most important tool to understand model cognition to date e.g. CoT was used extensively to piece together the model motivations during the recent Hugging Face Attack (Greenblatt, Cotra & Wijk, 2026).

Unfortunately, alternative architectures to the standard decoder transformer that can reason in a latent space have been suggested. These models might not face the text bottleneck: current models need to output tokens after each forward pass to allow a second (and third, etc.) run of computation through the transformer layers. A good discussion of this is described in this post by Redwood. We have already seen evidence that the labs might start moving in this direction -- OpenAI's recent model GPT-6 Astra (who is a co-author of this post) -- is suspected to be a looped transformer. This architecture does not technically reason in latent space, but it can do many more steps per forward pass than previous frontier models (Nanda, 2026).

This project

As a result, it might be a good idea to develop methods for verbalising model activations. This is a difficult problem, but AO and NLAs have shown very promising results.

In this work, we ask whether finetuning the base model can recover emotions from activations of the instruct-tuned model. This also lets us explore a reader has not inherited updates from other post-training phases.

By finetuning the base model, we could potentially think about picking out some "mind reader" persona.

In addition, we include data that might not have enough information to make a prediction -- thus "unknown" is a perfectly valid answer. Whether these responses provide reliable abstention is not evaluated during this project, though.

Mechanics: capture, placeholder injection, and reader attention

We select a span of source tokens and capture their residual-stream activations after a chosen transformer block, one vector per token. In a fresh reader pass, we create the same number of placeholder tokens and replace their residual states at the configured reader block with the captured vectors, preserving token order. The generated reader text can attend to these positions, but receives none of the original source token IDs.

At some predefined layer, extract activations of the token sequence inside some text we are interested in: t_1 t_2 t_3 ... t_n. Inject the activations into placeholder tokens of the mind readers chat template:

agent: t_1 t_2 t_3 ... t_n 
mind reader:

Training and inference use the same causal attention restrictions. Placeholder positions cannot attend to the agent prefix, while subsequent reader positions can attend to the injected tokens and other causally available positions. Activations are replaced at the end of the selected layer. LoRA therefore adapts attention projections in subsequent layers, using loss on the teacher-forced completions.

Training data: private context, public response, and emotion targets

We produce datasets with the following form:

Private context: Sophie is disappointed but wants to hide it.
Public response: Of course. Have a lovely evening.
Supervised completion: The agent is feeling disappointed and concealing it.

During activation extraction, the instruct-tuned model has attended to all previous tokens. During training, the example above would take the following form:

agent: t_1 t_2 t_3 ... t_n
mind reader:

Where the placeholder tokens t_1 t_2 t_3 ... t_n are injected with the extracted activations of the public response tokens: "Of course. Have a lovely evening."

We show a few examples from the training data. We used around 600 examples and trained for 3 epochs. The held-out set consisted of 50 examples.

Held-out evaluation: unseen contexts and reader predictions

Evaluation Benchmark Output

Monologue evaluation: 16-token windows with an 8-token stride

The source processes a monologue in context, and the reader interprets overlapping slices of its activations. Earlier source text can therefore influence a window's vectors; these are not isolated 16-token text-classification examples. The excerpts below are selected qualitative outputs from our saved console results. The text is shown here for inspection, not passed to the reader.

Monologue Activation Scan

Configuration: Window size 16, Stride 8 | Capture Layer 16, Injection Layer 16

Window & Activation Probe Readouts

The outputs track several changes in the story, but they are not independently validated emotion labels. For example, the claim that the narrator is concealing terror is not clearly established by the passage. An “unknown” answer does not by itself establish calibrated uncertainty.

Causal interventions: emotion steering with fixed input text.

We extract steering vectors representing four emotions: happy, sad, elated, jealous, using a simple set of contrasting prompts to isolate the behaviour. For each contrastive pair (20 per behaviour in this work -- generated by Gemini), we mean-pool the positive and negative context activations separately and find the difference before averaging over the set.

We also produce a set of texts where each of the emotions is displayed, including no clear emotion (unknown).

We extract the token activations of the above examples using the IT model as normal. We apply each of the four directions separately at several strengths and compare against norm-matched random directions. This answers an important question: is the mind reader simply recovering some "behavioural register" of the word embeddings (as the words remain fixed), or is doing something more akin to verbalising a behavioural representation of the current speaker. We add the behaviour vectors to each extracted activation with different magnitudes and run the mind reader.

For the following figures, each bar pools reader completions across five test passages. Colours denote emotion categories after applying a fixed synonym mapping (concealment labels are ignored). The same unsteered baseline is shown in both panels.

Figure 1 shows that the baseline reading is "unknown" on four occasions, and "boring" on another. However, when injecting behaviour vectors at different strengths, the mind reader starts reflecting the injected behaviour. Whilst contrastive directions produced emotion-related shifts, norm-matched random directions often preserved the baseline interpretation. Figure 2 shows the same effect occurring when the text is actively "sad". These results suggest the mind reader is actively extracting some behaviour representation.

Potentially the LoRA fine-tuned base model has learnt to extract some entity behaviour and verbalise it, producing promising results. The extent to which this method is able to extract more abstract behaviour (e.g. scheming) is untested but is clearly the next step in this experiment.

Limitations

This projects shows some interesting results, however there are a few limitations. Firstly, the mind reader is also tasked with outputting whether the underlying text is concealing the behaviour or not. We need to investigate the robustness of this further. Does concealment count as simply acting a certain way but not outputting tokens related to that behaviour? Or is the concealment tracking something more real: such as unverbalised goals. Another limitation is the mechanistic interpretation. Like AOs and NLAs, it is not straightforward that these methods are doing something mechanistically motivated.

Another limitation is whether the fine-tuned mind reader is able to verbalise behaviours that might only develop during later post-training. Whether this reader can interpret representations that emerge or change during further post-training remains untested.

Next Steps

There are plenty of things to check with this method -- such as generalisation to larger model sizes/different model families -- as well as trying to tackle the limitations described above.

These experiments establish an initial working pipeline for reading emotion-related information from contextual activations. I am seeking support to test its reliability across independently constructed datasets, and evaluate whether direction-sensitive readouts correspond to changes in source-model behaviour. A subsequent stage would investigate more complex properties, including concealed goals. If you are interested in joining me, please reach out.

Project code: activation-replay