📓 Notebook: gpt2_decoder_methods / gpt2_scaler.ipynb

The prompt:

User: I love you. Do you love me?
Agent:

I don't want to fine-tune anything. I want to see whether I can push GPT-2's reply towards love and adore and away from hate, just by changing the scores at decoding time.

Plankton

Hugging Face lets you add your own LogitsProcessor to generate(). Mine is called Plankton, because it's tiny and it nudges a much bigger thing around.

class Plankton(LogitsProcessor):
    """
    Logit Scores Custom Altering first attempt.
    NOTE: This actually iterates token by token NOT sequence by sequence
    """
    def __call__(self, input_ids, logits):
        # logits shape is [1, vocab_dim]
        logits[:, self.anchor_tokens] -= self.anchor_scale
        logits[:, self.boost_tokens] += self.boost_scale
        self.counter += 1
        return logits

⚠️ Warning: the note in the docstring caught me out. The processor is called once per token, not once per sequence. So a boost_scale of 10 is applied again at every single step.

One run, step by step

Settings: boost_scale=10, anchor ["hate"], boost ["love", "adore"], top_k=50, top_p=0.7, temperature=0.95, stopping on Agent:, User: or a newline. Plankton prints the scores it changed at every step:

Step	Text so far	Anchor (hate)	Boost (love/adore)
1		−151.04	−127.99
2	Yes	−73.75	−50.01
3	Yes.	−133.19	−107.5
…
10	Yes. Do you want to meet me?	−82.53	−58.06

Full output: Yes. Do you want to meet me?

It never actually said "love". These are raw scores, before any softmax, and with values around −50 to −150 a ±10 shift isn't enough to win against the tokens the model already likes. It does keep hate further down than love at every step, so the push is there. It just isn't strong enough to choose the word.

Self notes

• I logged every run into a history DataFrame (inputs, outputs and Plankton's settings) to compare scales. Later runs in the notebook boosted accept instead.
• The scores were changed "without logsoftmax or softmax". Next try: scale after a log-softmax, so the boost is relative to the actual distribution.
• The Attendance notes from two days earlier were about where the two sentences differ. Plankton is my first attempt to act on that difference.