Brain-to-Text '25 asks you to take the neural signals recorded from a participant's brain implant while they try to speak, and turn them back into the sentence they were trying to say. The score is Word Error Rate, so every swapped, dropped or invented word costs you.
The RNN already hears the sounds. What it's missing is a sense of which words usually come together.
The competition gives you a pretrained RNN, and my first instinct was to retrain it. I decided not to. The RNN's raw decodes are close but noisy: roughly the right sounds, sometimes the wrong words. That is a language problem more than a signal problem, so I'd rather spend my time on the part that knows about language.
The plan
- Raw decodes: run the pretrained RNN as-is.
- Rescoring: an online n-gram classifier works as a language-model layer. It looks at the words around each position and asks whether the decoded word fits there.
- Ensemble: combine the RNN's output with the rescored output, so the n-gram model can't override a confident RNN on its own.
ℹ️ Info: "online" here means the n-gram counts keep updating as decodes come through, instead of being frozen after one pass over a training corpus.
Why this one matters to me
Most of my hackathons are separate from Tom, but this one isn't. Turning a messy sensory signal into language is exactly the kind of sensory piece Tom is meant to be built from. If a small language layer can clean up a decoder's output here, the same pattern of a sensory model followed by a language-aware corrector could slot into Tom's skeleton later.
Where it's at
- ✅ Raw RNN decodes
- 🔄 n-gram rescoring layer. Self note: I need to measure WER before and after rescoring on the same split, otherwise I can't tell whether the layer is helping.
- 🔄 Ensemble weighting