Defining Tom's memory in practice is crucial before building his decision-making components. In fact, I wish I could have stressed to my past self how much its role influences every other build.
To build Tom's memory architecture intuitively, I eased the task by (ironically) simplifying the human mind and reframing the task as one of replication. Figure 1, adapted from Hung Thai Le (2021), simplifies the cognitive memory chambers of the human mind.
Figure 1. Simplified representation of human cognitive memory chambers. Adapted from Le, H. T. (2021). A simple model of the human memory. arXiv:2107.01390v1.
The main chambers, short-term and long-term memory, can be defined as follows:
- Short term: responsible for processing and navigating sensory information (sensory memory / temporal lobe), and acts as temporary storage for live information used in language comprehension, learning and reasoning (working memory).
- Long term: responsible for everything not in short-term memory, i.e. a filing cabinet that enjoys filing and updating itself with all memories, including past trauma.
Extending this diagram into an application for a bot means building intuition from the following table (Figure 2).
| Definition as a data problem | Definition in Neural Networks | Replica Examples in Large Language Models (LLMs) | Related Architectures Examples | |
|---|---|---|---|---|
| Short term | Short-term memory can be understood as a temporary, capacity-limited data buffer subject to decay, interference, and selective transfer. | RNNs like LSTMs or gated models temporarily storing previous data batch(es) to learn the next batch(es). | Persistent Session Cache for streaming chatbots | Streaming / In-Events Architectures |
| Long term | Long-term memory uses builds similar to short-term memory, except that its data cycles complete over extended periods. What gets selected from short-term memory is biased towards the individual's emotional responses or environmental influences. | Datasets fed to a neural network during training. | Data Retrieval / Manipulations from Vector databases | The architecture and process under the process of storing / retrieving / querying Graph / Relational / Indexing Databases or Data Lakes & Warehouses |
Figure 2: Table defining short / long-term memory in application.
Challenge: But which model is most efficient?
Intuitively, replicating long-term memory led me to set up a simple indexed database, but short-term memory was personally more difficult, for the following reasons:
i.) Collecting real-time information from human senses like vision and sound, which, depending on the computer architecture you interact with, requires running multiple processes in parallel. And even with that part backlogged for another day, there's a second problem:
ii.) Differences between session datasets. If you segment the day by hours, you can also segment it by minutes, and so on. The question is: which segment should take priority in the next inference window? Simply put, the answer depends on features related to the type of user the bot interacts with. For instance, it would not prioritise someone else's opinion over mine when reranking its own preferences.
These are the possible builds that replicate short-term memory:
- Database retrieval + prompting mini agent: This requires setting up a database (of any type) and a lightweight agent acting as its gatekeeper (or receptionist). This agent sits between the database and the main agents or humans, so a human needs no prior knowledge of SQL to query the database, and can instead treat the gatekeeper as their virtual librarian.
- Session cache models (KV caches): This involves reranking the KV caches within an LLM during chat sessions to strengthen or steer its text generation. It only works for short chat sessions, as prolonged use can lead the model to generate gibberish.
- System prompt extensions: Across user sessions, the main chat agent can be restored as a new model whose system prompt is periodically updated with past chats or highlights. This method is compute-intensive, in the sense that the context window keeps growing and no data is actually retained. However, for future parts that need a short-term solution, it could be ideal.
- Methods related to weight learning:
- Multi-head attention networks as a ranking model: This method learns from data through its weights and, given its dense architecture, is better suited to offline learning. I'm probably mentioning it more out of my own interest in attention than for its feasibility for the task at hand. I strongly believe that if you can capture chat histories, text-generation caches, etc., then such a model (built in a similar manner) could act as both the database and the assisting model that supplies conditioning vectors to steer the main agent's text generation: two birds, one stone. I am still tinkering with different methods, and while some may work on the current synthetic data, I am unsure how they would perform on large datasets. The ambiguity comes from the generic data problem: the noise imputed into a mock dataset never fully resembles real-life situations.
Over the past year, this part was difficult to build because of the constant updates in machine learning and AI. It often seemed that papers and articles echoed each other, merely rebranding the same methods, or that the reading could lead one to believe there's no alternative solution. But there always is. For instance, using a vector database to store vector representations of text sequences. Simply put, it is a solution like an SQL database with extra steps, and given the compute cost of running large models, you would want to preserve as much as possible during data processing cycles.
For the sake of simplicity, I decided to build the databases (build here translates to pulling a Docker image, so ain't I living in the most fortunate timeline?) independently for short- and long-term storage at this stage. This should leave me more time to read and tinker with different methods. Currently, my progress towards streaming live data from sensory tools is blocked by the challenge of defining the function or algorithm that sits between the large models and the database. Despite admitting that I have no patience for algebra as a beginner, I still did what felt right (but in code). One thing I deeply hate is finishing a part of the agent and staring at the screen, knowing I cannot extend it any further because it (all) relies on this part.
My Foot
- H. T. Le, “A simple model of the human memory,” arXiv preprint arXiv:2107.01390v1, 2021. Available: https://arxiv.org/abs/2107.01390v1