To initiate a stream of consciousness in a neural network, it is vital to establish temporal memory first, as this is the primary catalyst for steering Tom away from his robotic self.

Hopfield Neural Network (HNN)

The Hopfield Neural Network (HNN) is a type of recurrent artificial neural network, and comes in two flavours: classical and modern. An HNN is an associative memory: it excels at recalling patterns, even from incomplete versions of those memories. However, classical HNNs have these limitations:

• Limited storage capacity.
• A slow convergence rate to a stable state during pattern recall. In practice, it behaves more like a simulation algorithm, which reminds me of numerical approximation methods like Euler–Maruyama.
• Spurious patterns may emerge during recall, stemming from factors such as 1) the inherent capacity limits of storage, 2) what neuroscientists describe as "the neuron's reaction to being entrapped at local convexities", and 3) correlation among the stored patterns.

To summarise, more stored patterns means more retrieval errors. Figure 1 contrasts the inputs and outputs of a classical HNN for discrete states. Using the Hebbian learning rule, I fed it a set of 7 block-letter patterns. The initial state is the learned pattern, while the final state is the pattern retrieved.

Figure 1.1: Letter 'L'

Figure 1.2: Letter 'S'

Figure 1.3: Letter 'X'

Figure 1.4: Letter 'U'

Figure 1.5: Letter 'I'

Figure 1.6: Letter 'J'

Modern HNNs have mitigated most of the problems that limited classical ones. For instance, a recent paper, Hopfield Networks is All You Need, improved the HNN update rule using the transformer's attention mechanism. This increases pattern storage while lowering retrieval errors. For more on HNNs, check out Hopfield’s most recent work here, or for a quick summary, click here.

The slow development of this field may be due to uncertainty about how best to use it. I suspect that uncertainty comes from the unsolved cases in reinforcement learning, an area where I believe models like HNNs shine the most, especially when you can incorporate time into their memory storage.

Tom vs HNN

In Tom’s case, the HNN acts as a replica of short-term memory, distinguishing observations by associating current patterns with previously stored ones. Its role in Tom’s long-term memory is still being fiddled with, especially for things like recalling memory traces of people. Beyond that, I haven’t discovered anything breathtaking. At the moment, its results are most promising for storing signals from vision and sound tasks. In NLP, it isn't useful for prediction, but rather as a tool for observing historical conversations. It does this effortlessly thanks to how seamlessly it integrates into network architectures, particularly Markov networks and stochastic processes such as Markov decision processes.