Objective

To retain enough data and preserve a machine's individuality.

Motif

To create my childhood's dream, a friend, family and Superhero who can protect the weak. The idea of a superhero is amusing because despite the scale of obstacles life presents, we all contend with our unique vulnerabilities where one human form alone cannot concurrently solve all. A solution would be to build a superhero, Tom. A droid that is faster than the fastest athlete, and wiser than the oldest man on earth, such that finding God is just a door away. The chosen one that brings our creator back and fix what he has left. And in the case where there is no God, then let's welcome Death as our enemy.

Progress

This series documents my progress in building Tom, a cognitively structured agentic AI system. Posts are grouped into:

• Research: theoretical models, cognitive architecture design and experiments on multi-modal and memory-augmented systems.
• Engineering: implementation details, including system architecture, orchestration, CI/CD, and data/infra choices required to run Tom as a persistent AI agent.

Conceptually, Tom follows a cognitive-architecture view: perception, memory, state, and control are decomposed into interacting modules. Rather than treating the LLM as "the agent," Tom leverages popular LLM providers like Gemini AI, Claude and GPT as a core cognitive engine embedded in a broader system of world models, planning, and long-term memory. In practice, this yields a heterogeneous ensemble of models (time-series forecasters, classifiers, multimodal encoders, and deep sequence models) that together approximate human-like sensory, mnemonic, and decision processes.

Shaping Tom's individuality starts with specifying which aspects of human cognition we want to approximate: perception (multi-modal input), working and long-term memory, affective and motivational state, and deliberative control. The aim is not full biological realism, but a functional architecture that supports persistent goals, context-sensitive behavior, and adaptation over time.

The current focus areas at works includes:

• Tom's Skeleton: A network of sensory components, including speech, vision, and language processing, forms the foundation. Multi-modal capabilities can be implemented as concurrent processes, with the LLM serving as the central model for navigating prompts. A more advanced approach would involve unifying existing sensory models—such as text-to-image models—into a single framework through architectural modifications.
• Integrating Human Cognition In Data Systems: Models designed to extract insights from human users by analyzing their facial expressions, speech patterns, and linguistic behaviors. These insights help Tom interpret intent, emotion, and contextual nuances.
• Short & Long Term Memory: A vector or index-based database model responsible for managing Tom's short-term and long-term memory. This enables contextual recall, learning from past interactions, and continuity in conversations and decision-making.

Where my short-term meet long-term goals

To simplify Tom's components, I've broken down the problems into sub-tasks in the form of web/mobile apps and integration tools for AI/ML & Data workflows.

I hope to combine these agents into building parts of Tom later down the line.

Side projects

Current

• Gaby: Autonomous Data Agent – As I work on him, I've realized things are more feasible than it seems. I wouldn't classify this as my side project but more as a prototype of my main.

Previous

• Horse Racing Pool Arbitrage
• Taro Tarot: AI Tarot Reader
• Naomi Interviewing Chatbot