Scoping methods for efficient key-value dataset retrieval.

High latency between workspace kernels or pipelines is beyond my expertise at this stage. What I can control is how, and which, datasets are stored.

The tools at my disposal include Google BigQuery AI/ML. Not because it is a NoSQL database, but because it makes it easy to build statistical methods into the aggregation layers of data pipelines, e.g. time-horizon methods that learn the last 3 months' trends. These are especially useful for building analytical dashboards, or for anyone too impatient to wait for 3 months of data.

If we extend this idea to vectorised features, one can imagine that a key-value data structure for optimally efficient retrieval looks like:

{
    27-01-2026:09:09: {
        np.array([...])
    }
}

This suits cases where the vectors cannot be reverse-engineered to their original form, such as the latent space of deep learning models. Fortunately, that is not the problem here. Instead, the problem is understanding the patterns and differences between datasets I've observed over time. This is especially difficult once you have handled several datasets from the same domain: how do you distinguish or categorise them most efficiently, so you can decide how to handle the next new dataset?

As a human, I am thankful that the weights in my cortex have been automated simply by staying alive and enjoying small glimpses of life. In my bot's case, though, I am unsure how to calibrate his certainty in decision-making against his previous observations.

After some trial and error, I settled on two core strategies:

1. Simple method: key-value storage of the vectors, where keys are timestamps and values are the vectors. Post-processing to optimise the database would mean quantising them further with the sentence-transformers Python library, or performing PCA to retain the subset of features most significant to the current decision-making strategies.
2. Easy and lazy method: create my own way of hashing the pattern of column data types to categorise the features.

I went with the second method, mainly because only on the off chance would I need vector storage for deep learning models. The following dictionaries define the ruleset for assigning unique codes to each dataset my agent observes:

# Session Base Dataset
class GenerateCodeDtypeConfig(Enum):
    """ Configuration for generating code based on data types. """
    pandasDtype = {
        "bool":      "OBO",  # boolean
        "datetime":  "ODT",  # datetime64[ns], datetime64[ns, tz]
        "timedelta": "ODT",  # treat as time-like; keep same family if desired
        "numeric":   "NCO",  # all numeric (int/float) without value-based splitting
        "category":  "CBI",  # pandas 'category' dtype (categorical)
        "string":    "OTX",  # pandas StringDtype
        "object":    "OTX",  # object (often text/categorical; we treat as text to avoid heuristics)
        "None":     "OUK"   # unknown / other dtypes
    }
    categoryCode = {
        "numerical": "N",
        "categorical": "C",
        "others": "O"
    }
    subCode = {
        "continuous": "NC",
        "discrete": "ND",
        "nominal": "NN",
        "ordinal": "NO",
        "binary-label": "CB",
        "multi-label": "CM",
        "datetime": "OD",
        "geospatial": "OG",
        "text": "OT",
        "boolean": "OB",
        "identifier": "OI",
    }

The configuration defines the ruleset for computing unique, pattern-based codes for each dataset. Every dataset my agent receives gets 3 codes. Each data type code is suffixed with an integer giving the number of columns of that data type. For example, a dataset with 3 numerical columns, 1 categorical and 2 others gets the code N3C1O2.

In summary:

• Post-processed features: the categoryCode and subCode maps are the parent and child data types, respectively.
• Pre-processed features: the pandasDtype map covers the original data fields.

Forgetting about vector database retrieval and sticking with fetching and appending .txt files may sound like the boring route. But in most cases in life, boring is good and simplicity always buys us more time.

Example Codes generated. fieldPattern value displays the combined pattern and child data type categories.

For now, I guess that is sufficient as his temporal lobe. Given the time spent on this project, I am certain that the system architecture would support any new features I introduce in the future.