DataBy AI aims to automate every stage of the database lifecycle through an autonomous AI agent. The following sections present detailed work, organised by lifecycle phase.
This post documents the procedures for cleaning and transforming datasets, and how I enabled Gaby to self-orchestrate his decision-making.
Objective
Robust data-cleaning stages that adapt to any dataset, while allowing reasoning AI agents to steer themselves—balancing playful, curious exploration (explore) with more traditional, goal-driven strategies (greedy/exploit).
Environment Scenario
This is further divided by whether the user has defined their ML/AI modelling objective.
User Inputs:
| User Input | Description (and data type) | Required? |
|---|---|---|
Uncleaned data sample so fields would be of object or string data types. |
IP Subnet Address to connect to database. | Yes |
| Data descriptor chips: short or long text describing the dataset, e.g. 'finance' or 'This dataset comes from finance sales logs' | string | Yes |
| Modelling objective, e.g. to predict house prices for the next financial quarter. | string | No |
Data Cleaning General Steps
These are the data cleaning stages required for every dataset, regardless of the data type or modelling problem.
(
"Summarise, document, identify and understand the dataset and data columns.",
"Identify and process the missing values in each data field.",
"Reason about and identify strategies for handling duplicated rows and columns.",
"Identify and process anomalies.",
"Transform data columns, for example, encoding binary labels.",
)