Stage IV: anomaly detection & handling
Outlier detection cases.
Base framework:
- Statistical methods
- Machine-learning-based methods, e.g. OLS and random forests
- Time-series-specific methods (only applicable to continuous data types)
During EDA, statistical methods are generally preferable because they are faster, interpretable, and more aligned with exploratory decision-making. ML methods are better reserved for confirmatory or production contexts where their complexity pays off.
Stage V: the end!
The post-pipeline database management and housekeeping methods to run once the process is done.
Running Workflows
Subsets of a pre-configured sample size, N, are drawn from mean-pooled tabular datasets. These subsets contain only the raw values (column names are excluded). Each subset is then projected into a latent space with reduced dimensionality, aligned to the hidden block size of the sentence-transformer module.
From there, the transformed data is passed into scikit-learn or BigQuery PCA to further reduce its dimensionality, ensuring the dataset shape falls below 384 dimensions (the default embedding size of the model in use). PCA applies a numerical method, singular value decomposition (SVD), to retain the components that explain the most variance.
Database Schema
- End reports for each previous stage in this pipeline.
User Interaction - Frontend
At the end of this workflow, the user is offered these options:
- Download the cleaning report (lowest priority: it means connecting to their database's subnet IP address, which isn't available for some datastores and can be harder to implement).
- Proceed to the next stage:
- Generate the database's current quality scores, OR
- Data insights & significance