Lab: build a synthetic data pipeline: validate, deduplicate, decontaminate and balance
A runnable pipeline that turns 1,200 raw model generations into a clean training set, with a waterfall showing what each stage removed: format and schema checks, quality and PII filters, exact and near-duplicate removal, evaluation decontamination and class balancing. Pure Python, no installs.
Good fine-tuning data is mostly the result of filtering. A teacher model, whether a large API model or your own pipeline, produces plenty of text, but a lot of it is malformed, repeated, unsafe or already in your test set. In this lab you build the pipeline that turns raw generations into a trustworthy training set, and you measure what each stage removes. It builds on Data preparation and synthetic data and runs with plain Python.
Pro content
Keep reading with Inference Academy Pro
Unlock the rest of this lesson, including 1 animated diagram, plus every advanced lesson, hands-on lab, the larger playground models and completion certificates.