Knowledge distillation trains a smaller "student" model to mimic a larger "teacher" model by learning from the teacher's outputs — its predictions and confidence levels — rather than raw data alone. This provides richer training signal than simple right/wrong labels.

Black-box distillation is the variant where you lack access to the teacher's internal weights or architecture — only its outputs. This is the situation most developers face with closed models like GPT-4 or Claude: you query via API, collect responses, and use those to train a smaller, cheaper model that approximates the original's behavior.

This matters because frontier models are expensive to run at scale. Distillation lets teams capture much of a powerful model's capability and deploy it at a fraction of the cost — a key reason the technique has grown as APIs became the dominant way people access AI.