Model Extraction vs. Behavioural Cloning: Why Mixing Them Up Leads to Bad AI Security Advice

Two distinct threats are commonly mislabelled as 'model extraction': parameter extraction, which recovers actual model weights via API mathematics, and behavioural cloning, which replicates a model's capabilities by training on its outputs at scale. While stealing a full frontier model's weights through an API remains largely infeasible today, cloning specific capabilities for a few thousand dollars is routine and is the real concern for most practitioners. A third distinct threat involves recovering training data rather than the model itself, with memorised sequences posable as a serious risk particularly for fine-tuned models trained on sensitive or private documents. Research milestones include Tramèr et al. (2016), who demonstrated near-exact recovery of simple models via prediction APIs, and Carlini et al. (2024), who extracted structural parameters from a production language model through logit-exposing APIs. Defences vary sharply by attack type, ranging from limiting API output detail and adding sampling noise for extraction attacks, to deduplication and pre-training data filtering for training-data recovery risks.