Synthetic Text Generation for Training Large Language Models via Gradient Matching

What came before

Embedding-level text distillation (sldd-2021/TDD and successors) optimizes continuous word embeddings that are unreadable and tied to one embedding table; dilm-2024's DiLM fixes readability by training a generator LM with gradient matching, but that requires training a proxy LM from scratch (a scaling bottleneck) and, because the generated text can include memorized real examples, provides no privacy guarantee; separately, LLM-prompted synthetic data generation (zero-/few-shot GPT-4-style pipelines) is heuristic, offers no convergence guarantee for the downstream model, and does not preserve privacy of the source data either.

The problem

No existing method for generating synthetic text for LLM fine-tuning is simultaneously human-readable, scalable to LLMs with billions of parameters, and accompanied by a formal convergence, performance, or privacy guarantee; text's discreteness makes the underlying optimization problem (find a low-perplexity token sequence whose gradient matches real data) NP-hard in principle.

The idea

Reformulate discrete-text gradient matching as a constrained non-convex optimization — synthetic token embeddings must both lie in the model's vocabulary embedding set and satisfy a perplexity bound — and solve it with ADMM, alternating between a continuous gradient-matching update and a projection back onto (readable, low-perplexity) vocabulary tokens, which yields provable convergence and, by privatizing the target gradient before matching, provable differential privacy.

How it works

Given a pretrained LLM (Phi) and a target dataset $\mathcal{D}_{real}$, GRADMM introduces an auxiliary variable $\mathbf{Z}$ constrained to the vocabulary embedding set $\mathcal{E}$ and solves $\min_\mathbf{X} f(\mathbf{X}) + \mathcal{I}_\mathcal{E}(\mathbf{Z})$ s.t. $\mathbf{X}=\mathbf{Z}$ via ADMM: a primal step optimizes synthetic embeddings $\mathbf{X}$ (with Adam) to match a clipped, Gaussian-noised (for $(\varepsilon,\delta)$-DP) real-data gradient at the pretrained parameters (last-layer gradient only, for tractability and speed — 2.6x less memory, 2.3x less time than full- gradient matching); a projection step maps embeddings to the top-$k$ most probable vocabulary tokens conditioned on previously decoded tokens (top-$k$ decoding, not nearest-neighbor, to keep text coherent); and a dual step updates Lagrange multipliers. A post-hoc filtering stage removes synthetic examples with the wrong predicted label (via few-shot evaluation), keeps only the lowest-gradient-matching-loss examples per class, and rebalances per-class average loss. The convergence proof (Lemma 4.1, Theorem 4.2, Corollary 4.3) bounds the fine-tuned model's parameter/loss distance from a real-data fine-tuned model as a function of the residual gradient-matching error, under smoothness and PL*-condition assumptions appropriate to short fine-tuning runs. Nothing beyond the synthetic text itself is stored; no generator network is kept (unlike DiLM).

Evidence

Table 1 (SST-2/Tweet-emotions/Rotten-tomatoes, Phi model, hard labels, various data sizes): GRADMM ($\varepsilon=\infty$) beats zero-shot LLM generation, few-shot LLM generation, and Herding/K-center/ Random real-data selection by up to 13.1% (e.g. SST-2 n=5: 86.5% vs. best baseline 75.3% Random), and the DP version ($\varepsilon=0.05$) only slightly trails the non-DP version (e.g. SST-2 n=5: 84.2% vs. 86.5%) while being the only method with a privacy guarantee. Data-scarce regime (Figure 1): generating 100 synthetic examples from just 5 real validation examples outperforms training directly on those 5 real examples by 15.7-31.5% across three datasets. Table 2 (cross-model transfer): text generated by matching Phi's gradient, when used to fine-tune Llama-3.2-1B and OPT-1.3B instead, still beats zero- shot/few-shot/coreset baselines in most cells (e.g. Llama-3.2-1B SST-2: 89.4% vs. 88.4% best baseline). Privacy: loss-based membership-inference-attack advantage is -2.5% to -2.9% (near chance) for both DP and non-DP variants (Sec. 5.3). Cost: matching only the last-layer gradient (vs. full gradient) cuts generation memory from 44.6GB to 17.3GB and generation time from 4.6 to 2 hours on one H100 GPU (Table 4).

Limitations

Requires access to per-sample real-data gradients from the target LLM, so it needs white-box access to a (possibly large) model rather than only its outputs; the top-$k$ projection/filtering pipeline means synthetic examples are generated independently rather than jointly optimized, so the final filtered set only approximately matches the target gradient; experiments use one mid-sized base generator model (Phi) and three short-text classification datasets, not open-ended generation, larger sequence lengths, or full LM pretraining; the convergence guarantees rely on fine-tuning-specific smoothness/PL* assumptions that may not hold for longer training runs or pretraining from scratch.