CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting

What came before

Parameter-matching dataset condensation methods (DC's gradient matching, MTT's trajectory matching, and kernel-closed-form methods KIP/FRePo) were designed for classification, where a synthetic set is "well-distilled" once a model trained on it assigns the same discrete class label as a model trained on the full data, regardless of logit magnitude; applying them unmodified to time-series forecasting only matches model parameters, not what the paper argues actually matters for forecasting.

The problem

Forecasting has no discrete label to match — its correctness criterion is the pointwise distance between the two models' continuous predictions, which the paper calls a strictly more rigorous similarity criterion than classification's label-only agreement. Naively imported condensation methods only optimize a term implicitly related to overall parameter closeness and miss a separate term the paper identifies as the dominant contributor at low condensation ratios.

The idea

Derive (via a first-order Taylor expansion around the synthetic data point) an upper bound on the forecasting test-error objective that splits cleanly into two additive terms — a "gradient term" (distance between the two trained models' parameters/gradients, which existing parameter-matching methods already optimize) and a "value term" (distance between the two models' predicted values on the synthetic input itself) — and add a lightweight, gradient-free update that directly minimizes the value term, since prior methods leave it unaddressed.

How it works

Theorem 1 bounds $\mathcal{L}_{test}$ by a constant irreducible label-noise term plus the value term $\|\mathcal{M}_{\theta_{s,test}}(s)-\mathcal{M}_{\theta_{f,test}}(s)\|^2$ plus the gradient term $\|(\nabla\mathcal{M}_{\theta_{s,test}}(s)-\nabla\mathcal{M}_{\theta_{f,test}}(s))^\top(x-s)\|^2$, derived assuming (or first-order-approximating) a linear forecaster so gradient = model parameters. The gradient term is what existing trajectory/gradient-matching methods already minimize (proven via Cauchy-Schwarz, reducing to $\|\theta_s-\theta_f\|^2$). For the value term, Theorem 2 shows $\theta_ {f,test}$'s prediction on synthetic input $s$ (though $\theta_{f,test}$ itself is unavailable at condensation time, since it is only sampled at test time) can be approximated by the prediction of any "expert" model $\theta_f^i$ from the offline-trained parameter buffer, because all such experts predict similarly on a given input regardless of their random initialization (empirically confirmed via MDS visualization, Fig. 3). CondTSF is then a one-line, gradient-free additive update applied every $G$ epochs, interleaved with the backbone method's normal matching steps: $s^{(i+1)}_{label} = (1-\beta) s^{(i)}_{label} + \beta\,\mathcal{M}_{\theta_f^i}(s^{(i)}_{input})$, which is proven to shrink the label error exponentially at rate $(1-\beta)^2$ per update. No new network, generator, or label scheme is introduced; CondTSF only touches how the synthetic sequence's target/label values are updated.

Evidence

Table 1 (ExchangeRate/Weather/Electricity/Traffic, distilled length 48, MAE and MSE, backbones DC and presumably others, 5 synthetic datasets averaged): CondTSF reduces test error for every backbone on every dataset, e.g. DC on ExchangeRate: MSE drops 27.8% (0.875→0.632); smallest gains on DC specifically (1.7-11.7% MSE reduction) versus larger gains reported for direct parameter-matching backbones like MTT, because (per Sec. 5.2) DC only indirectly matches gradients-of-gradients each iteration, so its parameter error accumulates and CondTSF's value-term fix has less room to help. Table 3 (cross-architecture, DLinear-distilled synthetic data evaluated by training MLP/LSTM/CNN from scratch): MTT+CondTSF outperforms Random, DC, KIP, FRePo, and plain MTT across ExchangeRate and Weather on both architectures shown. Figure 4 (ExchangeRate, MTT vs. MTT+CondTSF): parameter error (gradient term) converges similarly with or without CondTSF, confirming CondTSF does not interfere with gradient-term optimization, while label error (value term) rises under plain MTT but stays low under MTT+CondTSF, and test error tracks the value-term difference — direct evidence the value term, not the gradient term, is the mechanism CondTSF fixes. Per the paper's own summary, gains are "particularly" large at low condensation ratios (distill ratios as low as 0.83‰-6.33‰ across the eight datasets, Table 2).

Limitations

The theoretical derivation explicitly assumes a linear forecaster (so that gradient = model parameter, letting the two terms decompose cleanly); the paper states this analysis "no longer holds" for more complex nonlinear models, and while an appendix reports CondTSF still helps with non-linear experts and non-parameter-matching backbones, that generalization is empirical, not covered by the theorem; CondTSF consistently helps least on DC among the backbones tested, and the paper attributes this to a mechanism (indirect, per-iteration gradient matching accumulating parameter error) it does not fully resolve; evaluation uses fairly short forecasting windows (24-in, 24-out) and small synthetic-set sizes (length 48, one training pair per synthetic set).