FedSynth: Gradient Compression via Synthetic Data in Federated Learning

Shengyuan Hu, Jack Goetz, Kshitiz Malik, Hongyuan Zhan, Zhe Liu, Yue Liu

arXiv 2022 · first public 2022-04-04 · arXiv 2204.01273

paper ↗catalogued✓ abstract read

In one paragraph

Instead of transmitting a sparsified model update, each client learns and uploads a small synthetic dataset such that training on it reproduces the client's local update, which the server recovers and aggregates as usual. Reports comparable or better upstream compression than random-masking baselines on three common federated learning benchmarks.

Where it sits

Setting: Image classification

Abstract (verbatim from arXiv)

Model compression is important in federated learning (FL) with large models to reduce communication cost. Prior works have been focusing on sparsification based compression that could desparately affect the global model accuracy. In this work, we propose a new scheme for upstream communication where instead of transmitting the model update, each client learns and transmits a light-weight synthetic dataset such that using it as the training data, the model performs similarly well on the real training data. The server will recover the local model update via the synthetic data and apply standard aggregation. We then provide a new algorithm FedSynth to learn the synthetic data locally. Empirically, we find our method is comparable/better than random masking baselines in all three common federated learning benchmark datasets.

BibTeX (generated; prefer the venue's official entry)

@article{hu2022fedsynth,
  title   = {FedSynth: Gradient Compression via Synthetic Data in Federated Learning},
  author  = {Shengyuan Hu and Jack Goetz and Kshitiz Malik and Hongyuan Zhan and Zhe Liu and Yue Liu},
  journal = {arXiv preprint arXiv:2204.01273},
  year    = {2022}
}