Application · Optimization in a generative latent space
FedDG
Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents
Yuqi Jia, Saeed Vahidian, Jingwei Sun, Jianyi Zhang, Vyacheslav Kungurtsev, Neil Zhenqiang Gong, Yiran Chen
ECCV 2024 · first public 2023-12-03 · arXiv 2312.01537
In one paragraph
Moves dataset distillation for federated learning from the client to the server: instead of clients distilling and uploading synthetic data, the server uses prior knowledge from pretrained deep generative models' latents to synthesize data representations from heterogeneous client model architectures, letting local devices train smaller surrogate models while the server trains a larger global model on a multi-modal distribution. Reports up to 40% accuracy improvement over non-distillation FL and 18% over existing dataset-distillation FL methods under high heterogeneity, plus faster convergence, with a theoretical argument that the process asymptotically resembles centralized training.
Where it sits
- Optimization in a generative latent space (Generative priors)
- Setting: Image classification
Abstract (verbatim from arXiv)
Data heterogeneity presents significant challenges for federated learning (FL). Recently, dataset distillation techniques have been introduced, and performed at the client level, to attempt to mitigate some of these challenges. In this paper, we propose a highly efficient FL dataset distillation framework on the server side, significantly reducing both the computational and communication demands on local devices while enhancing the clients' privacy. Unlike previous strategies that perform dataset distillation on local devices and upload synthetic data to the server, our technique enables the server to leverage prior knowledge from pre-trained deep generative models to synthesize essential data representations from a heterogeneous model architecture. This process allows local devices to train smaller surrogate models while enabling the training of a larger global model on the server, effectively minimizing resource utilization. We substantiate our claim with a theoretical analysis, demonstrating the asymptotic resemblance of the process to the hypothetical ideal of completely centralized training on a heterogeneous dataset. Empirical evidence from our comprehensive experiments indicates our method's superiority, delivering an accuracy enhancement of up to 40% over non-dataset-distillation techniques in highly heterogeneous FL contexts, and surpassing existing dataset-distillation methods by 18%. In addition to the high accuracy, our framework converges faster than the baselines because rather than the server trains on several sets of heterogeneous data distributions, it trains on a multi-modal distribution. Our code is available at https://github.com/FedDG23/FedDG-main.git
BibTeX (generated; prefer the venue's official entry)
@article{jia2023unlocking,
title = {Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents},
author = {Yuqi Jia and Saeed Vahidian and Jingwei Sun and Jianyi Zhang and Vyacheslav Kungurtsev and Neil Zhenqiang Gong and Yiran Chen},
journal = {ECCV 2024},
year = {2023}
}Nearby in Optimization in a generative latent space
Condensing Action Segmentation Datasets via Generative Network Inversion
Guodong Ding, Rongyu Chen, Angela Yao · CVPR 2025notableVideopaper ↗
GSDD — GSDD: Generative Space Dataset Distillation for Image Super-resolution
Haiyu Zhang, Shaolin Su, Yu Zhu et al. · AAAI 2024notableDense predictionpaper ↗