Evaluation & benchmark
A Comprehensive Study on Dataset Distillation: Performance, Privacy, Robustness and Fairness
Zongxiong Chen, Jiahui Geng, Derui Zhu, Herbert Woisetschlaeger, Qing Li, Sonja Schimmler, Ruben Mayer, Chunming Rong
arXiv 2023 · first public 2023-05-05 · arXiv 2305.03355
In one paragraph
Runs membership-inference, robustness, and per-class fairness evaluations across several dataset-distillation methods and reports that distillation reduces but does not eliminate privacy leakage, can degrade adversarial robustness to varying degrees, and can amplify unfairness across classes; proposes a large-scale benchmarking framework covering these axes.
Where it sits
- Setting: Image classification
Abstract (verbatim from arXiv)
The aim of dataset distillation is to encode the rich features of an original dataset into a tiny dataset. It is a promising approach to accelerate neural network training and related studies. Different approaches have been proposed to improve the informativeness and generalization performance of distilled images. However, no work has comprehensively analyzed this technique from a security perspective and there is a lack of systematic understanding of potential risks. In this work, we conduct extensive experiments to evaluate current state-of-the-art dataset distillation methods. We successfully use membership inference attacks to show that privacy risks still remain. Our work also demonstrates that dataset distillation can cause varying degrees of impact on model robustness and amplify model unfairness across classes when making predictions. This work offers a large-scale benchmarking framework for dataset distillation evaluation.
BibTeX (generated; prefer the venue's official entry)
@article{chen2023comprehensive,
title = {A Comprehensive Study on Dataset Distillation: Performance, Privacy, Robustness and Fairness},
author = {Zongxiong Chen and Jiahui Geng and Derui Zhu and Herbert Woisetschlaeger and Qing Li and Sonja Schimmler and Ruben Mayer and Chunming Rong},
journal = {arXiv preprint arXiv:2305.03355},
year = {2023}
}