The application of machine learning algorithms to high-risk industries such as health informatics, medical diagnostics, and financial risk modeling requires access to sensitive real-world data. At present, stringent legislation such as the Health Insurance Portability and Accountability Act and the European Union General Data Protection Regulation governs the sharing of private information. One potential solution to this dilemma is the generation of synthetic data which emulate the underlying distribution of the real-world records without capturing any of the private subjects’ attributes. Generative Adversarial Networks and Diffusion Models represent the state-of-the-art approaches to create tabular, imaging, and sequential data suitable for training confidential machine learning algorithms. The key challenge is to evaluate whether such a procedure provides adequate privacy guarantees for the re-identification risk while preserving statistical properties to train accurate predictive models. Generative Adversarial Networks involve the simultaneous training of two deep learning models – a generator and a discriminator – that compete against each other to learn the underlying distribution of a target dataset. Conditioned GANs and Tabular GANs represent specialized variations of this architecture that can capture complex nonlinear relationships and high-dimensional non-Gaussian distributions in financial and healthcare data. The main limitation of traditional GANs is the lack of formal guarantees for differential privacy due to the memorization of training samples by the generator network. Specifically, since the discriminator network has direct access to the training distribution, it can influence the generator to memorize unique statistical properties of specific training samples. Such an attack is known as membership inference and is applicable when the target model receives private data points at inference time. Similar risks arise from attribute inference attacks or distance-to-closest-record attacks.
Source: https://www.geeksforgeeks.org/deep-learning/generative-adversarial-network-gan/
The alternative approach to GANs is represented by the family of diffusion models that involve the step-wise transformation of randomly generated Gaussian noise into data samples that follow the distribution of a target dataset. Unlike GANs, diffusion models do not require the explicit definition of a discriminator network and operate by estimating the likelihood function of a forward process, which gradually adds noise to the input samples. Denoising diffusion probabilistic models and score-based generative adversarial networks belong to this category and demonstrate compelling results in the synthesis of high-resolution medical imagery. While the architecture of diffusion models does not involve an explicit discriminator, their deep neural network components might memorize the training distribution and thus reconstruct private records with arbitrary precision. Such a scenario is particularly plausible for large-scale vision transformers trained on sensitive patient data. A possible solution to this dilemma is provided by differential privacy guarantees, which can be added to either GANs or diffusion models during the training phase. The concept of differential privacy originates from the formal theory of privacy and involves the addition of carefully calibrated noise to the model updates to obfuscate the contribution of any given training record. Differential privacy stochastic gradient descent represents the practical implementation of this idea, which has been successfully applied to the task of private record synthesis. However, the increased level of privacy during the model training usually comes at the cost of decreased statistical utility, which might undermine the training of accurate machine learning algorithms. The central dilemma in the application of differential privacy to synthetic data generation is represented by the trade-off between privacy guarantees and utility loss. More specifically, the addition of large amounts of noise to the model updates during training decreases the ability of a generator network to capture complex distribution properties of the training data. From a practical perspective, it might be possible to mitigate this problem by experimenting with different hyperparameters of the generator and discriminator networks. However, there is no doubt that such a procedure will require additional computational resources to fine-tune the hyperparameters and find the optimal balance between privacy and utility. At the same time, it is possible to analyze the utility of the synthetic data generation process using standard statistical and machine learning evaluation protocols. The simplest approach is to train a predictive model on the synthetic data and measure its performance on the hold-out test set of real-world samples. A wide range of empirical evaluations has demonstrated that the application of differential privacy to the training of GANs results in lower downstream prediction accuracy. However, it should be noted that an unconstrained generator network can attain higher levels of predictive performance on certain health informatics and financial analytics tasks. To summarize, the generation of synthetically generated data samples using GANs or diffusion models provides sufficient statistical utility to train confidential predictive models if the hyperparameters are carefully tuned. Sources: https://arxiv.org/abs/2302.13861 https://arxiv.org/abs/2012.11774 https://arxiv.org/abs/2501.03941