A 7-criteria evaluation approach for the responsible use of synthetic medical data.
Authors
Affiliations (1)
Affiliations (1)
- Division of Imaging, Diagnostics, and Software Reliability, Office of Science and Engineering Laboratories, Food and Drug Administration (FDA), Silver Spring, MD 10903, United States.
Abstract
To develop an approach for evaluating and reporting the quality of synthetic medical data (SMDs). As SMDs become increasingly common, their responsible utilization benefits from a detailed characterization of their quality and value within the intended usage context. Here, we introduce an approach to assess SMDs across 7 dimensions: Congruence, Coverage, Constraint, Consistency, Comprehension, Compliance, and Completeness. We also describe an approach for reporting the quality of synthetic data to stakeholders. We applied the proposed approach to 7 digital mammography datasets. The analysis underscored the strengths and limitations of each dataset across these dimensions. For example, while datasets generated by generative AI methods show high Congruence, they often fall short in Constraint and Completeness compared to those generated by knowledge-based methods. Additionally, our findings highlighted that the quality of generated data varies across different subgroups (eg, breast density) with certain subgroups (ie, dense and hetero) showing lower quality. This suggests that subgroup-specific fine-tuning of the generative process may have an effect on downstream tasks. The proposed approach provides stakeholders with a tool for assessing data quality, which may be used to better understand and analyse synthetic datasets.. This study introduces an approach for assessing SMD quality. The approach offers a practical mechanism for reporting SMD quality, which can enhance the reliability and usefulness of synthetic datasets.