18 ways to check whether data can be trusted for AI
ETSI has published TR 104 180, a technical report that defines 18 metrics for measuring data quality, giving companies a way to check whether their data is good enough for AI before they use it. The report defines each metric and includes the formulas needed to calculate it.
The metrics fall into four groups. The first deals with the basics, whether data is complete, accurate, consistent, and free of duplicates. The second asks whether the data can be used, meaning it’s available when needed, documented well enough to trace back to its source, and up to date.
Fairness makes up the third group, looking at whether the data treats different groups of people evenly. Privacy rounds out the list, checking whether people in the data can be identified and whether sensitive details are protected.
“It is essential that data quality is measurable, especially for organisations who need to establish whether its data is fit to essential intents, like it would be the case of trustworthy AI,” Diego Lopez, Chair of the ETSI Technical Committee DATA, said.
“ETSI’s standardised metrics provide a common language for assessing data quality, giving quantitative evidence as to whether a dataset is fit for its intended purpose. This lays important groundwork for more consistent and repeatable approaches to data quality assessment, as AI and data-driven technologies continue to evolve,” Lopez added.
To test the metrics, the researchers behind TR 104 180 applied them to two public datasets. One held sensor readings from aircraft engines. The other was a US census dataset, long used in machine learning research, built to predict whether someone earns over 50,000 dollars a year.
The engine data held up well. It was complete, accurate, and steady over time. The census data raised more concerns.
A gender gap in the numbers
The researchers looked at the census data for bias between men and women. About 31 percent of men in the dataset were marked as high earners, compared with about 11 percent of women. That gap is close to three times over, which TR 104 180 treats as a warning sign for bias.
Two privacy problems in one dataset
The census data also failed on privacy, in two separate ways. First, looking at just four details together, age, race, sex, and country, was enough to single out specific people in the dataset. Some individuals could be identified on their own, which the report flags as a serious risk.
Second, when the researchers checked whether sensitive fields were protected, they found personal information stored in plain text, with no masking or encryption in place.
TR 104 180 treats both findings as data quality failures. Anonymity and confidentiality appear on the same list as accuracy and completeness, scored with the same kind of formulas.
TR 104 180 was developed with a working group that included Sejong University, EGM, TTA, Daejeon University, and CNIT. The group also built an open-source tool that scores any dataset against the 18 metrics.
