Security certification is a snapshot, and buyers read too much into it

In this Help Net Security video, Mariia Bakhtina, a research specialist at Masaryk University, explains what a Common Criteria or FIPS-140 certificate does and does not promise. In her team’s analysis, about 6 in 10 vulnerabilities matched to certified products, smartcards aside, were disclosed after the certification date. She also describes sec-certs and SCRUTINY, two open-source tools her lab builds, and the one question buyers should put to a vendor.

Picture a procurement manager handed a 300-page certification document for a product they are about to buy. What do they do with it in the first hour, and at what point does the document stop helping them?

Luckily, certification documents (e.g., Security Target in CC or Security Policy in FIPS-140) are usually up to around 100 pages, not 300. But that’s still a lot for a procurement manager who isn’t typically a security specialist. In the first hour, I think they’d look for a few things: the exact product name and its version, the assurance level (along with the considered threats, assets and covered security requirements), what the evaluated configuration is, and what’s declared out of scope. That’s where the real boundaries of the security claims are. Covering all of those could take more than 1 hour if the procurement manager is not familiar with the certification scheme details (e.g., how the documents are organized).

And then the document would stop helping for the reason that it’s a snapshot of the product in the past. It tells you what was evaluated at one point in time. It doesn’t tell you what’s been found since, or what’s been changed. In our 2024 analysis, roughly 6 in 10 of the vulnerabilities matched to certified products, smartcards aside, were disclosed after the certification date. Also, the single given certification document can’t tell you whether the certificate is still active, or how the product compares to the two others on your procurement shortlist. Those questions go across multiple documents.

“Certified” sounds like a verdict. What is a Common Criteria or FIPS 140 certificate promising, and what do buyers commonly read into it that is not there?

Yes, I absolutely agree that for the average person “certified” sounds like “it has been verified and it meets all the necessary criteria for this kind of product”. But in reality, it’s not exactly the case.

Let me take Common Criteria first, since it’s the one most of our data is about. A CC certificate says that an accredited lab evaluated a defined product, in a defined configuration, against security claims the vendor wrote down in a document called the Security Target. Very often that target is based on a Protection Profile for the product type, like smartcards. So “meets the criteria” really means “meets the criteria the vendor claimed, within the scope that was defined”. The certificate also says the evaluation followed the process at a stated assurance level, the EAL. The promise is about the claims and the process, not about the product being secure in general.

FIPS 140 is a different thing. It validates a cryptographic module, meaning the hardware, software or firmware that does the cryptography, against the FIPS 140 requirements, at one of four security levels. The promise is about that module within its defined boundary. It says nothing about the application or the system around it.

What buyers commonly read into it, mostly about CC:

• “No vulnerabilities”: That’s not what it says. ROCA, Minerva and TPM-Fail were all found in certified products, some at high assurance levels.
• “The whole product is covered”: Scope can exclude features. We came across a CC-certified network camera where the network interface was assumed to be trusted, so it was probably never tested there.
• “The version I’m buying is the one that was tested, and it’s still current.” A CC certificate applies to a fixed configuration, and it has a limited validity. It’s usually 5 years, after which it gets archived.
• “A higher level means it’s more secure”: A higher level doesn’t mean a product is more secure in general, instead it tells you how thoroughly the product was examined. In our data, products evaluated at higher levels tended to have less severe vulnerabilities, but not noticeably fewer of them. So it helps a bit, but it’s not a guarantee.

When certification documents become searchable, which question can a buyer answer in minutes that previously took a consultant or days of reading?

A certification document is a snapshot. It can’t tell you whether the certificate is still active, whether there is a newer certification of the same product, or what the product is built on. Once the documents are searchable, you can check all of that quickly. You find the certificate, see whether it’s still valid or already archived, and see whether a newer version or re-certification exists.

You can also follow what the document refers to, like the certified hardware or library the product is built on, or the Protection Profile it uses as a baseline. Those are separate certificates and documents. Before, you had to know they existed and dig them out yourself. This matters because what’s underneath the product is part of what you’re trusting. Search gets you to the right documents fast, but you still need to read what they say.

For readers who have not come across sec-certs, how would you describe it in a few sentences, and what problem was it built to solve? Then, who opens it first: vendors checking their own entries, buyers vetting suppliers, or security labs? Did the answer match what you expected when you started?

Sec-certs is an open-source tool and website that turns public certification documents into a searchable, linked dataset. That covers Common Criteria, FIPS 140, and EUCC. Every week it collects the documents from the official portals, extracts the key data, links certificates to known vulnerabilities, and detects which certificates reference each other. You can use it on the website or as a Python library.

We built it because all this evidence is public but practically unusable. It’s in thousands of long PDFs on many national portals, with no shared identifiers and no link to vulnerability data. The idea came from researchers in our lab (Centre for Research on Cryptography and Security, CRoCS) who needed exactly this during the ROCA and Minerva disclosures (led by my colleague Petr Švenda) .

As for who opens it first, it’s not really one group. From the feedback we’ve received, they all use it, but for different reasons. Vendors check their own entries. They also look for other vendors’ products that depend on theirs, because certificates reference each other. And they look at what their competitors certify and how they do it. Buyers come when they’re looking for products that use a specific technology, and then they compare the certification documents of those products. Meanwhile, security labs check which certificates in the categories of their expertise will expire soon and through this find the potential customer for testing as a part of re-certification.

And regarding whether it matches what you expected – I’d say that yes, in general. But at the moment, as we don’t track systematically who the users are and how they are using the tool, we have only a limited view from the users who approached us. But thanks to the CCAT project, the main focus for our team is to get more feedback from our users on how to make the sec-certs even more useful and provide more data to confirm or reject our more nuanced assumptions about their needs.

A vendor discovers that its certificate can now be searched, compared and cross-referenced in public. What reactions have you seen, and what do they tell you about how certification is used in the market?

The reactions are mixed, and I think that’s an indication that the tool meets its goal of improving transparency in the certification ecosystem.

Some vendors see it as a positive. They can show potential customers that their product has a “page” in sec-certs, which is an independent source, a non-profit research project. So customers get a better overview of the product’s certification than they’d get from the vendor’s own materials, the verifying authoritativeness of which requires, again, going through various certification portals.

On the other hand, we don’t only make certificates searchable. We also map them to vulnerabilities that potentially affect the certified products, and those matches are correct only with roughly 89% precision. The reason is that the certification scope is often defined ambiguously, and it’s hard to match to CPE, a structured, machine-readable naming standard for identifying software, operating systems and hardware platforms. We highlight that the detected matches are indicative and should be verified. Still, vendors are not happy when their product is shown with potential vulnerabilities.

What I take from it is that vendors like the visibility when it supports the certificate as a trust signal for customers, and like it less when the same page also shows potential vulnerabilities next to it. It also shows how much the certificate is treated as a status that customers can rely on.

SCRUTINY is the second tool your team builds. How would you explain to someone outside cryptography what it tests and why that matters for a company buying products with cryptography built in? Suppose it flags a statistical anomaly in a random number generator. A buyer might hear “broken,” while a lab might call it noise. How do you present a finding like that so people neither panic nor shrug?

SCRUTINY is a toolkit that tests the real product. You connect something with cryptography built in, like a smart card, a TPM chip, an encrypted disk or a cryptographic library, and it runs a set of tests. Which cryptographic functions does it really support, compared with the datasheet? How does it react to odd or malicious inputs? Does the time it takes to compute something leak secrets? Are its random numbers really random? The results become a profile that you can compare against a reference and turn into a readable report.

For a buyer, this matters because a certificate tells you what was evaluated on paper. Cryptographic flaws are also serious but invisible from the outside. Minerva is an example. It affected certified cards, and the weakness was in the timing of signing, which you find by measuring.

When it flags a statistical anomaly, I wouldn’t say “broken” or “noise”. I’d say what we know: “In the data we collected, this test found a deviation, and here is how strong the evidence is.” Then I’d say whether it repeats on fresh data, and what it would mean for how you plan to use the product. I’d also use simple levels: nothing found, needs follow-up, and reproducible with strong evidence.

SCRUTINY is a tool which helps you verify whether the product you are buying performs as good as the one that was tested for certification. And of course, if the buyers don’t have in-house expertise for independent verification of the difference between the claims certified and purchased products, it would be in the vendor’s best interest to provide the baseline measurements as the proof.

If a company reads only one thing before signing a contract for a certified product, what should it be, and what is the one question it should put to the vendor?

The one thing to read is the scope of the certification: what exactly was certified, in which version and configuration, and under which assumptions or exclusions. That is described in the Security Target (or Security Policy), and the certification report summarises it. “Certified” tells you very little until you know that.

The one question for the vendor: “Show me where in the certification documents the security function we’re buying this for is covered, for the exact version you’re delivering.” If they can point to it, you’re in good shape. If they can’t, or the answer is “that’s outside the scope”, you’ve learned the most important thing before signing. That is also when independent testing, like what SCRUTINY does, is worth considering.