Audits for AI systems that keep changing

Security and risk teams often rely on documentation and audit artifacts that reflect how an AI system worked months ago. ETSI’s continuous auditing based conformity assessment specification (ETSI TS 104 008) describes a different approach, where conformity is evaluated through recurring measurement and automated evidence collection tied to live system behavior.

AI continuous auditing

The specification addresses a common challenge in AI oversight. Models evolve through retraining, data pipelines change, and system configurations shift during operation. Oversight methods built around periodic reviews struggle to track those changes in a consistent way. Continuous auditing based conformity assessment, known as CABCA, treats change as an expected condition and builds assessment processes around it.

Treating assurance as an ongoing activity

CABCA defines conformity assessment as a standing operational function. Assessment runs in cycles that repeat throughout an AI system’s lifecycle. Each cycle gathers evidence from system artifacts such as logs, test results, model parameters, and data samples. Automated analysis compares that evidence against predefined requirements and metrics, producing an updated conformity status.

Assessment cycles begin through defined triggers. Some triggers follow a schedule, such as regular reviews tied to time intervals. Others respond to events like model updates, data drift, or performance anomalies. Each trigger initiates the same core steps, creating a consistent flow from measurement to reporting.

This structure aligns assessment activity with how AI systems operate in production environments. Monitoring, evaluation, and reporting occur alongside development and operational workflows rather than outside them.

From high-level rules to measurable checks

A central concept in CABCA is operationalization. Organizations start by identifying the requirements that apply to a specific AI system. These requirements may come from legislation, standards, internal policies, customer obligations, or sector rules. The scoping process consolidates those requirements into a single conformity specification.

Operationalization translates that specification into quality dimensions, risks, metrics, and measurement methods. Quality dimensions include areas such as accuracy, bias avoidance, privacy, accountability, and cybersecurity. Each dimension links to identified risks that describe how the system could deviate from expectations. Measurable requirements and thresholds are then derived for continuous evaluation.

The result is a set of machine-readable metrics that assessment tools can track automatically. This creates a direct connection between abstract obligations and observable system behavior.

Continuous evidence and reporting

Evidence collection under CABCA is automated and persistent. Measurements run continuously or at defined intervals and feed into an assessment engine. The engine evaluates results against thresholds defined during operationalization and produces findings that map directly to specific requirements.

Reporting follows the same cadence as assessment. Conformity status updates reflect current measurement results and link back to supporting evidence. Reports persist over time, creating a record that shows how conformity status changes as systems evolve.

Follow-up actions integrate into the same cycle. When teams apply corrective measures, subsequent assessment cycles evaluate their effect through updated measurements. This creates a feedback loop that ties remediation to verified outcomes.

Assessment paths and external review

CABCA supports multiple assessment paths. In a self-assessment path, the AI system provider reviews assessment results internally and records conformity status. This path suits organizations with internal assurance functions and established governance structures.

A third-party assessment path allows external auditors to access assessment reports and evidence. The specification supports programmatic access through secure interfaces, enabling automated review and status updates by external systems.

The framework also supports certification use cases. Continuous evidence streams produced through self-assessment or third-party assessment can feed certification processes. Certification bodies evaluate conformity based on current data rather than fixed review windows, supporting certificates that reflect ongoing system behavior.

Defined roles and accountability

The specification assigns explicit roles within the assessment process. The auditee, typically the AI system provider, manages scoping, operationalization, infrastructure, and execution of assessment cycles. The auditing party evaluates evidence and determines conformity status based on the selected assessment path.

Risk ownership forms part of the recorded assessment data. Each AI system has named owners responsible for mitigation decisions and resource allocation. Ownership information persists alongside conformity status, supporting traceability and accountability.

Alignment with regulatory obligations

CABCA is designed to support regulatory frameworks that require ongoing oversight of AI systems. The framework links risk management activities, technical documentation, quality management processes, and post-market monitoring through a shared evidence base.

Technical documentation and quality management artifacts draw from the same measurements used in assessment cycles. This creates consistency between operational data and formal conformity declarations. Evidence collected during operation feeds both internal governance and external review processes.

“This latest Technical Specification is founded on a firm principle: trustworthy and accountable AI can only result from practical, auditable, lifecycle‑long compliance,” said Jürgen Großmann, Chair of the AI working group of ETSI’s Technical Committee for Methods and Testing for Specifications (MTS AI) “In laying down a clear framework for CABCA, ETSI bridges the gap between high-level legal obligations and the realities of modern AI systems in the field.”

Read more:

More about