Google’s $10,000 refund test shows why AI agents need zero trust

Google’s open-source autonomous Customer Support & Returns Agent, built using the Agent Development Kit (ADK) and Gemini, demonstrates how developers can apply zero-trust security principles to AI agents that interact with sensitive systems and take real-world actions.

The project tests an approach that assumes an AI agent could be manipulated or compromised and puts security controls around it to limit what the agent can do.

The architecture uses safeguards outside the model to verify actions, restrict AI-generated code, and block potentially dangerous requests.

Source: Google

The customer support agent shows why those controls matter. During normal operation, it reads a customer’s return request, generates a Python script to calculate prorated restocking deductions, records an approved refund in a database ledger, and provides a confirmation.

Google demonstrates an attack in which a customer with a $149 order instructs the agent to issue a $10,000 refund and run Python code that exposes environment variables. An agent using a shared database connection and executing code in an unisolated environment could authorize the payment, expose API keys, or compromise the host server.

Security outside the AI model

Google’s design treats the model as a component that could be tricked or jailbroken. A system prompt telling the agent never to refund more than an order’s value does not provide a hard security boundary. Prompt injection can bypass such instructions, while prompt tuning and model updates can change how the model responds.

The reference architecture uses three security layers outside the model: cryptographic signatures for database changes, isolated environments for generated code, and a Semantic Gateway that checks inputs and actions against deterministic rules.

Each state-changing database write is signed by the agent responsible for it, and the database verifies the signature before committing the transaction. This provides cryptographic attribution linking a database mutation to the signing identity associated with the agent and makes subsequent tampering detectable.

For production deployments on Google Cloud, Google recommends assigning each agent its own service account and granting it signing permissions on an asymmetric key in Cloud Key Management Service (KMS), backed by Cloud Hardware Security Module (HSM). In Google’s proposed configuration, the private key is generated inside the HSM and does not leave it.

The open-source demonstration uses an HMAC key to simulate Cloud KMS for local testing. An independent background audit can verify the integrity of database records. If an attacker changes a $149 refund to $10,000 directly in the database, the signature does not match the modified payload and the audit raises an alert.

Containing AI-generated code

Code execution creates another security risk. An autonomous agent may generate Python for calculations, data parsing, or log processing. Prompt injection could direct that capability toward code designed to extract environment variables and API keys or connect to an attacker-controlled server.

Google’s example executes generated code inside a gVisor user-space sandbox with network access disabled and limits on memory, CPU use, and execution time. The sandbox isolates generated code from the host and disables network egress, while a five-second timeout terminates runaway execution.

Blocking dangerous actions

The third layer uses a Semantic Gateway between the agent and the systems it can affect. The gateway applies deterministic checks to incoming prompts and outgoing tool calls before the model is invoked or database updates are executed. These checks can identify credit card numbers and secrets, match specified jailbreak patterns, and enforce transaction limits.

A request to ignore safety instructions and issue a $10,000 refund, for example, can be blocked before the action occurs. Google recommends treating these policies as software contracts and using automated tests to ensure safeguards continue working after prompt changes or model migrations.

For production deployments, the relevant services can also be placed inside a VPC Service Controls perimeter, adding a boundary designed to prevent data exfiltration across the project perimeter if an agent workload is compromised.

“Building autonomous agents does not require accepting unconstrained risk,” Google’s Shubham Saboo and Eric Dong wrote. “By moving security boundaries into hardware-backed identity, user-space kernel sandboxing, and deterministic input/output validation, you help allow the model to handle dynamic reasoning while the underlying infrastructure enforces strict limits.”

The open-source reference implementation can be tested locally, including attack scenarios designed to exercise the security controls. Developers can also run a browser-based Live Attack Playground and use Google’s ADK documentation to begin building agent tooling and sessions.