How researchers are teaching AI agents to ask for permission the right way
People are starting to hand more decisions to AI agents, from booking trips to sorting digital files. The idea sounds simple. Tell the agent what you want, then let it work through the steps. The hard part is what the agent does with personal data along the way. A new research study digs into this problem, and asks a basic question. How should an AI agent know when to use someone’s data without asking every time?
What people share when the agent asks
The team built a large user study to understand how people judge data sharing in different situations. They created a website where participants interacted with a simulated assistant that asked for permission to use pieces of data tied to tools such as travel booking systems, calendars, or financial apps. The researchers wanted to see which data people were willing to share, which they rejected outright, and how these decisions shifted when the agent made mistakes.
A few patterns jumped out. About 95 percent of participants granted automatic sharing at least once. At the same time, many pulled back when the system presented data that did not belong in the scenario. When the assistant introduced unnecessary data, the share always choice dropped, and more people selected never share. The shift was noticeable, with share always dropping from roughly 83 percent of users to about 74 percent. This shows that mistakes trigger a protective instinct.
When convenience meets caution
The study found signs of both over permissioning and under permissioning. Many people handed over data that the task did not require, often because the information seemed harmless or because they assumed the agent needed it. About 90 percent of participants shared unneeded information in at least some scenarios.
Under permissioning appeared mostly with highly sensitive information. Social Security numbers, bank account details, and child names fell into this category. Participants withheld Social Security numbers almost half the time, even in tasks where the number would be necessary. The researchers noted that people often stayed cautious when the data touched on financial or identity related matters.
This tension between convenience and caution opens the door to new risks when such systems move from controlled studies into production environments. Brian Sathianathan, CTO at Iterate.ai, said the risk extends far beyond the model itself. “Arguably the biggest vulnerability isn’t so much the permission system itself but the infrastructure that it all runs on. If you’re deploying automated permission inference on shared GPU clusters, you’ve created a massive attack surface.” He warned that attackers could learn how a company handles sensitive data simply by observing inference behavior. “Attackers don’t need to break the model because they can reverse engineer your security posture by observing inference patterns across multi-tenant environments.”
Trust changes with each task
The team also looked at how decisions change in different settings. Entertainment tasks, such as music or movie recommendations, drew the highest rates of automatic sharing. Around 56 percent of participants chose share always in that domain. Finance tasks sat at the bottom with only 22 percent.
The same pattern held when the researchers drilled into specific tools. Within the travel domain, people were willing to let the assistant access weather information but resisted when the assistant tried to retrieve passport scans from cloud storage. Context matters. People fine tune their trust based on what feels safe and expected.
The study also showed how easy it would be for attackers to push these boundaries. Sathianathan pointed to prompt injection as a practical concern. “One of the more immediate threats here is permission drift through prompt injection. These systems constantly learn from context like user queries, tool descriptions, documents, and so on. An attacker embeds instructions in a PDF or tool schema that subtly shifts how the agent interprets necessary data.” He said this can make the permission model think it is following user preferences even though it has been manipulated.
Teaching AI to learn your rules
The second part of the research explored whether an AI system could learn these patterns well enough to make permission decisions for users. The team built a prediction model that combined individualized learning with trend analysis across similar users. When tested on more than seven thousand permission decisions, the hybrid system reached about 85 percent accuracy across all predictions and more than 94 percent accuracy when it limited itself to high confidence cases.
Accuracy alone will not solve security concerns in sensitive fields. Sathianathan said organizations need to treat permission inference as protected infrastructure. “Mitigation here, in practice, means running permission inference behind your firewall and on your hardware. You should treat it like your SIEM where things are isolated, auditable, and never outsourced to shared infrastructure. You can’t let the permission system learn from unvetted data.”
He added that sectors with strict rules will face extra challenges. “The paper shows that collaborative filtering can predict user preferences with high accuracy, which is good, but the challenge for regulated industries is more in ensuring that compliance requirements take precedence over learned patterns even when users would prefer otherwise.”
Ready to dive deeper into AI security strategies? Download Delinea’s comprehensive 2025 AI in Identity Security Report to discover the latest insights and best practices for securing AI in your organization.