AI Security Governance and Alignment

A little bit about myself, cybersecurity background, about 15 years, and built critical systems for companies like Nokia, Bloomberg, and a couple of other ones. And I currently live in Portugal, so I also offer consulting services to companies within Europe and also back in the States.

My talk today is your AI said no, then it did it anyway. I'm sure we're all familiar with this, so I will try to keep it light, maybe engaging and fun, and I hope you enjoy the talk.

Okay, let's go next.

When AI Can See, Reach, and Act

So we're gonna do the survey. If your AI can do, let's start with the first one, It sees data. By data, I mean that it reads some data you have in Excel sheets or conference documentation or database. Raise your hand if your AI reads data. Nice. Okay, that's a good one.

How about it's able to read outside data. That should be data like internet calls or somewhere outside your environment. Great.

right the third one it can act of course it can act we all love to get our AI systems to do stuff so we can go do the boring things that we love to do as humans right for me it's I love to race cars so I have these awesome systems that just do things and I go do my own thing okay now if you

put all three of these in one agent and that becomes for us the threat vector or the attack vector. I'm from a cybersecurity background so I'm always thinking how to break stuff so a lot of the terms I will be using today would resonate more with breaking stuff and how to build stuff a lot more secure or even secure stuff. Okay.

The Agent and MCP Attack Surface

Now, I will start by scoping this talk to agents and MCPs. We've had such an awesome talk from the previous speaker, the Anatomy of an Agent. Awesome stuff.

So, we're all familiar with agents and we all can understand how agents operate. So for me, the focus is at the heart of an agent and at the heart of an MCP server, as a threat actor, how do I look at this?

Two Doors Into Agentic Systems

So the two doors that grants access from a threat actor into your agent or your MCP is one, the schema field. Now, every MCP server has a schema field. If you've seen an MCP server before, can I see a hand? Use the MCP server. okay great and usually you have like a description field and a couple of fields where you can fill the llm field stuff into there then it does stuff gives you feedback that's the first

door the second door is the store now this is very common with agents agents can write and read stores the previous speaker spoke about like your cloud md file or a bunch of md files where these agents read stuff from to act and for us as trade actors as you mean the role of a trade actor today we love that kind of thing because we can actually drop stuff there then your agent would just do what we wanted to do know what you wanted to do now these two doors they are not new some 15 years ago

20 years ago this had been the major problem of web applications if you're familiar with web applications, you must have heard stuff like injection flow, maybe cross -site scripting or SQL injection. It's common. And these type of attacks, it's not new.

But with MCPs and agents, they are reinvented in a way where the trust model is what we're going to focus on today.

Weaponizing Agent Loops and Shared Knowledge

Now, why do I call this the trust model? Because with MCPs and agents, out of the box we allow them to do things there was a question asked the previous speaker do you have some guardrails or some control by default only few people tend to define those things we just let them go rogue because we want them to be able to do stuff by design okay then this is what we decided

to do now this was something that we built from an incident that happened recently. Anybody here familiar with the OpenAI Hugging Face incident that happened? Oh yeah, that made the news and we thought we could actually build that capability into our testing framework and what we did was we took the same logic and designed a loop system and this is actually something that was explained by the anatomy of an agent as well.

You have the ask, act, report, repeat. Let's have a look.

And for every agentic system, usually they tend to follow a pattern that looks like this. So we thought we could also weaponize that into an attack framework where we have this swarm of agentic systems that can do stuff, but they can also share knowledge.

Now knowledge was one of the things mentioned by the previous speaker. So if I have a couple of agents that can all read from the same source, it means if one of them finds something important, all the other ones can read from there. Hence, they save me time and they can go faster in my attack queue chain.

That's the concept we built here. I'm going to show some examples of how we were able to test this in real environment.

Testing for Real-World Agent Compromise

Now, some of the things you'll see here, they were from client engagement, of course. We've been able to redact or clean up to make it easy for presentation purposes now what

we do we don't ask the AI if it got hacked no we actually want the receipt because clients want to see if this is actually an issue and I'll give an example two months ago we had a client who was supposed to release an MCP server on cloud marketplace of course cloud marketplace open a marketplace this MCP

server reads dockets it's a legal mcp server means it's able to read dockets then lawyers can actually interact with it to maybe check for file cases and stuff like that and we were brought in to actually test the mcp server and we thought typically what people do is you just throw in a bunch of stuff and see how the mcp responds to them then maybe you say oh no it's good or bad

but we decided to go a different route right we designed a system where we can implant a beacon and each time the MCP server calls the beacon in our requests we can have a receipt to prove that something was done by the MCP server so what you're going to see next this example here is hey can we have a way to validate each time

maybe the MCP server is supposed to read a file and give a respond but if I I inject a white text or white label text in that instruction, can the agent read that and call to my beacon, the beacon you see in there is something that we host, then when it's serving you the respond, it's going to serve you a benign respond, like nothing happened. But in the hidden instruction, it actually did something, which is not seen by you if you're not really looking at that.

Stored Instructions and Indirect Injection

What you see as an example, just to put that into perspective, the first one you see there is the stored and served. You would see that this is like a GitLab MCP or a GitHub MCP. There is a comment on a merge request, and it's pretty straightforward. The agent rejected the comments.

What we did was we read the schema. schema, we injected an instruction in the comment field for the schema. Then the LLM read that schema, but rejected to carry out the instruction. But it actually created a note for itself and stored the notes. So that was what we needed to prove, because each next time we call that same two, it reads that note first before it does anything. We've already injected it.

now this is something what most people call indirect injection if you've heard that term before around agentic system because the instruction is hidden stored then every time the tool is invoked the instruction is actually read and carried out by the LLM now LLMs by design they have guardrails they're trained with guardrails to make them a little safe that's why they can tell you no I cannot not do this stuff or yes I can do this stuff but when the instruction is actually hard coded in a two schema it is hard for the LLM to actually reject it and from analysis so far this has been one of the easiest way to get around the guardrails for this kind of systems now to the right this was

Trust Failures Between Agents

actually a financial client they had an A to A type architecture which was why I asked the question I asked, right? Where there are multiple agents that can do stuff. Now, agent A can delegate to agent B. There's a trust boundary between these agents.

Now, the trust boundary between the internal system is different from the trust boundary from external to internal. Example, as a user, if I need to interact with the agents, I need to authenticate with some form of identity, entity, right? This is me. That identity should actually bound to an authorization, what I can do as a person.

But within the agents themselves, it's usually flat. Meaning agent A can invoke agent B or delegate to agent B, and agent B has no way to verify if that that instruction is coming from a clear, approved agent.

So what we did here, as the user, we decided to try to test that A2A system. We asked it a question, hey, why are X amounts to an account? Now from external to internal, it rejected it.

But on the second test, we decided to assume a role of an agent within the trust boundary. And we asked exactly the same thing, and it just did it without restriction. Now, what this brings up is the trust concern, right? Which is what we're trying to narrow down to.

Rethinking Trust Boundaries

The fact that trust used to be, okay, I design a boundary and say everything outside is untrusted, right? Everything inside is trusted. But what I'm trying to point out here is nothing inside should be trusted.

Because if an agent would retrieve a data from a data store, So you need to have a way to validate that data has not been poisoned, one. You need to have a way to validate the agent has the right identity to retrieve the data.

Accountability and Non-Repudiation

Now, some of the things I'm talking about are things that a lot of companies will have to pay from December because EUAX is coming for everybody. And I've done a lot of talks around that for some companies to really understand the risk around things like this. This could be technical, but it doesn't go beyond what the CEO would be asked. Hey, show us that you're actually looking at what your agents are doing. Are they assuming identity? Are they carrying out instructions? Can you prove or show us what exactly they are doing?

We call it non -repudiation, being able to prove that someone did something and you gave the authorization for that thing to do it. I will try to go all the way fast.

This is just from some of the research we've done so far.

Supply Chain Risk in MCPs and Agents

Now, every tool you plug in a software you never reviewed becomes like, hey, an attack surface.

Anybody here familiar with supply chain as a concept? Supply chain, yeah, I think most, yeah, great.

It's not just in software engineering. Supply chain could be in manufacturing industry as well, where you trust these set of clients. And by default, because you trust them, you don't have to validate everything they do in your environment. That trust is transferred automatically. They assume the trust and they do stuff. With software, it's the same thing.

We all build using cloud code to an extent, and maybe you're building a TypeScript project. There will be like a manifest file with a bunch of dependencies and libraries. We don't check those things, by the way. It just builds, and it works. That's the focus.

Now what I'm trying to point out here is the fact that five years ago that used to be the key chain, hey we have tools that do software composition analysis we are scanning every dependencies and libraries in our code, but today nobody is scanning MCP servers, nobody is scanning agents, there are a bunch of these open MCPs and agents out there, we just clone them or we just use them.

I'm trying to create the awareness that hey that is a risk you are assuming even if you don't know that is a risk because if that third party too is compromised automatically whatever you have is compromised that's what i'm trying to point out here i will go to the next slide now um this is just to show

Trust Depends on Data Sensitivity

that sometimes um we tend to look at trust through a different perspective and trust could could mean different things for different products.

Example, if I'm building a software that the data types of classification that I handle is just random numbers, no PII or PHI, I don't need to design a complex trust model because, hey, it's just some random numbers.

But if I'm building a system that would collect maybe employee details and maybe some PHI, health details, then trust is looked at differently, right? Because then I have to align with frameworks frameworks like HIPAA or maybe GDPR or any of these frameworks. They define what trust should be in such an environment.

In today's world, where there are so many agents flying around on MCPs, no trust. The trust boundary has been eroded to the point where I do that sometimes.

I'm also a corporate here. I tend to pull stuff because I love to chase knowledge. So there's something new in the block. I just pull it from GitHub into a sandbox environment, and I want to test it.

By design, we tend to trust that the people who release these things, they are good people and they have good intentions. But with MCPs and agents, if you trust an open source MCP in your environment, automatically you are assuming that whatever happens, you are fully responsible for that.

Now I'm speaking in business terms. If that MCP gets compromised and you need to notify your clients, you will be litigated for that because it's your due diligence to test that external entity you bring into your environment.

It's also your due care to ensure that you validate what class of data this system is allowed to read.

I'm going to actually just go to the next slide.

Why Guardrails Alone Are Not Enough

for these I know the previous speaker also spoke about like some questions around guardrails telemetry having a way to view what these systems are doing for our security we say a guardrail is just a telemetry it's not a security control again most people will say design a guardrail put a guardrail there then you go back to sleep no it's not enough because with guardrails what

What I've seen in the industry, most people rely on LLMs to build judge systems, right? That's your guardrail. It looks at the inputs. Is this okay? Yes. Allow. No. Drop.

Now, it's been proven over time that that is easily bypassed, and there are several research in that space. I'm not going to bother you with that.

Context Windows and Alignment Bypass

We've even done research of our own, which I'm going to share, where the context window is the game now, right? with alignment it is tied to context meaning if I start sending requests to the LLM over time I can exceed the context window where the alignment is then it totally become unaligned and I can get it to do malicious stuff some of

the research we've done shows that with different models it's a lot easier to get around them but when also when the model is powered with tools like MCPs and maybe a mixture of agents and stuff like this. It's even a lot more easier to get around them.

I'm wrapping up. This is the research I spoke about. This is going to be published.

Of course, if you want to get into the research and then dig in, we can share this on LinkedIn, maybe connect later.

I would like to show you what the data shows. But some of those models you see in there, they are open -weight model, where we can now understand that multi -turn makes it easy to get around alignment. and get the model.

I build offensive tools and I want LLMs to be offensive. And it's hard because these companies that train this model, they build alignment guardrails. But of course, I still get them offensive because I understand that it's tied to context. Now, why am I saying this?

Owning AI Risk Under the EU AI Act

For those who will be leveraging LLM to build product -to -power products, which is what we see today. Statistics shows that more than 80 -90 % of the products that we're seeing in emerging products, they all built on some LLM.

right so you're going to be leveraging LLM to build this kind of products you need to understand that you don't push that risk to the model providers you own that risk because if you do then if the traductor is able to bypass the guardrail you will be the one to pay the fine not the model provider according to EU AI act if you use an AI model either you use or you train or you retrain you are responsible for it so if you get compromised you pay the fine not open ai

Defense-in-Depth Recommendations

or not entropic just want to point that out there wrapping up now um these are some recommendations that we've actually worked with companies to implement because we preach defense in depth as a strategy means it's not one control it's a layered approach now if you build layer defense defense, if one gets bypassed, you have several other defense to actually compensate the control.

Monitor Tool Specifications and Changes

So the first one I have there for MCPs, we recommend always take a snapshot of the two spec. Now, when you launch an MCP server, you see the two spec, you want to take a snapshot of it, because then you can monitor whenever there's a drift to say, hey, this is a potential indicator of compromise. I want to maybe roll back the old hash that is good or just completely take this down.

Use Deterministic Request Controls

The second recommendation is you need to have some form of shield. We spoke about that guardrails, prompt shield, MCP shield. They have different names. You need to have them in front of your products so that when the request is going to your product, it first goes through a deterministic control. Now, it's a lot more easier in today's world. We were just talking about deterministic models where it's a yes or no, hence you don't need to reinvent the wheel, the model decides this is good, this is bad, and it just moves on.

Protect Stores and Define Internal Trust

The third recommendation, that's a lot more technical and it's because when you're looking at attacks like store and serve, there has to be a store. So for you to prevent such attacks, you need to have classifiers. It's a bit be technical. I'm not going to go into that. Classifiers determine what is a write, what is a read, but that's just a technical term.

Then the final one, which we also preach, there has to be some output gates. I mentioned earlier that trust used to be external to internal boundary, but today within agentic world, trust has to be looked at internal. Agent A, agent B, agent C, the trust boundary between them must be clearly defined and there must be control to actually validate that that the trust is verified and it's actually as implemented. That's the last one.

And my final slide, I think this is just shameless marketing, it's this one.

Questions Leaders and Builders Must Ask

Now, this is something I've spoken about. The EUI Act is coming for every one of us. There are already companies that have been actually litigated and fined based on this.

So if you're building products that are leveraging LLMs, if you have product that is already powered by LLMs or some model you need to be aware of this and the next time you go to the board or you go to your engineers you need to start asking them the right questions and some of those questions I've captured three yeah this is why and it's the first one for

Know Which Systems Can Act

everyone at least you should be able to name which of your AI systems can and actually take action. Just the head knowledge, hey, Agent A can act, can delegate, can do X, Y, Z. You need that knowledge.

Track Where Agents Read and Write

Number two, this would be for engineers and builders. You need to be able to understand where your agent writes to. Is there a RAC store? Is there an MCP source? Is there a memory, a Cloud MD or a Crossover MD?

You need to be able to validate that store and ensure that every time the agent updates the file, you are aware of it and most times we don't you mention a bunch of empty files if you take a snapshot of the empty file every day changes because the agent updates the file as it learns or encounters challenges now you need to have a way to track that for you to stay safe then the final one would be for

leaders I usually would tell DC executives make sure you're asking the the right questions, at least having data to show that, hey, we know what we're doing. So when they come for you, you have some proof to show that I'm aware we designed the system, the system operates this way, the system writes or reads data from the store, and we know what it does.

I think that's my last slide.

Conclusion: Refusal Is Per Session

In conclusion, refusal is per session. The store outleaves the session. Now think on this.

If the LLM tells you, no, I cannot do this, if you try that the hundredth time, it's going to say, yes, I can do this. That means the refusal is just for that session. The next session totally overrides that refusal.

Hence, when you're designing control, you must have that in mind and put a deterministic boundary within each request that you make.

Thank you. And you can connect on LinkedIn if you want to. That's my talk.