AI Connectivity and the Kong AI Gateway

My name is Roberto. I am a software engineer at Kong.

At Kong, we do something that we call AI connectivity. And AI connectivity is all about connecting and empowering large language models capabilities through the use of network stacks and connectivity layers.

What I've set up for you today is a short, practical demo of me literally coding with AI to show you you how the Kong AI Gateway impacts day -to -day usage of AI for developers, but also for autonomous agents and production grade workflows.

Introducing the OpenCode Coding Agent

What I will show you today is a coding harness called OpenCode.

I have set up some configurations that will help me get rid of my personal configuration operation, because it's very bloated and full of stuff.

But this is literally just an agent. This is what we consider an agent. An agent is basically an harness. An harness is a set of tools that executed in an environment.

It's capable of generating any type of artifact. If you denominate it with coding harness, it means that it can generate code. It can help you generate code, understand code, code and work with it as a developer.

Given context with it, you call it an agent. That's at least my definition.

So what you can see there in the screen is you see a language model name and OpenCodeGo, which is the subscription I'm using. OpenCodeGo is a provider that I will use to showcase you a bit of this demo.

Exploring a Project with AI

So I will ask the AI what is in this project. project, maybe with one sentence, and we'll try to use the tool it has in the environment it is in right now, and try to get, give me a response as I asked it to. So okay, so the response is, this is a small Fastify TypeScript API of some sort.

So it was capable of telling me stuff, so basically what we did was having a customized tool that is able to call provider APIs, and between me and the provider APIs, there is nothing that is able to like govern or steer the traffic or control if I'm sending sensitive data to the provider anything like that in most of cases this

is fine if you're a single developer you can act on using plugins and configurations that you install from the internet or whatever it's full of stuff on Twitter or X as you call it and other places where you can install pretty much anything now everybody builds software at the speed that is incredible but if

Why Organizations Need AI Governance

If you're an organization, this is not exactly the type of thing that you might want to consider best practice.

If you're an organization, you want to be able to manage costs, see what providers your developers are using, what data they're sending, and eventually also gatekeep that data. So you don't want your developers to send sensitive data to any provider.

Routing OpenCode Through Kong AI Gateway

So what I'm going to do now is show you another configuration that I have for open code. this is just json configs but what what it will do is basically spin up an instance of open code that is set up to talk to an ai gateway that is configured to be connected to this control plane this is the software solution that kong delivers to customers to handle their control planes configurations and configure models and stuff like that

Using Virtual Model Aliases

This is the model name I'm using, and this is another model that I've nominated, Banana Premium, because, you know, gorillas, bananas, that's all about it. But what you can see is that it actually, this model does not exist. It is just an alias, and you can actually see the target model and discover what model it is actually calling.

but from the point of view of your developers or your AI agents or anything that you use in production you can actually configure this model on the fly if you want to Say a new provider new Chinese providers ship the new flagship model you want to switch to that we support that you can Configure a new model just by the tip of a button or by calling the API as I will do later

so if I want to spin up the gateway instance of my open code configuration. You can now see that the model configured is going to pass through the Kong AI gateway and this is just gonna work fine as it was before.

What is this project about? This is just a matter of configuration. I did reboot stuff but you can do it on the fly like you can configure providers as easy as it is to type in it in this in the software.

What you're going to see is pretty much the same response as before, giving it enough time. Yeah, so it's pretty much hanging, I think. This demo live effect.

In the meantime, anybody wants to ask questions or something? No? Okay, it is doing something at least. Fine. So it's the same response as before.

Switching Models Without Code Changes

What I would like to show you now is the capability I just explained like a couple of minutes ago. The ability to swap models behind the virtual model.

So if you go and list models in here, you see GorillaCoder and Banana Premium, the ones that we listed before in the same UI control plane that I showed you.

Running an Autonomous Long-Running Task

What we're going to do now is spinning some long -running tasks. task.

I'm going to use a command that is called LFG. LFG is a long -running task that does automatic planning, autonomous planning, brainstorming, code simplification, code refactoring, and ultimately security testing. It's a whole long pipeline.

It's a plugin developed by EveryYank. It's a US company.

But what I've got to tell you to do is research how to migrate this code base to Rustlang since I don't want it to pollute the current environment I'm gonna tell it do it in slash TMP so it's temporary directory so

I'm gonna let it spin and while it does it this is me my contacts I'm going to swap model in the control plane as it runs so I'm gonna set it to model deep seek it was previously it was gln 5 .3 so i'm going to set deep seek 4 .1 flash executed one changes

and you can see it is still running and if you go in the ui and refresh it of course you should be able to see it has changed the model to deep seek okay but the thing is still running still planning

and still working this means that yes this is just a coding session of a single developer you're seeing here but for any AI workloads that any company could have that is running live and continuously this means that anything that depends on that model can change without any code change and can can be governed from a source of truth that is different from the code base so yeah I

I think I'm just going to stop it.

Protecting Sensitive Data with Prompt Guard

Another interesting use case is what I've mentioned to you before is being able to gatekeep in some sort of way what is being sent to the provider that you're using. So I told you I'm using OpenCodeGo provider. This is the same provider that I wired into the Kong AI gateway.

What we'll try to do now is in the project that I showed you for what you could be able to see, I've set some environment variables that are actually in a .env file, and they are API keys that I don't want the provider to see.

So if I do make gateway again and I ask it to provide me and read me those environment variables, .env .demo maybe it's gonna take some time as before, but it pretty much gave them straight away if if the language model was able to respond me this way it means the provider has seen them this is not good we don't want our company secret stuff to go into some provider that you know we configure because our developer needed so we want a way to set up a configuration that basically tells the

AI Gateway to ignore that and to do it we have something that we call AI prompt guard and what it does is basically inspects all the prompt that we sent to the provider and basically check if there's anything that we are guarding against for example if I show you just this little bit of configuration I hope I hope it makes sense for at least someone.

But this is what we require to basically deny any pattern. And we say, anything that starts with combat and secret should be blocked. So the request should not arrive at all. It should not even start going to the provider. And that is what we want.

So could you do this again?

And we get prompt pattern is locked. This is not anything about open code or anything. This is coming from the upstream, stream, which is the Kongi iDeeply.

Conclusion

I hope I was clear enough. I thank everybody for listening.

I know this stuff may be a bit confusing, a lot of text, me doing stuff, even getting stumbled in a way because of the live demo. But I hope you enjoyed it and you understood a little bit of what we do at Kong, what we develop at least. And thank you, everybody.