Building Self-Improving AI Agents

You also asked all the questions I wanted to ask to start. Just one question who uses Cloud Code, Codex, coding agents on a regular basis? Okay good.

For the people who are beginners I'm a bit sorry so we're gonna go into a tech a bit. I'm gonna give like a little vision outlook how the future is.

So I'm gonna dive into best practices in agent building and then I'm gonna basically tell you why agents are better in doing this than actual humans.

So I'm gonna give like a a little mission vision statement in the back at the end of the talk yeah then

From Early Language Models to Agent Development

let's get started so you guys all know I got access to GPT in 2019 for the first time yeah the response time of JetGPT was roughly one minute to two minutes so you type the query and it came back like forever I thought like god this is never going to be a theme we'll be like okay this is a nice tool then now seven years later i'm standing here and i'm basically part of this whole hype cycle bubble whatever and i'm building a venture in it so yeah kind of caught me a bit surprised this whole wave as well um but i realized the

last three years when we started building agents at my former company is that it took a long time to build agents just and the most important part was always the testing and the reliability liability.

The Challenges of Building Reliable Agents

So just assume you have, you build a product, a chatbot for your own company whatsoever. And at the beginning, you don't have any users, right? And you test yourself, you put in a prompt, you look at the result, you do it again, and then you kind of are in this loop.

Now, what happens afterwards is kind of, you have to optimize this if you want to go live, right? So you have to make sure this is credible, your users get the right answers, you have to make sure containment basically it's reliable yeah how much can it actually do autonomously can it actually refund something can it actually do multiple steps does it actually do what it's supposed to do and then obviously at

some point you're gonna get more users out there and then at some point you suddenly gonna have cost as an issue we always thought when we talked to clients cost is a huge deal then they all tell us nah we wanted to work first but yeah Yeah, so our first campaigns literally went down the flood. Nobody really wanted to optimize the agents for cost, but one and two was really like what people were looking for.

So I just described, so you're like the builder, you build your agent, does whatever invoice processing, any chat bot, any agentic company. So how was it? Maybe some of you people can kind of think about this or can feel this pain.

Why Manual Iteration Breaks Down

yeah you implement a change you kind of have an idea your user gives you feedback you test it you generate some samples you run like ten or five queries then you look at the results and you like you tinker with the prompt you tinker with the tool like basically with the harness as we call it yeah then tweak it by hand and then you do this again and again and again and after

three months you start seeing the same failures you have fixed three months ago because you started to forget why you put certain stuff where and so on and so on. So this is kind of how we built agents when we started two, three years ago. And then we decided kind of, okay, this is not how you should do it.

A Data-Driven Agent Development Lifecycle

So the value basically and the real information, like the basis on which you should build your agent is actually in your data.

the agent did so this is number one and this is going to be the basis for

Define the Agent and Its Capabilities

should kind of follow this agentic development life cycle you kind of start the agent it has kind of a spec you define okay it's role what it's supposed to do the tools one can access what it can't access kind of what databases what it can read all the information it should collect what should go in his memory etc etc then you start here on the on the right side and this is

Build an Evaluation System

kind of the important thing after you build it like you built the first version of the agent what you really have to do is create the eval system so this is the third thing you should do this eval system is going to be the basis for all further development so what is an eval system it's basically when you come from normal software development it's writing test cases but in this case you want to write test cases that tell you if something in your your agent does is correct

or incorrect so when the next time you make a change you press the button and you run 100 examples you actually get the result of your evil system telling you hey did it did the right thing and did it do these 10 things correctly yes or no and it's really important when you build the evil system to focus on easy questions don't don't build your evil system around was this true or is this this polite or is this like this.

Make it like, did it say hi? Yes or no. Did it greet the user? Yes or no. Did it send my contact information? Yes or no.

Because then in the end you have a list of 100 of these things the agent should do. And if you make a change to your agent, you run it and then you'll see the result and you can kind of get the score. So it completed 100 of 100 of these eval criteria correctly.

And this is how you can get an optimization cycle later where every time you change something on your agent harness you're going to be actually able to score it and see was this a positive impact or not and what comes afterwards now because you have the eval system you have you can go through this optimization

Monitor Production Performance

loop now i deploy it you go live with your users then what you have to do is you have to monitor but now you're already prepared because you were smart you built your evaluation system already and yeah so you can just put this in your observability platform continuous monitoring you put your scores there and you can directly see live what your

users do so they start talking to to your bot or whatever they upload invoices it does whatever you want to do and it actually then tells you scores because you built a good eval system and because you're also smart you set up the tracing system you can also see what all your users did so now you have the

foundation of what I'm going to tell you next because this is why yes so we kind

Diagnosing Agent Failures at Scale

of talked about this already so now you have all of these traces coming in from your users so this was the next bottleneck we we were basically facing so we had like ten thousands of traces coming in every day so we all know if

If you use Cloud Code or ChatGPT, you can put some kind of context window. You can put like one million tokens into one context window. Now, what we were getting every day was kind of 400 million tokens. And now we needed a system to actually diagnose this. So how do you diagnose 400 million token of responses from an agent? And yeah, so you have all of these million traces.

So what we had to do, we had to build kind of a system that actually breaks out and goes in the individual traces and looks for certain root cause patterns and you then look what root cause pattern you find in what traces and like this you can build clusters and then once you build the clusters from this

you can derive then the fixes because you don't want to go once your life looking at individual traces you want to find bigger systematic patterns in your agent and this is where yeah you kind of have so then have a system that that diagnoses your agent at scale when you have a lot of traces.

Closing the Loop with Agentic Optimization

Okay, now bear with me. Yes, so we're smart. We built the scores, we built the diagnosis. So what do we do now?

We just basically ask your agent to look at the scores, to use your diagnostic system, and then basically max out on the scores. So what you built is kind of a reinforcement loop.

And you can have a coding agent just look at these two outputs and basically put it in a loop and say like hey look at these two things iterate on your agent harness change the tool change the prompt change everything and run your tests again and again and again until all your stores are green and once they're all green you basically have a new version of the agent so everything i just explained you what you did manually for a long long time once you set it up correctly with observability evals and a good diagnosis system you can close the loop and an agent actually builds your agent

so you can then plug a system inside and this is kind of what we did at our company so we had to build agents for three years and we basically just thought okay we build a tool for developers that puts all of this process into an agentic mode.

The Technical Foundation: Reinforcement Loops

I'm going to show you one important thing which is now for the very deep technical people just one minute.

What I just basically described from a scientific standpoint is that you have a task specific agent which is here in the center and it has some environment we call it environment whatever it's it's basically what it's supposed to do and then you have a feedback agent which looks at the results it basically looks at the scores and the diagnosis and then it feeds it back and updates the harness and the code and this is kind of this reinforcement loop that you're talking about and now

From Harness Optimization to Fine-Tuning

Now, when you have this, you already built, basically, the foundation for actually fine -tuning a model. And fine -tuning a model, in the end, is then having example test cases, a data set of examples, having scorers, and then you basically have a fine -tuning RL environment, and you can plug it in. And now the important part comes.

Specialized Intelligence Beyond Base Models

Now, if we look at very complicated tasks, like now everybody reads, okay, Opus 5 .5 is so smart, GPT -6 Astra is smart, they have 96 % on this benchmark and 99 % on this, and that's not benchmarks you should look at. Because if a model is so good at something, it basically already learned how to solve this task. Look at the benchmarks when the model fails, because there is room for improvement.

So, there's one example in this paper, which is called Self -Improving AI with Harness and Fine -Tuning, which is basically the process I just explained to you, and the baseline, the model is really bad.

So, apparently, it seems like Chinese law is very complicated, or at least there's not enough Chinese law in the Western world's closed open AI training data, but it really has like a bad score so it can't solve these tasks in the benchmark if you now

put a harness around it you can already see a harness loop around an LLM already gives you some improvement of a few percent but if you now create this harness optimization loop around it basically this feedback loop I just described you what every engineer do would do manually to build a good agent

then you can see this significant shift and this is what I wanted to show you how important it is that you build a good agent and how much power you have by optimizing your harness.

So you're actually not really dependent on the base model. You can use cheaper models and open source models and still build amazingly great specialized intelligence or agents.

And then this is kind of the visionary output.

Fine-Tuning as the Next Step

Once you did a good job and you really started with good foundation, the step to actually fine -tuning is just one fine -tuning service away from you and there's a lot of provider who do fine -tuning as a service for you.

Demo: An Agent Development Toolkit

We still have time for a demo?

Okay so everybody knows how Cloud Code looks like.

We built our own version of Cloud Code because we build agents and we love building agents and everything I just showed you we built basically a special command into this tool.

Commands for Building, Diagnosing, and Evaluating Agents

So if you have an agent or you want to build an agent open this in your repository and you have a lot of special commands and which you can then start so for example you will have a build command which will start building an agent so we'll start kind of building

you then you have like diagnose command which was basically what I showed you before it helps you basically sift through all of your traces it does this kind of process out for you we make basically all of this knowledge in here

then we have this evaluate command which helps you write really good eval scores scores, which is the other part of this loop. For example, you can discover evals from your production tracers. You can discover data sets from your production tracers. So the tool helps you to max out your coverage of scorers.

Automating Harness Optimization

And then at the end, there is the command which is called optimize. It's just basically a loop around this. So yeah, it's the scientific version of reinforcement learning for harness optimization. Fine tuning is all baked into a tool.

it's free to use just install it try it out and yeah with that questions right