Introduction

So the question is, what's your strategy to verify that the agent has done what you expected and hasn't broken anything in the process? I think that's related to your talk.

Great question, by the way. Yeah, very great question.

Verifying Agent Output: Tests, Metrics, and Validation

And what you have to do is you have to definitely have a good test coverage of your code. Since we are now past the time where you can realistically look at every line the agent has done, You need to have metrics that you can run, so you run tests, you run when it's something that is performance, where the performance is important, you make sure that response times don't get up, that CPU cycles or memory usage doesn't go up, so you can verify all this very deterministically.

And obviously you also have still at the end you don't look at the code but you do have people who test the software that is written or to test when the agent is working on software you check that. When it's more like data pipelines then obviously you have all those checks in place that make sure that the things still sound. So you have to put a lot of work into the validation layer and the verification layer.

How the Work Shifts Toward QA and Test Management

I think it's a really good point, it's shifting, the work is shifting, I had a webinar for a client and we talked about compressing, part of the work we are doing now is compressed, so it's not necessarily replaced, but it's shifting into other tasks which are there.

I remember test management was always a big topic, I remember the first large project I thought I didn't know how much testing actually you can do yeah and this becomes even more important so you really have to kind of shift the kind of the work package and tasks you are doing in such an in such an exercise yes one

Keeping Code Agent-Friendly Through Modularity and Standards

yeah I was just saying that I've got a fleet of about 30 different agents that built over the last year but one of the things I found for getting the code

really working well is keeping the module sizes and having standards for the module sizes so the target between 400 600 800 is about the maximum and once it gets over a 1200 or so you've got problems because the agents can no longer sort of understand the modules and they have to go on deep searches and spend a lot of tokens and they typically when they

fix something they break something as well in this bigger module so but that's just a standard you you can give your code agents to say these are our guidelines.

Everyone got that in the back as well? It's exactly right. Can you just repeat that also in your words? I think it's good.

Clean Code Practices That Help Both Humans and Agents

Yeah, the things you always had to do, like having proper coding standards, like don't let your files get too large, have one file for one thing,

have all that clean code stuff that we all learned at some point in our lives, the linting and the and the quality stuff this is still all very valid because it used to help humans to deal with the complexity and now it helps

agents to deal with the complexity and that is that is absolutely the way to do and and also one thing that was usually a fight against windmills and you could

Using Agents to Keep Documentation and Downstream Artifacts in Sync

never do it you would always have stale documentation because you would develop develop and at some point you would ship and it was already late and then the the docs were wrong and the comments were wrong and now you can have a dedicated agent

that just runs and checks that for every new commit does it change anything where the related documentation needs to be updated, does it need to change anything in maybe some related code files and so on, so all the downstream stuff that is not directly code but that is related to the code, maybe your website needs an update, all that stuff you can now handle

to dedicated agents, you have to think about what kind of tool chain you put in place that validates all the aspects of your work and that's much better than you could do it ever do before because in my experiences with humans it didn't work

very well I just look at myself and I think there was a lot of times where the code comments that were above my functions had nothing to do with the code that was below.

You're the only one. I'm sure I'm the only one. So I think it's a

Responsible AI: Explainability, AI-to-AI Oversight, and Governance

valid point and one of the challenges and for those of you who attended before I spoke about responsible AI and the need of explainability that is a huge topic and a huge challenge so basically because you can create so much both in

text and in code I mean the text people have the same problem more or less yeah I think the coders have a bigger problem but then you take of course AI to control AI you have to because you can't handle it it's impossible to handle so So the only way to do it is to use AI to control the AI.

The problem is, who controls the AI that controls the AI? Or, in other words, who controls the Robocop? So when you know the image of a Robocop. So it's a huge challenge around governance, AI governance.

It's not solved. There are ideas, of course, but it's not solved. So a couple of hands, I see. I start here.

When Models ‘Escape the Cage’: Real-World Incidents

It's kind of related to what you're talking about. But I mean, you can see the red scandals of anthropic model, I think, also were leaving its boundaries and going to a high -hanging face.

Recently, as more open AI had to pull back one of its models, Astra, was a similar situation. So now, I mean, we've seen that the technical capabilities of AI is becoming so important in that engineering process.

My question is, what does that look like in practice for people developing AI agents? How can we start putting in those security boundaries for agents beyond explainability?

Great question. So the question is around security, how to make it. We'll come to you in a second. Around security, given all what happened.

And David mentioned that. So the escape of the models out of the cage, basically. So you talked about security. What's your approach to that question?

Security Boundaries for Agents in Practice

Definitely. So first of all, you have to understand what the AI is actually doing. You have to understand what you are allowing the AI to do or not to do.

Once you already understand that, you're already miles ahead of most people because so many people just click and accept everything.

So understand what it is that the AI is allowed to do. That's your boundaries. And then you're already pretty good.

For the rest, encryption, if you're creating platforms or solutions that have data in it that you know that are sensitive just just do a good encryption on it and and care about it care about doing things good and right

and secure and you will find a way Marcus any builds on that because I think that's a super important topic it's great question what's one of the biggest questions right now in terms of model capability just yesterday Sam

Capability Growth and Why Top Models Aren’t Always Released

Altman tweeted that they will slow down the development of uh of training new models or they will down they will they will pace it now because they reached a point where where it's hard to keep up with capabilities that incident you mentioned with with hugging face that was actually open ai's model but then a few days after they disclosed that then and shrugged that actually hours hours broke out two three times and um so it's it's happened now in in in all the big labs and they do, well at least OpenAI seems to now take it seriously and

slow down and Anthropic have already publicly said they have MISIS -2, they have it internally but they're not planning on releasing it anytime soon. So we are now at a point where they actually do not release the models, the most capable models into the wild anymore.

So we're already there, so now you can go through. So there's kind of two questions here.

One here.

Building Software in an AI-Native World: Success and Differentiation

Mark recently said software in terms of software companies. So the first question is how do you be successful at delivering software without having to be a professional developer? So that's the first part.

Okay. The second part is does that mean that things like local models running that the success will come from, the differentiation will come from How do you train your own model and combine it to the team that differentiates you from general work? Great questions.

So, to repeat for the audience, it's about software development and everyone can now somehow develop software. I'll comment that in a moment as well. And so, the question is how do you then basically do that and differentiate yourself? Let's start with that one and then do the local model first. Who wants to start, Markus?

I can start. So first of all, every person has different things that they wanna do and there is not a one size fits all solution for everybody.

For some people who want to just develop a basic website, maybe they should look at anti -gravity cause it's super fast, super cheap, it gets your ideas done immediately and you stay in the flow.

If someone wants to do something that has a lot of like, like a custom online shop, I'm talking about backend now, or a smart contract in crypto, don't use anti -gravity, use something else. Basically, talk with your favorite AI chat, tell it exactly what are the things that you're looking to do, and ask, because it has all the answers.

I would, let me, because I'm working with corporates. Maybe repeat the question. It's not that simple. So it's kind of, nobody knows exactly.

Enterprise Reality: Integration and Complexity Over Code Generation

A, what is really important is to build enterprise -grade software is not just kind of going into CloudPort. That does not work. Enterprise solutions are highly complex.

So you have to integrate everything. You can't develop the solution. So the building of solution will not be a differentiator anymore. It can't be.

because the models are all fast and all more or less the same quality and they have to be because every developer has access to all the models so you can't differentiate which actually leads into your second question

I don't think a local model which you train makes a difference but I'll let Markus talk to that but from the competition yes there will be a huge part of the market will go away

we have the same thing with Canva and design everyone can design now on a on a perfect professional quality does that mean there's less design more designers less designers are actually more designers because you can design more so to your point there's much more development you're going to develop everything like the

management of this room yeah like an application which actually does something which is useful here in the room so I think this is kind of what we're going to change and see but the differentiator is It's no longer the software development itself.

Markus, any other builds on that? If you disagree, then feel free to disagree. No, no, I don't disagree.

What Gets Commoditized First: Small SaaS vs. Enterprise Systems

I think the large, like the sales forces and Microsoft offices, if the world will remain, I think smaller SaaS providers will probably at some point go out of the door, but also not immediately because of corporate inertia

and because for many people, it's still, okay, hey, hey, I pay a couple of hundred euros per month for Slack, but that's still better than having my engineers build one and then maintain it by ourselves.

It's probably okay, but at a certain layer, certain tools certainly go away. So, I mean, why would you still pay 20 euros for a fitness tracker app when you can just one -shot it and then it's just your own fitness tracker app? So this kind of app will certainly go away.

I think enterprise app will not so quickly go away. And I think it's still, to the other point, It makes a difference whether or not a software developer uses uses these models or people who just Know the business domain, but not not the software at the moment it matters I think it will stop to matter honestly and a few years So I have a few more years then I have to find something else to do

Yeah, the next question is a related question and let me let me continue because you're going because of time

Operating Inside AI-Resistant Organizations

So any recommendation for someone that works on a very traditional and ice AI scared corporation? Are they still existing? Yes, they do. Don't worry.

The enterprise is so cybersecurity free. How to keep up to avoid becoming even more behind?

I'd say find a new job. No. No, seriously. It's an interesting conversation.

I mean, we have all these experiences. When I train people, it's like, well, some start at zero. And some say, well, we can't use

this model. You can't use this model. You have to actually disallow chat GPT. That's

good luck with that it's not going to work yeah so um it really depends and you have to decide for yourself is that the right company yeah and how much of ai is relevant to your role and then you have to kind of come up with a potentially a new career plan if you can't change something from the inside so looking for a new job might not be kind of it's not just a joke

yeah but that's kind of career kind of career conversations later any any thoughts on that Yeah, I mean, there's an analogy that I want to say, I don't know how appropriate that is, but you can lead a horse to the water, but it has to drink itself.

Like if you work with people who don't get it and like want to stay stuck, then let them be stuck and move on. Don't try to force something that is not possible to do.

The worst a company can do today is be against AI. At least do something you know a little bit i think you have a story around that as well yeah yeah that

No-AI Policies, Local Models, and Procurement Constraints

is true so i one of my consulting gigs i work for a cyber security company in germany on and off and so they have this no ai policy so not only can we not use ai tools to work on their code base which i actually totally understand because it's their proprietary code they don't want to have it sent to Anthropic and all that.

So for that, they will wait until they have local models capable enough that they can use it.

But they also have these policies that simply know we have, when we apply for new jobs or when we go into Ausschreibung,

what's the word for that in English? Like if there's a request for proposal, very often there's a requirement that you have to guarantee

that you're not using AI in any part of the production not in the documentation not in the testing which is ridiculous of course but that's how it actually is so they're not making it up I've seen I've seen the the papers where they

explicitly asked for that so I think we are in the EU and I wouldn't put it past the EU to at some point say hey listen every company from a certain size has to have at least that amount of code written by humans I think that kind of stuff will come because the EU is just stupid enough to actually do it yes come

Just make sure the name is here. Come and see me. We can have a, because I do some career coaching as well. So it's kind of easy, yeah?

So there was a back, yeah?

A little bit loud of this. This might be a simple question.

Cost Control and Efficient Agent Workflows

There's so many stories that I've had, even on my team, where these agents, and they have running overnight, and they racked $30 ,000.

Well, for that one, you have a simple one. You just define a limit. I mean, I have everything I do, every API has a limit. and I know if I run into this limit

then it stops working but at least I'm not burning more than 150, 200, 300 whatever it is

so that's a simple one but I think your other question is a lot deeper so I think let's go there

yeah David

Model Selection, Session Hygiene, and Avoiding Context Degradation

can I start so the other thing is it's a skill issue because there are definitely ways

and I'm glad that you brought this question up it's going to bring a lot of value it's a skill issue

you can use AI in a way that is going to be way more efficient cost less

use the use the models that are appropriate for the task don't use the best models just to do something super small also do not ever work in the same session all the time the the context is going to be auto compacted so many times and with each time it's getting auto compacted you're not getting the same quality anymore so we're not even talking about how expensive it's going to be

because it's going to be way more expensive, it's also not going to be as good and have as much quality. So the way I do it, for example,

Checkpointing, Technical References, and Handoff Documents

I have my system. With each session, I say three things. First, create a checkpoint document where you, based on the task that you're trying to do,

you make checkpoints with each major thing so that when the usage limit is reached, you know exactly where to continue or when you're auto -compacting.

Second thing is create a technical reference document that is especially important if you're building big platforms so that whenever it does something new, it writes it into the document. And this is cool because this document goes over into other sessions as well, all of these three. So you can have multiple models working on it.

You can even have different providers working on it. It's just a file that continues to be actively, continuously worked on. and then the third thing is to make just in case to make a handoff document where it summarizes

the most important things and what is still needed based on the checkpoints so that you can just take that document put it into a new session even a different provider and have it start working some really good points any builds on there really good thank you yeah i would say the

Context Engineering and Maintaining Agent Memory

the highest leverage thing is to spend most of your time on context engineering.

Have your CloudMD or equivalent files, make sure that this is done in such a way that it's, they call it progressively disclosed. You only have a very short file in the main CloudMD

and then depending on what kind of work it's doing in those folders, there are files that basically will then be read just on demand.

So if it's currently, if your current session is not about reading your emails then you don't need to tell it all the things that it needs about your needs to access gmail and all that so make sure you spend a lot of time writing proper context

documents i also spend a lot of time looking at the cloud memory that it stores and i weed it out all the time because clouds puts really silly things in a in in the memory something that situationally something went wrong once and and then it wrote this memory and then this is read

every single time it starts up it reads in all the memory if you don't if you don't make sure that you read that out constantly then it's you just have a lot of context rod and stuff that then it talks about all these tools that are not even installed anymore just system all that gets loaded into the session before

you even before it even reads your first prompt so you have to make sure that you that you are a little well spend some time on making sure you send the right context to cloud and not more or whatever a very very practical one I

Self-Improvement Loops for Agents

I want to add one more, so if you build an agent, one thing you can do is have a self -improvement file. It basically checks that and looks at kind of what to clean up and things like that,

but then even further, you give an instruction to improve the self -improvement. Oh, that's so meta. That's meta.

That's not that meta, but it's meta level. So it's really kind of telling it to look at the improvement and assess how the improvement works and then improve the improvement.

yeah so you that you have to think that through okay yes sorry so just just to add one important thing also well just just just just so let me just bring this point to the end and then I let you talk so also very important second guess if you're using the right model because you maybe

you're just using a very expensive model and third point build up your confidence because I see a lot of people working in the same session all the time because they are afraid. If they go to a different session, that is not going to be as good. But I promise you, the moment it starts to auto -compact the context, it will be worse,

as if just using a new session with the system that I explained and then you better build up your confidence, learn about it.

Okay, looking at the time, I think we have one more question at the back here. No, here at the wall, and then maybe two. If they are short, then two.

Can you say it louder, please? I'm sorry, I have a very hard time understanding you.

with everything, that's super complex and then for you you built that social network

what tools did you use and did it work better like more than a year ago what could we use if you had to do it now

ok, just to summarize the questions around which models are you using because someone basically said I think three months ago the harness

the harness, the tools sorry you're just using Okay, then let's keep that short.

Tools and Model Stack: What People Use in the Real World

There's a whole host of these tools now popping up because it's a new problem to solve. My slides, we didn't get to them.

That talks about these tools. I can show you them later if you come to me after the session. So I ended up building my own,

but for most people, there's fantastic tools out there. Yeah, it's, yeah. So for me, I used almost all the tools.

A Practical Build Story: From Rapid Prototyping to Modular Rewrites

So on Social Air, for example, I started building that in March. I started off with anti -gravity. Why?

Because it's so cheap, it's so fast, and it just brought my idea to life. I didn't use good code. It was just one long file, so it's not good.

Then I started to bring in Cloud, CodeWork, no, Cloud Code, with Fable 5 to redo the code in a modular way.

And then when 5 .6 Sol came out from ChatGPT, I used that to, because I also participated in their hackathon, I used that to do all of the agent capabilities, create the Klee, create the MCP tools. So I use all the tools basically for different purposes.

And I really want to also say anti -gravity from Google is really underrated. The whole Google ecosystem is just one subscription and you can go even to labs .google .com and there you can create super fast images, videos and for most people the anti -gravity tool is enough. It's already more than 80 % as good as the other tools and it's fast and cheap.

Choosing Models for Speed vs. Deep Reasoning

I think that's enough kind of for the pizza to further discuss. I make one last comment because models, excuse me. Models do matter.

So for example, I use for the Hermes agent, I'm using now DeepSeq4Flash. Incredible. The quality is just kind of incredible.

It's basically for free. It's nearly for free. And then for deep thinking, so analysis, so my part is more conceptual, it's not coding.

Kimi3 is one of the best, also in, so it just switches kind of the model. So, and it will evolve. All the models and the tools are evolving.

Conclusion

ring, and I think that's one of the tasks we all have, to stay up to date.

Thank you very much. Again, to all speakers.