Hi and welcome. This talk is called Thinking Agents. That's what I put on the website. And I think it's complete bullshit, the title.

I just had to make up anything. In hindsight, I should have called it everything I want to tell you about AI stuff. Because it's a very broad topic.

But it's a speed run. Because we've got 50 minutes and 81 slides ahead of us. Good luck.

A Fast Tour of AI Agents

Hi, my name is Thomas Siebering. My LinkedIn description is father of one daughter and 27 agents.

This is my daughter. This was taken in Italy last week. We're on holidays. My daughter's on the left, not on the right.

While I was on the beach, I was working. AIs were working. I was using my phone to control my weird agents in the cloud, and they did develop code, test the code, deploy the code to production of a client. It worked. and it yeah kind of what I want to show you a little bit about but first we're

gonna establish some Asian vocabulary because there's some words that people like to use but don't fully understand and I know there's like a mixed experience in this room so just so we got the same vocab to to talk about then we talk about what you can control with your aids and what you should where you can give control away where you should keep the control and how I put my agents to work as a small demo at the end my job title currently is agentic engineer at hr campus which is in dubendorf nearby zurich whatever agentic engineer means we're still figuring that out

because i started there two months ago i'm also working as lead developer at edin which is a startup in new york um founded by anna she was on the first presentation one of the women's in the a car. She works for Kickstart. I was at this event once, and I met Anna, and now I work at her New York startup. Networking works, I guess. Talk to people, guys.

I'm also doing AI workshops and consulting freelancer when people want to book me. I also DJ sometimes on the side, and I also work as a mind reader and magician on stage for company parties, reading minds. Did that at the hackathon last week as well. But yeah, similar to the first speaker, why can I do so much stuff? Because there's AI involved everywhere.

I did a master's degree in AI. It was the first time this was offered six years ago in Karlsruhe in Germany. Then I worked for a few years SAP development and tech lead and now a genetic engineer, whatever that is and yeah I automated my old job to a degree where I didn't want to work there anymore because it was chill and relaxed but I wanted to have a job again so that's why I moved companies this was me beginning of the

year on the left not on the right this question came last time I shot the slide yeah I didn't really sleep much because I thought people have to wake up guys guys, there's so much happening, so much AI. I was spending 600 euros in tokens every month. I wanted to test the news cloud model, then the new GPT one, and then the GROK came around, then the open source one. So I wanted to just stay ahead, but like every day some new paper got published and I wanted to kind of follow that along and it was hard.

From AI Anxiety to a New Toolset

Small disclaimer, I have no idea where this whole AI thing is going. And I would correct the sentence, I would say, nobody knows where this is going.

So this is just my perspective from a computer science engineer whatever what I think I do and whatever you do is probably right as well I read this end and I liked it some run one way others run the other way in the end we all meet somewhere again so whatever you do it's nice because no one knows the fuck they're doing and a few years we probably converge again somewhere but for now run anywhere but move this is a nice slide it's kind of

shows the development every engineer scientist computer working person goes through from the left to the right. I'm gonna read it out because the text is a little small, I guess.

It starts on the left with, hey, it's not enough. Prove me that this isn't hype. So this, the AI skeptic.

Then they start experimenting with LLMs a bit. They're playing around with it. Then they are in the, oh fuck, what's going on moment. It's called D and the headlights. Will I have a job tomorrow in the future?

This is, I was there in the beginning of the year because I thought, fuck, why did I study computer science? It's useless now.

Some people then go back to the the left again and post on LinkedIn how bad AI is. Others make the jump, land on the right. They start learning more about it, getting deeper, and then realizing at some point that it's just another tool in your toolbox, that AI can be programmed, that it's one other tool.

Steve Jaeger, a computer scientist very far in the future. If you want to know what happens in one or two years, look him up. Interesting interesting guy. Everybody says he talks shit and then a year later it happens, whatever he says.

How Developers Work with Agents

He just published eight stages and I added a ninth one that most developers go through first. They just write code by hand, the old school way, two years ago.

Then ChatGPT came along and we asked ChatGPT to write this function and to do this, and then we copy paste it, copy paste it, so we worked with ChatGPT.

Then the tools integrated the chat chat windows into the IDEs, so we write code, we could ask questions right in the code editor.

Then the YOLO mode came around, which was stop asking me if you're allowed to do this. Just do it. I don't care. Do it. Do it. I trust you. Do it.

Then people started to expand the chat window and didn't really look at the code that much anymore.

At some point, they were just ditching the whole code editor at all and moved to a terminal to just talking to an agent, don't show me the boring code.

Then I realized it's quite boring because it takes quite some time when the agent's working and he writes codes and blah, blah, blah.

So I opened the second and third window and then 16 windows at once. I have like an ultra -wide screen at my desk and there was just like 16 different terminals all working parallel and it, yeah, it gets stressful.

So then people invented orchestrations, which means you just talk to one agent who then distributes the work to sub -agents, to other agents in the cloud, wherever. You just talk to one manager.

And then Claude started branching out to be in the Slack channel. We can now take the Slack channel, which leads to these funny images. It's Claude from the finance team. I'm not an AI, so it's an unfortunate name to have nowadays.

AI Basics: Models, Tokens, and Harnesses

And now we come to into some boring AI basics. Yay.

I know what you're thinking. Fuck yeah. Basics.

We start

Tokens and Large Language Models

with tokens, the smallest unit that you should know what it is. Everyone hears that. You can look further into it, but basically it's somewhere between a letter and a word. It's not a full word. It's not just a single letter. It's part of words.

That's the unit we use to measure the speed of a model, the cost of a model, the context of it, so how much can we stuff into the context of a model? So just knowing what a token is can help.

How exactly it works, we don't know, because every company has their own algorithm to cut text into tokens. So I can't tell you how JTPT does it, because they don't publish that. But basically, they use small groups of letters, parts of a word. So Thomas could become Thomas. mass maybe we don't know then llm large language model like the math stuff i used to study

it's basically a magic machine where you put many tokens into the top and then the next token comes out just one so just one part of a word comes out then all the scrap gets put in again and the next token comes out very iterative takes quite a while and kind of like when you type in whatsapp and it proposes you the next word that you're most likely to type but just not a word basis and very fast and then it generates thousands of tokens which become source code text pdf whatever

tokens cost that's what you pay or you get a subscription but input tokens are way cheaper than output tokens input tokens is what you write what you send to the llm and output tokens is what's generated that's more expensive more expensive because why because you can't do that in parallel but the input can be written in parallel so the generating part is the element that answers the model that answers that's the part that's more expensive for example for opus five an output one million output tokens cost 25 just be aware that the input is cheaper than the

Models as Horses

output horses this is the metaphor i came up with explaining what a model is for me a model is a horse because a horse doesn't have like a real direction a real job but it's some power horse horse. It can do stuff.

They're different kind of horses. Some horses are fast. Some can jump high.

Some look nice. Some are weird and called grok. But they have different strengths, just like models.

So it doesn't make sense to always pick the most newest Fable, GPT -6, whatever model. Sometimes the smaller model is faster and better for the task.

So examples, 5 .6 Luna would be a model. Opus, Gemini 6 Flash, those are models. Models are are an LLM, which is the technology behind, a shipped LLM becomes a model.

So it has a cut -off date, it's done training, and now it's accessible for the public. This model is done. It doesn't learn after that. A model is done and won't learn.

If you ask any of these models who won the World Cup, they wouldn't know, because the cut -off date when they stopped learning was before the World Cup. Yes, they could Google, but it's not in their training data set.

Harnesses and Tools

Now the harness, the harness is the thing you put around the horse to control the horse. And a harness supplies all the tools that the model can act, can do something in our world.

A model can't edit a file. It's impossible. A model can just output tokens. That's all a model does.

There's no magic secret behind it. It's just outputting tokens. That's the whole thinking process.

It cannot create a file. It cannot edit a file. It cannot read a file. But the harness can.

The model can can tell the harness, please create this file, then the harness is the action, it does the action.

There are different kind of harnesses. Some have more tools, some are more minimal. For example, I use Pi, the Pi harness.

Microsoft Copilot is a harness, which sucks, but it's famous because of Microsoft. Cloud Code, as you know, ChatGPT iOS would be a harness. Joule from SAP would be a harness. Just examples of harnesses.

these harnesses have tools like this thing for example the hammer a tool is a thing that the model can then ask the harness to do and the harness does it like reading a file searching the web like who won the World Cup because I don't know please google it for me and then the harness does the web search and reports to the model the model chooses the tool the harness

executes it and the more tools you give to the harness the more than one can do but if you give give too many tools then it can be I don't know which tool to use there's so many tools um I don't know what to do so too many tools can be harmful as well

that's why I don't like Lord code anymore because they 58 tools nowadays and I just need four of them so but but we can't remove them so yeah choose your harness wisely and then there's a user which is you having the harness in your hands controlling the model so use the user uses the harness to then steer and control the model below.

Where Control Lives

Where do we have influences? What can we choose? What can we do?

The model, we can choose a model, but we can't change the model. Yes, open source. Shut up. We probably just use GPT, Cloud, whatever. We choose a big model and choose which one, but we can't train it. It doesn't learn anymore.

The harness, we can adjust the harness. It's called harness engineering. Your Cloud code settings, your chat GPT, user user profile, your memory files, your agents, MD, your skills, your tools, that's harness engineering.

You could do some stuff with the harness, but not too much because like Amphobic doesn't allow us to customize it too much.

And then the user, what you input, what you ask them to do, that's where you have the most control, what you learn, what you do.

We all use the same models, but we have different results. And that's the definition

What Makes an Agent

of an agent. We've got a model and a harness, and that's the magic word agent.

Inside the the harness. We've got tools, skills, instructions, which all are part of the harness.

The model is just this magic machine spitting out tokens.

Then we've got skills, which is basically a prompt template.

A skill is cool because you can write a lot of text in the book, but the only thing that the model needs to read is the back of the book. It knows, okay, I got these five books. I don't have to read them all because I don't need them right now, but if I need them I know I got a skill for commit and push or for write an email so it yeah it can load more of the book when it needs to and this is what I

built into the slide when I get two more minutes to present which we're probably not going to make it but we'll see maybe you want to skip the super crazy theory

Context Windows and Their Limits

at the end a context window all the stuff we put into the context into the the model, so all the model currently knows. Most modern models have around 200 ,000 tokens of context.

Yes, there are bigger ones, there are smaller ones, but I think most of them are 200, 270, which is roughly one longer Harry Potter book or the first three Harry Potter books.

The context window is all that we carry around behind us, all the context that we give the model and the harness. We got our instructions there, our PDFs, our dev guidelines, all the files that a model can read or reads and has it in its working memory kind of in the context.

Then we put in more and more, and at some point it's too full and the model gets more and more stupid. Then we add another PDF and then it's done.

Most harnesses do then compaction, so they compact all the stuff you wrote, but you don't have any control about how it compacts, so it loses information. You don't know what what happens there, so better just never run into a compaction.

On the left, I got a picture of this context window, what information's in there, what strings, what text. There's a system prompt, so the basic rules for the model, please do not do this, and Grok is very bad at that, because it's still the weird shit.

Then there's the harness prompt, the instruction from Cloud Code, from Cloud CodeWork, from GPT Desktop, and what it can do, list list of tools that it can call. And this takes up space. This takes up tokens.

So if you write hello to Cloud Code, you already consume quite a lot of tokens, because you don't see that the system prompt and the harness prompt is injected before your single word hello.

That's why I use PyNow, because in Cloud Code, you've got around 10 ,000 tokens in this system and harness prompt. So if you write hello to Cloud, you basically pay for 10 ,000 tokens, because it reads all of that, and then your hello.

And then open got a little bit less and code is a little less and I like pi because it's just 1k and you can Read those 1k tokens and edit them if you want So it's way more minimal you get more space to work in that big context window because it's not that much It's consumed by the basic system prompt And I also like to use pi because the harness is open source and I can modify All of it if I want to so the harness raises to 100 % of customizability

then all these skills gets injected the MCPs you got then the agents MD which is like the basic construction also in the context window and only then your user message gets appended to this long wall of text so quite a lot of text before you even started to work now for the deep ship theory

part which we probably gonna skip but I even added this for people who didn't want to follow along long the details they can watch the second screen top right basically it's how the harness and the model exchanges messages and it rises exponentially the more text it produces it feels like the chat wasn't that long but the model has

a lot of work and at some point you reach the so -called dumb zone where the model gets dumber and dumber because the carriage gets bigger and bigger so the quality of the model decreases over time the more stuff you put in and it's around 100 to 120k depending which papers you read where tokens in the context window where the model just gets stupid so start over if you know how many tokens you used already in that

Orchestration, Customization, and Cloud Agents

context where you'll start over at 100 to 120k i'm gonna skip that this is orchestration you get multiple models with the shared context or different contexts this is the orchestration just one guy sitting here and orchestrating the different models and another reason i hate cloud

code now because they change stuff every day they change the system from they add tools they remove tools they change tools so you build your cool nice in the workflow and tomorrow doesn't work anymore because anthropic changed something that's why you went to pi you can also keep the harness

the thing on the horse but change the horse that also works that's why it's in my opinion worth investing in the harness because then there's a new model you want to switch models you don't like anthropic enemy or whatever but the stuff you put into pi or your harness remains there which is nice to have. You can switch models, provider, whatever.

Nowadays the agent doesn't need your laptop anymore. It's on the cloud. That's the next step.

We're moving away from working agents on the PC but in the cloud.

I'm going to show you in a little bit.

Giving Agents More Capabilities

Yeah I gave my agent a tool. It's called email, agent mail it's called. So I can write the agent, hey whenever you're done please write me an email with a screenshot when you're done.

So I can I can leave my PC, go away, have lunch, and then I receive a mail from my agent. Look, I'm done. You can test it here, link. And I click the link, and I can test it.

Why? Because I gave the agent, the harness, another tool called email. So he could email me whenever he wants, which is nice. And it also can record videos.

So you can build a product, change a new feature, record a video so the agent clicks through the app, records a video of the new feature, sends me the video, asks me if it's fine, I approve it, then it gets published, deployed, whatever.

Some tools that I like give to agents, agent mail for writing mails, linear is just like Trello, Jira, ticketing system, Playwright to do automated testing, but we can talk at pizza for like the nerd stuff.

Tools I use, Pi, Herder, this is Herder to talk to multiple agents at once, looks nice. guys.

Ampcode, the German company to run cloud agents, so you can have them work while you sleep and you turn on the PC. Whisperflow to just talk to the PC.

Live Demo: Agents Handling Feature Requests

Now to the small demo part and then we're going to wrap it up. For the demo, I ask you to take out your phone please and go to this website, demo .nexus .ziva .ing. Ziva is my last name and it's a nice domain.

main thank you thank you thank you thank you dot ing go to that website and on that website there is one of the mails that my agents can read so you can click the copy button copy the email it's called sievering at agentmail .to and then write an email to that address in that email please write any short thing you want to see on that website so you're the client now you want to have a change request a feature request on that website you can write make the background pink make at a countdown to Christmas write a paragraph about pizza what just write anything write any feature request you want and then send that mail to sievering at agentmail .to subject doesn't matter and this one is linear

just a normal ticketing system Jira Trello whatever I use linear and on the very left you see the backlog there an email appeared from Tom hello Tom thank Thank you for your mail. So Tom asked, no message body, use Wingdings font. So he wants to use this font, and now more and more mails should arrive. Second mail, show running horses, nice idea.

So if I want to approve now Tom's feature request, I just move the ticket to to -do, and another, now more and more tickets are coming, and I don't do anything now, but now in Frankfurt on my server, an agent spawns. he reads the description of that ticket and in a few seconds he will move it, I won't do anything, he will move it to in progress, he will implement it he will comment on the ticket, he will attach a verify screenshot now it's in progress so the agent is working now and he comments on what he does in the ticket

I can read other tickets pizza sounds good, give me a dancing kitten okay feature request, add an embedded video I don't know if you can find the video because I didn't give him browser access. Margarita Colombo, order pizza.

And now it's working, and in a few minutes the website should update again and again and again. I won't do anything. Yes, I'm not in the loop here anymore because I just auto -deployed. For a real customer, I wouldn't do the auto -deployment process, but for the demo, we do it like this. Yeah, doesn't matter. We'll see what he does.

but yeah this is amp code basically having workers in the cloud here asking to create a video of the new feature that he did and you can use that on your phone as well and it uses my GPT subscription so I don't have to pay extra for the tokens they just use the subscription and that's how I worked

on the beach just in the program in the projects here it sets up kind of like a virtual sandbox box with all the tools that I allow him to use. So it's always the same setup.

But yeah,

Questions

that's the demo, and we're over time, so go to questions.

Yes. I have one, too. Yes. Thank you.