Keeping AI Agents on Track with Better Planning

Why Long-Lived AI Projects Need Structure

Hello everybody, so my name is Alistair. I have been wrestling with agents pretty much since ChatGPT came out.

I was running a mental health startup. I knew I needed AI and ChatGPT came out exactly at the point I needed it, so we've been doing that for the last four or five years, whatever it is.

My background is tech but I also retrained as a psychotherapist and I'm very interested in how people interact with AI so the pitch today is very much for non -technical users

if you're doing loads of AI already this is going to be fairly straightforward but if you're thinking of using AI for your projects I'm going to talk about some of the complexities and how

to wrestle it into shape and there's a reason for that and that is because in reality if you've got a project most plans change so you start with a project you think you've written it down you know where you're going. And then you give it to a customer and they don't like it. Maybe you're working with clients and their requirements change. Maybe someone leaves your team. For all sorts of reasons, plans do not survive contact with the enemy, which is of course reality.

So you have to deal with that as a person, but so do your agents. And the problem is that we all lose track. Now you have memory, which is great, which means that you have a chance of remembering what happened if you spend enough time looking at it. Your agents will struggle a a bit with that even the memory that they have isn't that great. And if you're using harnesses that do things like sub agent orchestration, those sub agents aren't going to have the memory. And if you're using like cheap agents, they're not necessarily going to have all the sort of Claude's built in memory.

So the question is, how do you keep everyone on track, especially over a long lived project, where there's lots of changes? Because when those sub agents, the cheap agents, the ones that you're using to save yourself all the tokens come in, they need really rich and detailed context they need to understand what has happened how you've got there it's not just good enough to give them a website that is what it is today they need to understand how it got there and all the principles behind that the problem with that though is that if you do it in the naive way you're going to give an agent the job of looking at everything every time you spin up a new session and that's going to consume your tokens really fast so how can we solve all of this in one go or not one go but with some principles that keep your agentic planning world sane over time so we're

Build a Planning System Your Agents Can Use

going to start with one very simple principle which is your planning has to be real it has to be tangible you have to write it down it's not good enough if it's in your head now you can do really sophisticated stuff but at the very

simplest level put a planning folder in your project you've got your references your original proposals whatever started your project stick it in a folder inside your project you've got a new folder for your outputs if you're a developer that might be your your code repo for example if it's marketing campaigns this might be assets all that sort of stuff just create a planning folder and let your

agents manage it this is really crucial this is not for you to manage this is for your agents to manage and you want to give it a little bit of structure start with something really high level you want to have your project overview it's going to need that that's your grounding and then break down your

process milestones if you're doing technical stuff you might have architectural principles and then you've got tasks do not make this a nested folder structure that doesn't really work with agents just loads of files let the agent manage this you're not responsible for managing that you want to talk about all the stuff on the right with your agents what actually turns up in

Start Every Plan with Why, How, and What

those files though i'm going to give you two core insights that i think are probably the most valuable although there's many more the first one is the one that i've already started banging on about simon sinek's golden circle is my reference here why how what you always want to start every plan document with why why does this exist that's your agent's first question why am i here the

second one is what are the constraints what are you working towards what's the high level stuff don't worry about details this is the the how the general principles and then you get into the what and you want to be doing this across your project plans right so this is the why these two are how these are what and within each plan so this principle works really well if you

just give it to your agent and say everywhere that you do any planning use golden circle by Simon Sinek make it ask you the questions also stuff it's just going to really sanitize your plans and make your agents really ground

themselves very well the other bit though agents when they summarize stuff they will use huge amounts of jargon they will cross cross reference files files and you will read their outputs and wonder what the hell it's talking about so always ask it to give you a human summary anytime you do this and I like to say tell me like I'm 12 with ADHD

you don't have to have ADHD and you don't need to be 12 but on a day where you're super busy and you're in a rush that's really helpful you just want the high level summary and know what's in that you don't need all the details let the agent manage all the details you just need to know that this plan all right yeah that makes sense there's loads more principles these are

the two main ones that I think are crucial and important to note whatever your existing planning methodology keep it this is an addition this is not a replacement so that was the why and how

Managing Context, Change, and Project History

so let's do a bit of what now I want to visualize how a plan can actually change over time we start here with a hierarchy of original planning documents that we created and your agent for task one you're going to set off and obviously it needs to know about these two it needs to know the the process it's under, and the project overview, at least a summary of.

It does not need to know all the details of task two, three, and milestone two. It just needs to be where they exist.

If you don't have a structured plan and you just ask a competent agent like Fable to work out what's going on, it's going to read the whole world and consume loads of tokens. You don't need that. You just need it to work up the tree from where it is. That's where your well -structured plans help.

Over time, your plans change. Stuff happens. You do some work or you make a change. these are literally representing changes to the document so time is that way if you're a developer checkpoint one might be a git commit and if you're non technical that might be an end of day summary over time though that's going to

extend get really long this diagram would get very complicated you have loads of tasks multiple milestones how do you know what's done live tell your agent to track it it's that simple you just want to make sure that your agent is constantly updating all of your plan files at every level with the stuff that got done

and what changed this is your provenance this is your history this tracks everything the agent needs to know that or at least the bit that's relevant to it and that's actually pretty much it

Demo: Improving a Walking Holiday Website

so let's do a demo um i've invented this um fake website um it's women aged 30 to 45 who want to go on a walking holiday and a bit worried about keeping up because of fitness. That was one of the 20 ideas that the AI came up with.

That's what it looks like at the moment, but it actually looked a bit crap previously because the value proposition wasn't that good. I'll show you how we changed that.

This is literally a visualization of the planning docs. This is real. This is what is in the planning docs. I just asked the agent to visualize it in in a certain way I could make it bigger for you so that timeline we start with the plan

that is the fundamental that's all I said help people get fit who we're working towards I don't know yet we haven't worked out the ideal customer profile but this is your way this is your grounding for an agent the actual agent details long complicated all the details I don't want to read that I don't care right don't worry about it this is the bit that I care about so I've told my my agent golden circle summarize for me thank you very much I know what the plan is by looking at

this and then by looking at this I go cool milestone one clear welcoming website what's the first thing that we do we build the first page then we made a change to the plan we changed the audience and then this visual identity I said the first page is really boring it needs to hold attention let's make a change that's captured there and I'm looking back at the plan I can work out everything that happened on this website just by looking at this and then we go on I'm

going to show you what this last one here is I've got a video I'd love to do it live but agents take about six seven eight minutes as you've seen to do stuff so it's not really great for a 50 -minute talk so let's do the video I'm going to

need to zoom in on this cool so um this is what it was originally so you can tell that there's no information about your customer profile here there's no wind there's no cell what is this website about you land on this you've got no idea I've shown you this so I'm going to skip that and I'm going to come

Using Planning Context to Make Targeted Updates

here and I'm going to zoom in now one of the lovely things about having a good planning system look how short that first prompt is familiarizes yourself with the demo particularly planning in the current website I don't need to say any more than that it's going to ground itself and orientate itself against everything that I've got in the context I've told it what I want to do next just so it points at the right things which is helpful and there you go it's take some time to think all that through and it's worked it out yet that's the first

bit so let me scroll down because I then come in and I say I want to make a change okay so I want to improve the hero can you read this I'll just read out the current hero doesn't straightforwardly convey the value proposition or speak directly to the ideal customer profile so it's failing my golden circle why check can you fix it i'm happy with the visual style and general layout concept it's the copy that's bugging me the most if the copy alone won't be enough to fix it suggest what other changes we need to make and then use full planning ceremony as per this demo website's project planning so i'm just telling it you know look after the plans um any questions if not proceed and it decides No, I'm fine. I'm just going to proceed and it's going to do a lot of thinking and this is going to take loads of Minutes, so I'm going to scroll through it until we get to About here ish.

Okay, so it's telling me it's done an update Any moment now and Right, so that was the old version and That's the new version so So no ideal customer profile, it's not talking to anyone, it's just a whole load of like weird text. And now it's four minutes, 30 to 45, worried about keeping up, build confidence. I didn't have to invent this, I just said I wanted the outcome.

But my plans have got all the information about what's going on, because I've had that long conversation with it previously about the project, it's in the project overview. I don't need to tell it every time I have a conversation, I just tell it the changes because I've built a good planning system with basic principles. tools. So that is pretty much the demo.

Key Takeaways and Resources

And as a summary, for what I've said, these are the core points, there's so much more.

Put your planning in your project. Make sure you're using a golden circle everywhere, because it just makes life a lot easier for your agents, and ask it to treat you like you're 12 with ADHD, because it's just literally better.

And then you get this visualization stuff, which is very cool.

Now, if you want, you can take a picture of this, I can make these bigger.

Getting Started with Prompts, Skills, and Tools

this is just about enough of the core concepts I'll do the first one so you can take a picture of that that's just enough of a core concept that you can give to your agent when you start a conversation or start a project that it'll spin up the project for you you will probably have to have more conversation with it to iterate that that's just just enough to get started

and then this is when you want to make an update it's just enough again you you probably want to ask your Claude or your ChatGPT to save it as a skill and update it.

Or I have built a significantly more sophisticated version. I have plugins that I've been maintaining for Claude and ChatGPT because I'm spending a lot of my time as a consultant, essentially handholding non -techies to get started with AI agents and kind of

building little systems and little skills that just help like the I'm 12 and have ADHD. It's just a little skill. I get people to install that makes their life easier. yeah.

So this one will get you to the tools. It's on the website. So it's very friendly non technical install instructions.

And then if you want to know a bit more about me, it's the one on the right. And that's my website.

And hopefully, that was interesting enough, but simple enough to be of some value to you.