Introduction

Okay, so yeah, I'm Peter. I'm a product engineer at MindStone. You can tell because I have the lovely purple t -shirt on. That's how you can normally tell.

We are an AI transformation company, and we have a bunch of different products. But one of our core products is our learning platform. Our learning platform is for non -technical people. We teach them how to use AI in their jobs. startups. And this story is about how we rebuilt this platform from scratch with AI.

And if AI built it, how can we trust it? I just want to get a quick show of hands. Who's an engineer here or comes from, like, a tech background?

Okay. Quite 50 % of people. Okay. Put your hands up. Okay.

Put your hands up if you're not from that background but you've built something with AI. Okay, a smattering. Now, you have to be careful about those people as they're coming for your jobs.

Okay, so this is quite a technical topic, but I've really done my best to be inclusive for everyone. Half the room isn't technical. And what I'm talking about here applies for developers and for both knowledge work as well. So hopefully it's applicable for everyone.

Why the Learning Platform Needed a Rebuild

So our core product I was telling you about, our learning platform was our main revenue driver at MindStone. Over the past six years we've pivoted quite a few times and it's grown quite complex.

An Architecture Built for a Much Larger Team

It was actually engineered or architected by our old CTO who was an ex -Monzo engineer. The whole of the architecture was based around around Monzo's architecture.

Monzo's a bank with 400 engineers, and they have, it's sort of split up into sort of 86, well, multiple different services so that each team owns their own service. But we're just four engineers at MindStone.

So for the techies out there, it's 86 Go microservices in Kubernetes with Kafka and Redis, all talking to each other with RPC.

For anyone who's not technical, Imagine a huge machine with 86 moving parts, and they are all having to talk to one another to get one simple job done. It's kind of complicated, nobody wants to touch it, and we can't really break it because it was our main revenue source.

The Speed Bar: What AI Made Possible Elsewhere

Now, we have, just for context, we have a new tool called Rebel, which is a similar code base, 500 ,000 lines of code, and we developed it in three months with AI. AI, and we want to be able to move at the same pace with our main platform as we can with Rebel. That was one of the main goals for this.

The Migration Plan: Monolith, TypeScript, and Broader Contribution

What should we do? We want to simplify it. We want to turn the 86 different services into just one. In technical speak, we call this a monolith.

It was also in the Go programming language, and we want to go to TypeScript. All of our other applications are in TypeScript. It's more AI -friendly. friendly, we don't need ghost performance anymore, so it's better just to use TypeScript to get all the libraries and the AI understanding that it's trained on.

Also we want to get to a place where anyone in MindStone can contribute to our platform. Not just the engineers, but Niamh running the events, if there was an issue in the events platform, she can get on and contribute to it without an engineer having to jump on board. We want to make it simple enough for anyone to be able to contribute.

Setting an Aggressive Timeline to Force Better Systems

Now, when we talked to the AI about how long this would take, normally, to rewrite this whole platform, it kind of estimated it, and also us in the room as well. We sort of said six to seven months for humans to rewrite this. So, pre -AI, it would never really be a thing that would be touched.

Like, you know, all these banks running like these old legacy mainframes that no one wants to touch. It's because it's a hassle, and it's working, so why change it?

it, but with AI, we wanted to try and do it in one week, which is kind of a ridiculous timeline. But we set it because we wanted to drive creativity and innovation in the process.

It would be easy to go ahead and fire up core code and just start cracking on it, but realistically we need to build like a system and a factory in place to do this big, huge migration. So, that's kind of the drive for why we wanted to go for a small time frame like this.

The Trust Problem: How to Verify AI-Written Code

So, yeah, like I said, we could boot up our favourite coding agent and just say rewrite everything. But then how do we know it's right? How do we trust it?

Well, we have these things that in engineering terms called tests. So a test is like a contract. You can give it an input, and you get an output, and you can validate that output as you expect.

So what we can do is we could use our old system as our single source of truth. We could create tests against that, and then run those exact same tests against our new platform to see if they're the same, and if they are, then it's like for like.

We could also test all the different outputs that our old platform calls, so if it goes to send an email or has an interaction with the database, we can test that our new platform makes those exact same calls with these same verification checks.

And also a key part of it was going bug for bug. So there is issues in code, there's always issues in platforms. We could fix these as we migrated over, but then it's a bit of a mess because you've got one platform behaving differently to another.

The idea of creating it bug for bug made it way easier that we could use the tests as the single line of truth, and the test that we had written against the old platform.

Using Tests as the Contract

You might know this chap from The Simpsons, Ralph Wiggum. There's a term called Ralph Wiggum loop, where basically you give a coding agent a target.

So, in our case, we wanted to use test coverage. So, I know this is quite technical, but test coverage is very straightforward. All it means is are these lines of code tested? So if all of the lines of code were tested in your application, you'd get 100 % test coverage. And what you can do is you can put 100 % test coverage as your aim.

You can kick off the tests, get the AI to look what the coverage is like, check the the gaps, write more tests, and just keep going in a loop until you break out of it, until you've reached 100%, or until you find that there's no progression with the iteration.

The Ralph Wiggum Loop—and Why Coverage Can Mislead

So I kicked this off, I wrote loads of tests, and I was like, oh, my God, I've got so much test coverage, this is great, it's going to be so straightforward, and then I found the The AI was cheating me. There it is.

It's got its lovely green tick of truth that we know as developers, that everything's green and good, and then behind his back he's got all his wires crossed. So basically the coverage was great, but the quality of the tests were shocking.

The AI basically just, the sub -agents just tried to create the quickest way of creating the tests it needed to do. So, it was chasing coverage and not correctness.

It was sending in an input, getting the output, but then not checking the output. It was calling to send an email, but not checking that the parameters it was calling to send the email were correct.

Making the Agent Prove It: Unbluffable, Deterministic Checks

So I then was like, okay, well, we can't assume it's right. We need to make the agent prove it's right.

So I'm a week in and back in October I had a little climbing incident where I fell off a cliff and I get a call from the hospital saying that I've got a week until my surgery so the very funny one week deadline became two weeks but a very real deadline now as I needed to get this done and I didn't have anything to show for it because all my tests were rubbish.

So you might You might be thinking I'm sort of failing at this point, fair enough, but the job became building a system that checks the AI.

So the idea was that if we make a call out to an application or if we get an output back, making the tests fail or making something fail and providing a fast feedback loop to the AI so it understands that we haven't, like, tested that thing or there's an issue with the code, or it is a rubbish test. Providing that instant feedback loop was the key to this.

It is telling the agent straight away that it has done something wrong, and in this case, using deterministic checks to tell that agent that it has done something wrong. Through these deterministic checks, because deterministic is reliable, we could have a sense of it being reliable even though AI is not always reliable. That makes sense.

So, I found, I tried to, I was like, great, this is great in Crate tests now, but I tried to fix the original 7 ,500 tests that we created, and it was going around in circles, burning loads of tokens, and it was actually quicker just to throw out the old tests and write new ones.

And again, that's a mindset shift, I guess, for developers, because it's not saying we just throw away and build something new, because we're not used to fixing things, rather than and it is like throwing it away, so something to bear in mind.

Throwing Away Bad Tests and Rebuilding the Workflow

While the tests were running, I went ahead and used my engineering brain and was like, okay, these are the particular patterns in the code. How do I want to model these with best practices to develop the code?

I went ahead and built playbooks and a factory to build the code to migrate it across the new system. system, it would run the new tests, fix anything broken, review it, and again, go around in a loop to do that.

Playbooks, a Factory, and the Iteration Loop

Another kind of Ralph Wiggum situation. And the anti -climax of the talk is it worked.

Outcome: Shipping Before Surgery and a New Kind of Handover

So, two days before my surgery, we went into production, and the handover these days has has become interesting because we're no longer writing down and giving loads of meetings.

It's now make the agents .md and the claude .md really well -documented and signpost to all your documentation so that another developer comes along and there's a bug or they need to ship it to production, they can easily do that via the agent and they haven't got to worry how to do it.

Can We Trust What AI Builds?

Can we trust what AI builds? I would say yes, but not blindly. Interestingly, the trick here is to create an unbluffable check. This applies to code and knowledge work.

The Principle: Fast, Honest Checks

I told you I'd get back to non -technical stuff.

In non -technical examples, for example, if you create a report from an AI agent and you need to validate that report is correct, maybe you could add up the totals, check the formulas that it's used, and understand how the report's been created.

You may create a script to create that report so that you can test that script rather than just getting it to deterministically create this report each time, which may be different.

If you've got an answer from ChatGPT coming back, you might demand citations or the sources of the things it's coming back with, and then you can fact -check those things.

You could have another agent do it, or you could build something else. house. Deterministic or not, just a fast, honest check.

Get it to prove it's right, not just ask it's right. It's the punchline.

The Human Cost of Agent Swarms

Finally, I just wanted to talk about the human cost of agents. At MindStone, it's like a huge thing we're tackling with and the engineering team is tackling with in particular.

Running a swarm of agents is incredibly taxing, having Having so many windows open, trying to kick them all off in parallel is a huge cognitive load and a huge burnout risk.

It's probably a talk for another time, but I just wanted to touch on it because it matters to me quite a lot.

If you're struggling with this currently, and you're an engineer or a knowledge worker, I think you've mentioned it in your talk about having this cognitive overload, just know you're not alone with all of this.

Conclusion and Next Steps

So, we're currently hiring two product engineers based in London, and we also, as I said, offer AI training for non -technical people and for developers. So if you're interested in working for us or training, come and talk to me afterwards.

You can also contact me on LinkedIn, and there's also a really deep dive technical blog that It probably goes into way too much technical detail if you're fussed about any of that.

So yeah, thank you very much.