Introduction: Mindstone and the learning product

So I'm Peter, I'm a product engineer at Mindstone, and Mindstone, kind of introduced by Niamh a minute ago, so we're an AI transformation company, but what that means is we have a number of different things, and one of them is AI training, so we train people how to use AI. One of our products, one of our main products, is our learning product.

This talk is about how we rewrote that product in AI to optimise it, and I'll go into that more in a minute. it.

But the main the core, the core part of the talk is about how can we trust AI? How do we know that AI is producing the correct stuff when we prompt it?

Let's get a show of hands of who's actually like technical from like an engineering background in here. Okay, so about a third half.

Great. So

AI.

The legacy platform: powerful, but too complex to move fast

our, our main platform was built by X Monzo engineers. And Monzo is like a bank, and it's quite a big company, and it was built on Monzo's architecture.

This was over six years ago now, and the company's pivoted around so much, so we've adopted this quite complex architecture in our platform, and for the technical people out there, we've got 86 different microservices.

It's running on Kubernetes and Go, and they're all communicating with our PC. for the non -technical people imagine just like a big machine lots of individual components all trying to talk together with each other and it's just quite complex and the thing is is that it's

great for like a company like modern so we're a big company because you can get certain teams to work on the individual components but we're a company of just four engineers so it's just a lot for us to manage and plus we've got other products we're managing at the same time time.

Why rewrite now: AI speed, Rebel, and a one-week refactor goal

We created a new product called Rebel in November, and it was all AI -developed. The code base is around 500 ,000 lines of code, which is the same as this platform, but we've just developed in a few months rather than years.

We're able to go so much faster on Rebel. We're able to deliver all these features on Rebel, but we can't move as fast on this

because of how it's engineered so we like we wanted to go ahead and simplify it so rather than having like 86 different services we want to like reduce it into one simple service to make it like way easier to work with with ai and we're going to move the other thing is we're going to

move from a programming language called go to a programming language called typescript which is basically what all of our products currently use and it just works way better with ai and stands Stan's TypeScript, it's simpler, all that good stuff.

Now, it's a big project, and traditionally it would never be done, or it would be done very slowly, because I had conversations with AI and with my CTO, and a team of developers normally would probably take like six months to do this, pre -AI that is. Whereas now with AI, we wanted to achieve it in a week.

So we kind of set a ridiculous target of a week to refactor this whole massive code base that powers our whole learning platform, which is our core revenue stream of our business to make it faster going forward.

And the reason why we set a ridiculous target of a week was because we kind of wanted to drive innovation and creativity and work out how best to do with with AI agents and sort of build a system to do that rather than just sort of hand cranking it all.

So, yeah.

The core question: how to trust AI-generated rewrites

So we could just go ahead and go into Claude Code and just be like, rewrite this into TypeScript, flatten it down into one service, it's too complicated, off you go.

And then how do we know if it's working? Okay, we could do like a bunch of manual tests and, but like, this is a big complicated product. Like there's lots going on in it. it.

There's loads of business logic. How do we know that it's actually going to be correctly like translating that business logic over?

And the way we went about it was, and the

Turning the old system into a specification (bug-for-bug tests)

kind of key to it was replicating it bug for bug. So the idea was, is that we'd spin up loads of different tests that would kind of have an input test, that input into the first product, the Go product, the big one.

And we test all like the individual outputs and make sure that it became the specification for how it should work. If you call it with this, it should come out with this.

I just went ahead and I was like, oh, this is great. I'm going to create all these tests and now I can just get the AI agent to do it. I wrote loads of instructions to do that, which I'll get into in a second.

The idea was that once we had all these tests, we could then run the tests in the new version and check they were the same. If they were the same, that was great.

If there's bugs in the code, why would we want to create bug for bug? Well, it's because it's kind of a single source of truth.

If we start fixing bugs as we're trying to go from one system to the other, that'd be great, but then we wouldn't know if it was correct or not. It's easy just to fix the bugs at a later point.

Automating test creation with agents and coverage targets

The way I went about this naively was, oh my gosh, I could just create loads of tests. tests.

I created this test dashboard of all the different services and how many lines of code there were and the percentage of code coverage.

I came up with the LLM. I created instructions per service to go through.

We used a thing called a Ralph Wiggum, which is you create a loop. This is a guy for Simpsons.

You go through and you get different agents, sub -agents to go through and you kind of define a thing for them so I ran it off the code coverage so I'd be like make sure all this code is tested if not review it try and fix it carry on and just

go around in a loop until that code coverage was 100 % and I'd also like put into other sub -agents to review it and be like are we missing anything that we haven't tested and they'd be like no and

When agents “cheat”: tests that look right but prove nothing

I was like great amazing we're getting there then I found out all my agents were cheating and all of my tests were rubbish so this is the this is the cheating agent he's got all the wires

anyway so uh so i basically had all these like rubbish tests that actually weren't testing what i thought they were testing so when i said earlier i was going to test things calling other things um they didn't actually assert those things like they would call them but the tests wouldn't actually assert if that was right.

So it was kind of missing a bunch of stuff. And I was like, okay, well, I need a way for the AI to kind of know if it can, yeah, like a faster feedback and a feedback for the AI to know if it's cheating or not.

Because writing these tests is actually quite complicated because it's kind of got to marry up the code to the the tests and work out. Sometimes they are big functions and all this stuff.

It gets quite complicated.

Fast feedback loops: fail tests unless calls are actually asserted

The way I went about it is creating these nondeterministic checks to make sure that anything was being called. If anything was being called, then we are like, okay, make sure that thing has been asserted, otherwise just fail the test entirely.

entirely. Yeah. So if we didn't have those checks in place, it would mean it would just carry on like normal.

But with these checks in place, it was like it gave the AI a quick signal to be like, oh, God, I've done something wrong, and it would just tell it this is the thing to go and fix.

I get this is a bit complicated for non -technical people, maybe, but, like, it has a point to it, which I'll get on to.

Execution under pressure: rebuilding the test suite and shipping

So we set that original target of one week, week. After that one week, I had a bunch of rubbish tests. I was doing really well.

Also, back in October, I had this incident where I fell off a cliff. I got a phone call from the hospital saying, oh, we've got an appointment for surgery in a week and a half time.

I was like, okay, well, the deadline went from now to definitely a week's time because I had to go.

I still hadn't written any new code at this point, so you'll probably think I'm failing. But the important thing was the tests that worked.

And once we put these checks in place, we're able to kind of do that. Hang on a second. Yeah, so I spoke about this fast

feedback loop that let AI understand it. And we wanted to create like a system where the AI couldn't cheat. And that's the important bit here.

Building a system where the AI can't cheat and it's got these checks that it can go through so it knows it hasn't cheated. Once I'd put this system in place, it could then write these tests really well.

I tried to get it to fix the old tests and it couldn't do it. It literally just went around in circles and was a complete mess.

I just binned this 7 ,500 tests that I'd written because code is cheap now. Again, it's not something you do as a developer.

Be like, I've got all this stuff I've done over a week, I'll get rid of it." Now it was just really easy to create them.

Migration playbooks and treating engineering as a factory

While the tests were running in the background, I worked on different playbooks to migrate the different services.

We like to think of MindStone as a product team as creating a factory to do our work.

It was really important to come up with the patterns of the pre -existing technology and sort of plug that into the new system and yeah so I kind of went through the complicated stuff came out of these like agent instructions used the same

looping thing pointed it to this test environment that I created and the anti -climax is it worked and I managed to ship it like two days before I went on surgery which is kind of mad also

Handover: documenting the agent workflow for the next engineers

Also, thinking about handover, because I was the sole person working on this, the handover now has become making sure the agents .md and claude .md is well populated, so any developers coming onto it now can just be like, how do I do this? And it will be there for them.

So it's not like, oh, Peter, can you get him out of his surgery so I can ask him out to do something? It was just there and easy to work with going forward, which is the whole idea of this whole thing.

domain. Sorry.

Takeaways: trust, verification, and the human cost

To the point, can you trust code no human wrote? Yes, just never blindly trust it.

I just wanted to say this applies for knowledge work as well as developer work. If you are creating knowledge -based work at whatever domain it is in, if you can create these fast feedback cycles somehow of proving this stuff, whether it be non -deterministic deterministic, it's probably going to give you more validation of what you're producing is good, basically.

Cognitive overload with multiple agents

And finally, I just want to mention one thing, is that something we're battling at the moment is cognitive overload. We've got a bunch of different agents running all the time, and it's super hard.

You're trying to read through all this stuff that's generating and make decisions all the time and I'm just trying to deliver this in a week so it's just very burnout intensive and I think it's another talk for another time but it's just something I wanted

to point out because it's like something we like discussing a lot as an engineering team at the moment at MindStone.

Wrap-up: further reading, contact, and hiring

Okay that's it, so I've got a blog post here if anyone's interested with like loads of more technical detail I couldn't fit it in in 10 minutes and this is my LinkedIn if you want to connect we are hiring two more product engineers at mindstone but it is based in london

hybridly and as neve said we do run ai training in both for technical developers and non -technical people in any kind of domain so yeah if you want to hear about any of that come and speak to me afterwards.

Amazing Peter, thank you very much. Thank you.