Introduction

Thank you, okay, so before I get cracking quick sense of who's in the room Can you give your hand up if you're an engineer software developer?

Fair few of your hands up if you're a designer One hands up if you're a product manager one Two okay.

I was just about to say my collective noun for Product managers is backlog so small backlog there for you everyone else who's none of the others Three okay on most of you here.

Okay, great. You don't need any of the other people anymore

um joke it's a joke um so interesting okay so um uh drummond has done a nice little intro to me

Speaker background and what to expect

um thank you and for that so uh a little bit so yeah my background last 20 years working in mission -driven technology from just giving through to most recently chief product officer lightful and I was also been very interested in how AI has been has has been adopted in in the charity sector so in this book as well but so what I'm going to talk

A discovery-first AI product process

through is three things how to run this what I call discovery first AI AI discovery spin desirability first rather build a good enough prototype that gives you some insights and then kind of how to align your team and demonstrate your your impact.

So the story starts really at Lightful when generative AI became a thing. What we did is we brought together a cross -functional team of people to investigate this technology, starting from the perspective of what are the problems that our customers have and how can AI help solve those challenges?

So we were quite kind of early adopters of this this sort of gen AI API technology.

Start in the problem space

And what we worked on was building this AI product process where we'd start really, really importantly, first thing is problem space, like what's the problem we were trying to solve?

And we'd look at kind of other things, ways of solving that problems, other aspects of the other tools.

And then once we had an idea, we'd sort of do a lot of prompt testing, come up with then what we called at the time a rapid prototype, which would still take, I don't know,

a few days depending on the fidelity of that prototype. type. And we thought this was a much faster process than we'd ever used in ever before.

Rapid prototyping and “vibe coding” tools

And then kind of vibe coding came became a thing.

So hands up anyone here used any vibe coding tools. So some of you some of you know, okay, so I'm going to just demonstrate how they work while I talk.

So one of the big topics in AI is like, when is the first one person unicorn gonna be created?

And you know what, we're going to do that out tonight.

So imagine Airbnb, but for people who want to find great sandwiches and can't find them. So we're going to build an app called AirBLT.

So we're going to get this running in the meantime. I'm using Loverball. There's lots of different websites.

So we'll just let that run in the background while I carry on talking. Why did you pick Loverball? Why did I pick Loverball?

Oh, straight in with the question, just because it was the first one that came to mind. But as I explained, we actually tried quite a number of different tools. at Lightfall to see which one worked best for us.

Start at the end: define the story and success criteria

So my advice, kind of when you're looking

at this product process, really start at the end, actually, when you're thinking about creating a prototype and testing it with people to see if there's legs for your idea, start at the end.

So one of the things we do is start by creating a deck, creating like a report of what you'd want to see after your prototype.

This is a little bit inspired by Amazon, Amazon, who would often get their product managers to write a press release for their product before they'd ship it, because it forces you to think about, what are the benefits I'm doing?

What are the problems I'm solving? Who is this for? How will I demonstrate that to them?

So this is a good way to get your thinking clear, and this will help in getting you to create better prototypes as well.

So actually, one of the things I would suggest

you do as well is you can use tools like, this is an example of Gamma, which will create an entire deck for you from this prompt so you can scan the qr code and you can see what it actually generated from this prompt but you can see here what i'm asking encouraging you to do is think

about all of these different aspects of oh i shouldn't stand in front of that all of these different aspects of um the story you want to tell i really love storytelling i think it's really underrated uh skill in in product and technology so you're thinking about who's your target customer

what's the problem you're trying to solve like how are you going to solve it what and then project what would the prototype look like that we're building how would we have tested it how do you've demonstrated that it would it was

actually useful to people it's often you know now it's really really easy to build stuff but you know how do you measure whether people actually want that stuff you're building so when you're doing sort of user testing or if

User testing questions that reveal real value

you built a prototype and you're speaking to people the most important question you can ask is like you know would you actually care if you couldn't use this anymore? Because that suggests there's some real value in what you're building.

And these follow -up questions try and kind of tap at that as well. Like, when would you use it? Would you stop using it?

Would you use it tomorrow? Why and why not? And it starts to unpack in people, like, if they answer things like, oh, yeah, you know what?

When I'm doing this thing, this would be really helpful at this point. And it starts to get them to place it into some sort of workflow or some sort of context.

So you know that you're building, whatever it is you're building is scratching an itch that your users have.

So, you know, the two product managers here will understand their roadmap is always very busy. So how do you kind of bake time in for this sort of thing?

So really, you can start very quickly with just 90 minutes, you know, thinking about what's this problem I want to solve, do all the thinking about the problem.

And then the last little bit is about writing the prompt, using the tool to actually generate the prototype.

Making prototypes realistic: scope, context, and throwaway work

And so this is one of the things about prototyping that's very important to understand how it's very easy to build stuff.

Something that's good enough has got to be realistic enough that it's like triggering that real feedback. It can't just be lots of static stuff that doesn't really have context.

1Context is so important for when people are trying out new tools, new technologies, testing things.

You want to clearly define what is and isn't in scope. It doesn't need to do everything you need to kind of think of like what are those really important parts of the feature and do? They work really well doesn't matter if some of the other bits don't work very well

when we were doing this at life or and We you know I remember working on a prototype with one of our designers and they spent like two hours trying to get the logo put in The right place now that is quite a design free thing to do no offense designers But it made no difference to whether the prototype was good or not. There's just a waste of time

so thinking about like you know what what is it you're needing to test and again it needs to be throw away able that's not a word but I think you understand what it means like if you don't have put so much work into just this test that you feel really bad if it doesn't actually work because it might not work and so one of the other things that we found working with these these

Prompt craft: add context, iterate early, and try multiple tools

tools with it and it's really important to spend time on your first prompt and and make sure that you've actually added a lot of context about what you're trying to do, the people you're trying to reach, to get really good prompts.

So spend time on that.

A lot of these tools now have planning mode that will help you kind of answer some of those questions about what you're trying to do.

They're even good to take your prompt out of the tool and work on ChatGPT or your LLM of choice to help you refine your prompt before you start that process.

Because once you've got the prototype, it's really, really hard to then make massive changes.

that after we found kind of after a few goes you would just start to get a bit flaky and if you're working in product if you have things like personas or jobs to be done that's jtbd ways of saying you know these are the specific problems i'm trying to solve that again will help those

models create better applications for you again try loads of different tools they all have free versions it's great when you run out of credits try a different tool and see how that works as well.

So for us, we found that this vibe coding had a number of different advantages and challenges.

So really fast. And we built loads of prototypes.

A six-week sprint model for building and testing prototypes

And we did this in in six weeks, we carved out a kind of six weeks period of time in a roadmap where we did six one week sprints really trying, working across our team to build and you know, having product

having design and and tech all together to build these build these prototypes. So again, they were really good at getting

fast, but they're a bit unreliable, a bit unpredictable, and so you have this sort of lack of control. So let's see how our lovable prototype is going.

So this is what it has created. Find your perfect sandwich, featured sandwiches. Oh, look, what's this?

The Mindstone AI Meetup Club. A snap, only a thousand pounds. What's it got in it?

Mr. Armin, chat GP tomato. Oh, and who are the reviews from Elon M, Sam A, and Jan L, over -hyped. Okay, so interesting.

Got a sense of humor here. So you can see this is, you know, not a bad site. It's got a bit of, you know, some filters in there. And it's got kind of decent display.

It looks a little bit like Airbnb. And you can see the comparison here where I put in a very, very long and elaborate prompt talking about different home pages, different shop pages, different sample stuff, you see it's developed quite a lot of richness and depth here.

And another version I did earlier today where I literally asked this prompt like build a simple two -sided marketplace called AirBLT with, you know, and that was the extent of the prompt, it was a much sort of thinner product, it was just a single

page with a few photos, didn't have the reviews, didn't have the kind of nuance. wants. So the difference is the more effort you put into those prompts, the better results you'll get, kind of like any AI tool, really.

So that was our learning. And really, what was actually

What really changed: team culture and blurred roles

interesting, we tested V0, we tested Bolt, Figma Make, and all of these. And it wasn't the tooling that was the interesting thing of like, some tools are better at some things than others.

others, actually what was interesting was the culture change and the fact that we had designers and developers who had very distinct roles before were sort of clashing. And that was uncomfortable and that was hard.

And that's only starting, we were doing that work in September, October, and the tools now have progressed much further, and particularly in the developer world.

world so this is a sort of real source of tension of how do you work as a team when all of your roles are blurring and stepping over each other so for us what worked was having much more like pair workings you might have a designer and a developer working together on one prototype and and that was really good at having this sort of great knowledge transfer between the two teams the two different disciplines and that helped kind of support all of that a little bit bit of teamwork, but it was uncomfortable.

And I think you have to acknowledge that this is actually quite hard for people. It's a real cultural shift in how product teams work.

Wrapping up: synthesize learnings and decide next actions

And so then towards the end here, as I wrap up, one of these important things to do is if you're building these prototypes, again, it's not just about the build, it's about what you have learned from building it, testing it, and speaking to your customers.

So this

is the last slide from that Gamma Deck. So you always want to make sure that you've got this so what question afterwards and so for us working across those six weeks on these sprints

we learned you know doing this product design using five coding was really powerful but it was a bit opaque unreliable a bit unpredictable but getting those ideas in front of people was really helpful and actually the hardest part wasn't the tools it was team it's a process

and humans and how we work with these tools and again that's a really really big change a really really big challenge for all of us.

And so there's just one slide left. It's kind of all of that discovery process in a slide.

If you want to take a picture, remember all of that stuff. Because the really important thing is to just follow all of these steps.

And I feel there's one thing you take away from the talk, apart from investing in Air BLT, is really think through the problem you're trying to solve before you start building whatever prototype it is.

The more you can think about that, the more you can describe the users, the more you can describe the typical things they will do, the better your prototypes will be, and then the better your testing will be, and the richer your learnings and and your actions afterwards will be that much more powerful.

So I think I'm hopefully at time, but that is me.

And yes, if you do build any exciting prototypes after this, I'd love to see them and tear them down so I say how great they are. But yes, I'll pause there if you have any questions.

Q&A: execution details and team dynamics

Did you end up with a bunch of solutions looking for problems?

So where people didn't spend the time in the problem discovery the definition phase and just oh this is a cool idea i can build it

and then how did you handle having six prototypes in six weeks the showing that to customers and then getting under six weeks and i don't know how many you built but then going back to those customers and going yeah those things that we showed you we're not going to deliver but we'd love your time again next week to show you another idea how did that pan out for you

Preventing “solutions looking for problems”

yeah good question so um so interestingly enough we didn't start by building stuff we started by thinking through problems and so the first week was actually a lot of um having a big mirror board of like what are everyone's big ideas what are we you know what what are all of the personas we're trying to reach so we came up with a few couple of different personas um and then tried to get

everyone in the team to contribute ideas which was uncomfortable in a different way because not not everyone likes doing that. And so we had a bank of ideas.

And then together, we said, okay, what are the most impactful things we think we should do? And then what we had was every

Thursday, we do user testing. So we sort of put a line in the sand and say, every Thursday, this is going to be something we do. So we'd have to put that pressure on the team to deliver something.

And the ideas were a lot kind of aligned with this like future product vision of where we wanted to get to.

So at Lightform, we ran a learning program for charities, helping them with their digital skills. And we had kind of one of the elements of that was one -to -one coaching with humans.

We had an idea that we could do a version of that coaching with an AI -powered tool that would help kind of some of those circumstances where particularly it was like different languages. We worked with organizations in like 120 plus countries. So there's a lot of different languages as well.

So we were choosing things that we had a very good domain knowledge of. and then each week we would sort of do a first test and then that would we do a little bit of iteration of that but then actually start on the next one because it was um we were speaking to different people each week often so we were like again this is really i don't think it's on this

slide actually but one of the hardest things in this process is always finding like arranging user testing so part of that team was like a customer relationship person who whose job was to make sure there was people coming to that and actually we did a lot of internal testing as well

with people who were on different teams who weren't quite so involved in uh in the tech side so they were like quite good kind of temperature test of like does this even pass like interest or not and then should we bother doing anything and yeah we did kill a few um but uh but most of them were well received so that's a very long answer sorry did they ship unfortunately

we did not get the chance to ship any of them they were on the roadmap the main reason being that we actually ended up winding up the business a month ago so otherwise they would have and but and again one of the interesting and tech

things that we built along the site alongside all of this was our first like mcp server so that was our first effort doing that which was then going to be like the brains of all our ao products but sadly did not see the light of day how did you come up with the test

parameters the test parameters how do you decide what the test plan is the limits of our testing

How to define test parameters and a usability script

so um i guess working with the design team in particular was a as a case of um what are the kind of what's the script we're going to be talking through in terms of we built a product with a particular problem in mind or a feature with a problem in mind for a particular user

and so the test parameters are like okay a do they understand the concept because all of them had a sort of landing page of like here's the thing um getting them to talk us through like do you understand what this thing is to start with and then b now that you understand it does Does it seem interesting to you?

And it's like, okay, if it's interesting to you, can you use it? So there's a sort of base level of understanding of usability. And then through the process of them using the tool,

and actually all of the prototypes we were building had elements of AI in, in the actual prototypes. So they were calling APIs, bringing back things. So, for example, one was going to help charities write better emails.

So it'd ask a bunch of question about who they're trying to reach and then it would generate some content based on those answers So for us it was then okay. Once they've generated the text of those answers. We're like, okay.

Does this is this good? And when I said is this good? does this You know, does this seem like something you would send is it better than what you just do a chat GPT?

It was kind of one of the primary questions for us because it was like we were trying to add context text, we were trying to add some best practice into the advice. And so I guess on those levels

is like, understand a bit, understand it. Do they want to use it? Can they use it? Does it do the

job? I guess. Great question.

Managing role overlap, frustration, and healthy collaboration

So you mentioned there's a bit of sort of like clashing between roles and sort of overlap because of, I guess, new AI tools. How do you manage that?

How do you manage that?

I guess the first thing is to know that that will happen now, that that isn't uncomfortable. I don't think I was prepared for how that would actually play out kind of culturally.

And so it was difficult. It was like week two was quite painful.

So I think where I would suggest is like having some structure around that. So there's a sort of

defined sprint owner whose job is to kind of organize the time, make sure people have got clear roles. So you might have, you know, this pair of people are working on this prototype, that pair working on that prototype and sort of splitting it up so it's not like everyone working

on everything I think we found the collaboration around all of these tools very very poor which kind of makes sense that they're still mainly aimed at like a single user or they were at that stage in like sort of October so I think having that structure having the grammars and then just like understanding it's going to be uncomfortable and kind of having a safe space to talk about that

so we had a retro every week as well to talk about what went well what didn't go well what should we do better next week and from that we were then you know adapted our timelines a little bit take

a little bit of pressure off sometimes because um i think particularly if you're a non -technical person it's great you can build those things as i've seen like it's really easy to just paste it you know right i want to build this app but then when you're like but i want it to look

like this or i want it to do this thing and then it doesn't do it you get really really annoying really really annoying and especially if you don't know how to debug the code or that the app itself was just like trolling you saying I fixed this but it hasn't you know and

that that can feel really annoying so I think being prepared for that and allowing there to be spaces where you can vent and kind of come back together in a more positive mindset afterwards helps