Introduction

Thanks for coming to check out my talk. I'm really excited to be here, so I really appreciate the opportunity from Mindstone, so thank you.

Today's Topic

Today I'm going to talk about AI data products and how we can build them. I'll talk about some lessons that I've learned over the course of my many years of building and releasing AI products powered by like GPT and all those amazing models.

And then I'll try and go through some practical use cases.

About the Speaker

So my name is Gareth. Just a little background about me.

I'm a full-stack data professional, and I really love building software. It's something that I've been really interested in over the course of my career, and it's really only really caught on maybe over these past five years or so.

I'm super interested in product-led growth, A-B testing, building experiments, experiment design, and just testing new products in general. I love digital strategy and, of course, AI. That's why you guys are here.

Current Work

So some of the brands that I've worked with in the past, you can see right there, some great brands. And right now I'm currently running my own business. I'm freelancing on analytics and data science and AI, and I'm building my first startup called Prototyper.ai.

So Prototyper is an app that helps businesses test their business assumptions with data and AI. It's pretty, pretty exciting.

Agenda

So today, the agenda, I'm going to walk you through some things that I wish I knew when I first started out on this crazy journey. I'm going to talk about some practical use cases, about how I'm using GPT to help support data analysis. And if I have time, I'll explore an AI workflow that I've been building to help validate some business assumptions.

Three Key Learnings

So three things I wish I knew when I started out in AI.

Lesson One

So I wanted to show this quote because I think it really spoke to me a lot. I was watching Aravind Srinivas, the CEO of Perplexity. He was on a YouTube podcast recently, And it really spoke to kind of my modus operandi about how I'm building products.

And so he said, and this really resonated with me, he said, you want to create a product that gets better as new models get introduced. I think that's really, really interesting. And I'm going to walk through just a couple of the things that he said during that interview because I think it really, really hits home.

And so then the more people that use your product, you'll collect more data to fine-tune and improve your product. And so this is something he took from Larry Page, who's the co-founder of Google.

He said, you want to identify a problem where if you worked on it, you would benefit from the advances made in AI. And the product would get better. And because more people use it, that helps you create more data for your AI to get better. And that creates a flywheel.

Product Improvement

And so I'm going to talk a little bit about a flywheel and what that is. And what's interesting is it's not actually easy to have this property for most companies. And that's why a lot of them are kind of struggling to identify where can they use AI, right?

And to the CEO of Perplexity, it's obviously very obvious and beautiful. So when you think about that first lesson, you want to create a product that gets better as new models are introduced. Think about all of these improvements that we've seen since GPT-3 came out in 2020.

That's like two years before ChatGPT. We've seen like 1,000x increase in the context window size. We've gone from uploading like a paper

one or two pages to over two million tokens of context, so with Google Gemini. We're seeing models, they now understand just more than text, right? They understand audio, they understand video, and they understand images.

We've gone from this general ability of the model being on par with a high school student to now having PhD-level reasoning models like O1. We can search the web with ChatGPT. We can generate images, and we can also have conversations.

It's really, really exciting. So what I'd say is the key takeaway from my first kind of lesson is, really, if you have an idea, start building. Now is the best time, right?

Embrace New Models

1And one of the things that I love, that I'm really excited about, is whenever a new model comes out, I'm so excited because, really, I can update that one line of code, and I'm now using the newest and latest and greatest model. and your app just gets so much better, right? Just because you've just kind of changed one little thing, right?

And so if the model isn't good enough for today for your project, it may be better tomorrow. Right, so, and I'm gonna call on my hometown legend, Wayne Gretzky, and he said a really great quote. He says, skate to where the puck is going, not to where it is being.

I think that's really relevant in AI right now. So if you're experimenting with it, you're trying to figure out how to use it, maybe the model is just not like, delivering the output that you kind of expected, it may tomorrow, right?

Lesson Two

So keep at it. All you need to do is eventually when the next GPT-5 comes out or whatever, maybe that's good enough for you. Lesson number two.

Prompt Engineering

So I'd say learn how to design better prompts that improve your outputs using some sort of prompt engineering kind of framework. What is a prompt engineering framework? Well, it's a system that's designed to streamline the process of creating high quality prompts.

So this helps you be more consistent, more efficient, and obviously it'll help you drive better business outcomes. Why do you want to use it? Well, we want to get better outputs, right?

And so these frameworks have been developed from leading AI researchers using their own tested techniques, right? And by using some of these frameworks, you start to learn how to better communicate with these LLMs. Your prompts become instantly scalable, reproducible by other people on your team.

And prompts can end up actually being a lot more clear and better structured. So how do you choose one? There's lots of different prompting frameworks out there like Tree of Thought, Chain of Thought.

RPG Framework

Tons, right? But I would say if you're still kind of investigating how to write the best prompts, start with something simple, right? So I'm going to walk you through one that I use that I developed and it's called the RPG framework.

And so RPG stands for role, problem and guidance. So when you're thinking about your prompts and how you're building them in ChatGPT or whatnot, you can actually break your prompt down into three distinct sections. So you have the role that you want the model to take on.

So you would write this in your prompt. You would say something like, you are blank professional specializing in this. And the problem that you want this model to help you solve

is you could say something like, a user needs your help with this task, and your objective is to do this. And the last piece is guidance. And guidance is probably the most important piece of this framework, because when you think about it, these LLMs are a lot like us in a lot of ways.

And you have a role at a company. You're probably working on problems, I would imagine, that you're trying to solve. and you look to leadership at your companies to provide guidance.

So you are the leader when you're communicating with these models right now. So the type of guidance that I like to deliver when I'm working with these models, I usually like to offer guidance as like an ordered list, like a numbered list or bulleted list of, hey, these are the requirements that I want you to kind of consider. I will include things like examples of inputs and outputs that I want the model to follow.

I'll explain what's the output look like? What kind of format do I want it as? Do I want it as code?

Do I want it as JSON? Do I want it as text? And then the last piece of guidance, which I recommend, is put on that responsible AI hat.

Outline what's an acceptable input, and then just tell the model to refuse anything that doesn't fit that criteria. Because at the end of the day, we want our models completing the tasks they're designed for. And if someone's trying to break it or do whatever, sure, that's good to know, but it's not relevant.

So that's the RPG framework kind of in a nutshell.

Lesson Three

And the next lesson that I've kind of learned that I really wish I knew is really all around like architecting data pipelines. And so one of the things that I think you could do or that you should start considering to do when you're architecting your data pipelines for your AI products is to think about like how you can use your pipeline to recursively improve the AI model and figure out like how you can create that data flywheel.

So kind of going back to what the CEO of Perplexity was saying, he he was building like data flywheels because the more people they used it, the more data he collected. And then he figured out how to send that data back into into the model to improve it.

Architecting Data Pipelines

So like our goal when we're building our AI data products is obviously we want them to solve a problem, but we also would love to have them start to figure out how they can improve themselves, right? So as you build new AI data products and you release them into market,

You have your users engaging them, and then you start collecting a lot of data. And within that data, there is a signal that you can find, right? To then take that and then put it back into your AI data product, right? Just like that. Send it back in, right? And then you keep doing that, right? You keep finding these good examples, and then you keep training and making your model better.

Practical Example

I want to show you an example of something that I used in production. And like I said, I'm building a no-code data platform that helps companies research, generate, and measure ideas faster with large language models.

And one of the things that I created in this app was a UI generator. And it helps people generate different types of UI. It can generate web pages. And even with O1, it can start to generate these pretty cool looking games.

And so why did I implement a flywheel? Well, I was actually experiencing a problem when I first started figuring out how to get these models to generate different user interfaces. The problem that I was experiencing was that I always had to restyle all these generic outputs. And I was just like, oh, this takes so much time. I just don't want to keep restyling stuff. I want something that's in my style. I love design. And what I wanted really was a database of, say, my designs that the model could then reference and start outputting UI based off of what I like, what I want to do, how I want to show up.

and so this is this is the pipeline that it builds this is the ui with ai data flywheel and so the basic idea is as you save more ui the model uses that data to help personalize its output towards your style right so If I can walk you through this. Imagine you're a designer, you have your own style, and you essentially have an idea for maybe some new UI that you want to test.

You can write a prompt, you click generate, and then the app would then go fetch some of your saved UIs. It would take those examples, and it would place it within your prompt, your RPG prompt. We put some examples in there. And then we would send it off to Anthropic using and get Sonnet 3.5 to send us some model output.

And we can continue kind of iterating on that. And when we're happy, we can then save that final UI design. You save all the HTML, CSS, the JavaScript. It goes back into your database. And so when you kind of come back the next day and you want to create something else or whatever, then you have this database of designs that the model can reference, if that makes sense.

AI in Analytics

So now I'm going to just talk a little bit about analytics. So I'm an analytics guy. I love data. And as you can kind of see with some of these pipelines, I'm making all these crazy things.

Use Case: KPI Analysis

So I'm going to talk a little bit about how I'm using AI to support my analytics journey. Just to take a step back, so the app that I created called Prototyper, I integrated it with Google Analytics because I love Google Analytics. And so I wanted to actually see if I could fine tune some models, some GPT models to help me just chat with the API.

So one of the things that I wanted to do, the first use case was to use AI to help surface segments that are driving KPI up or down. So imagine your boss comes over to you one day and he's like, Gareth, what's going on here? Like sales are down like 50%. Can you drop everything and just like get on this thing? And like, yeah, sure. Like this is going to take me the whole day. I got to drop everything.

And so I was like, the problem that I'm solving for here is, is like, it takes time for me as an analyst or a data person to like kind of unpack all this data. Right. I wanted to, I want to understand like what's driving this KPI up or down. And so I thought, okay, well, these AI models are so amazing at like synthesizing so much data that why not use them to try and help me analyze this KPI across a number of different segments and see if it can write a nice summary about what's kind of happening.

And so here you can kind of see like here's like for yesterday, I had 111 users come to my to some landing pages. And I just asked it to write me a summary of what's driving that increase of 94%. And so it kind of worked through, and it went and looked at sources of traffic, looked at countries, looked at what landing pages I had and what devices I was. And it just said, these are all the segments that are driving this KPI up. And so there we go. Now I can just validate it really quickly. And it's really, really good. It's working really well.

So this is the pipeline, how that all works. Basically, you open up your scorecard. You hit Generate AI Insights. It goes, it fetches all the queries, sends that data to Google Analytics, and it comes back. And then we build a prompt to analyze that response, send it to GBD 4.0, and then you get that. So it's pretty cool. It works pretty well.

And here I can see if I can show you. I prayed to a demo god, so we'll see. See if this works. We'll just hit Generate AI Insights. And so this is the 111. So it's looking at yesterday versus the previous day. I don't know how long this will take, but we'll see. There we go. Perfect.

So you can see it has the number correctly. We have 95%. It says Google right there. Let's just do a quick little spot check. It's grown 110%. And 110%. So there we go.

That's perfect. So the great thing about using a tool like this is like it's doing a lot of analysis and then you kind of become the fact checker, right? Making sure that it outputs correctly.

And so how I kind of designed this was it's gonna take all that data, but these tables right here, those tables are actually the raw data that I transformed myself using different functions, right? So I can feel confident that this data is always gonna be correct. And if the summary comes back with a little bit of, I don't know, something wrong, then I can just tweak it. I'll just say, oh, OK, well, I didn't get it this time. Oh, well, send it off to marketing. There we go. Here's what's going on, guys.

Use Case: Chat with Data

And then the next use case that I was thinking of was I want to chat with my data using GPD4. The problem that I was solving was really it's like I want to help make data more accessible in companies and whatnot.

And so the problem at companies is not everyone at a company really understands how to build data or build queries and request data. They don't know how to do it. But they know the natural language query that they want.

They know that they can say, hey, Gareth, can you give me some data for what our conversion rates are like in the past 30 days? I'm like, yeah, sure. I want to be spending all my time doing this. I want to do higher level, more higher value analysis.

Right, so what I did was I created a solution where I fine-tuned GPT-4.0 on a number of different things where it was some natural language queries and also the corresponding Google Analytics for API queries. And so I just kind of created this fine-tuned model and then hooked it up and it seems to work.

So you can see kind of how this how this works right here. So this is what an example of what it would look like. And I know I'm kind of running probably short on time here, but I just want to show you one more kind of thing that I'm working on.

AI in Workflows

So think about like how how you can integrate AI into your workflows. Right. So so

Idea Validation

I love testing ideas. I love just figuring out what's the problem that I want to solve, and I want to get an idea in the market, see if I can validate my business assumptions.

So I've created this workflow where I use steps one through six I use AI to help me define the problem, right? So AI is fantastic at doing research, right?

So you say, hey, I got a problem. Users aren't, I can't get good feedback on my UI designs, right? Say that's a problem.

And then the model would go out, and it would say, oh, OK, well, I searched the web, and here's some examples of companies trying to address this problem. Here's some more data. And then the model can then help you, say, generate a hypothesis that you want to test so you can hold your idea accountable.

And then why not just keep going? Let's generate the prototype. Let's generate some UI.

Let's build a survey that can measure that idea from customers. And then that's also just generate a dashboard so we can measure how users are engaging with your product, right?

So you could use AI across all of these steps and get your idea kind of ready for launch. And then the next step really is like, how can we How can we get this live?

Let's speak to a developer, or maybe someone on your team can help you. And it's like, basically, this is the problem we're trying to solve. I have this hypothesis. Let's test it. Let's test and learn, move fast, and hopefully not break too much stuff.

Conclusion

So that's my talk in summary. So that's it, that's my talk.