Introduction

Okay.

Hi, nice to meet you.

I'm Emma.

I'm the founder of Velvet.

We make an engineering tool for warehousing your AI requests to your database.

And I'll describe more about what that means as we go.

Can I get a pulse on the room?

How many of you are software engineers?

Oh, awesome.

A lot of you.

And then how many of you are working at companies that are actually using AI in your products already?

Cool, and then how many people want to build something with AI?

And how many people just want to be a consumer of AI?

OK, cool.

Super helpful.

Understanding AI Usage

Three Buckets of AI Applications

So three buckets of what you can use AI for, and there's a lot more.

But the first one is making decisions.

So something like, should this person get a loan?

Yes or no.

The second one is generating text or images.

And that's the most common thing you probably see today, if it comes back.

When you see reviews on Amazon, it's going through hundreds of reviews, summarizing it, and then putting it at the top.

That's kind of like a very common use case you're seeing today.

And then the third, which I think will become way more popular over time and is not as

known in the consumer space is performing some kind of batch job.

So like going through and classifying images or books and then letting an AI return something interesting about it.

That's really expensive, it's really valuable, but it's completely behind the scenes.

So I'm very oversimplifying, but three ways to start using AI in your app.

Getting Started with AI

The first is connecting to the API.

So that's just adding the code snippet to your code base.

The second is telling the LLM what to do.

So let's say you're making a book app.

saying you're a librarian, speak to people like you're a librarian and tell them which books they should read.

And then finally, once you have some version of product market fit, it's about optimizing.

So how do you make this better?

How do you make it more cost effective?

How do you actually create value with the AI beyond just innovation?

And these LLMs are super powerful, but they're also really expensive and generic.

So OpenAI is meant to just survey the entire internet and then automate things for you.

And it's really good at certain things, but for some very niche applications, you have to tell it what to do.

And getting beyond product market fit requires quite a lot of R&D and lifts to get to that place where it creates value long term.

And that might mean you have to fine tune your own model or build your own technology as well.

About Velvet

So I'm the founder of Velvet and we help you do that.

Once you get to the place where you have an AI integration and you want to make it better, we sit as a proxy between your application and the AI model.

And so why that matters is because you send something to OpenAI, you receive something from OpenAI, and if you don't warehouse it anywhere, you can't access it.

So you can't see what's happening as a baseline and then do things like cost analysis, fine tuning, batch jobs, whatever it might be.

And so we help you do that directly in your database, which means you can take it and do whatever you want with it.

Our first feature was an AI SQL editor and you can see it working here and I'll show you more in the demo.

But we learned a ton from using OpenAI for this feature and that's where a lot of these learnings came from.

Demo and Features

So I'm going to show you a demo that will make a lot more sense.

So this is just an open source AI chat app.

And this is the most basic implementation of OpenAI.

I can ask it a question.

and you'll see data starting to populate on the right hand side sorry the screen is a little washed out i'll do another one okay so you see over here there's a few requests that got warehoused and this is showing an example of what happens when it's in your database i can open one of those up and you can see the json blurbs of the data that's being sent and received from openai so this is the request we sent over it says

I asked it for a joke and then it responded with something.

You can see the response.

It said, what do you call fake spaghetti?

Then you see a bunch of data here that can be queried.

Then in the metadata section, this is where you can actually include custom data about your application.

So the only opinion we're including here is the cost.

But you can include things like which model are you using, what feature are you attributing it to, or really anything.

And so this gets much more interesting when you're building deeper AI integrations like agentic systems where you can query it and uncover exactly what's happening here.

Customer Use Cases

So I'll show you a workspace that's already set up.

So this is an example of what you'll get once your logs are flowing in.

This is one of our biggest customers that's spending like $6,000 a day on OpenAI.

And you can see we're breaking down the number of requests.

We help them implement caching, and so you can see that breakdown.

And you can also see how much money they saved by implementing things like caching.

They also are using batching on the back end to do a ton of classification.

And so we're the only proxy that actually unwraps the batch API so that you can do really accurate cost analysis on top of batching of how much money am I actually spending on a batch process as opposed to chat completions, for example.

What's more interesting than just those basic analytics is your application is very specific.

And so with these logs, you can actually uncover a lot of information about your product and also how the models are performing.

So first I'll ask a question, get all of the requests from the last month.

So this is the feature I was talking about that uses AI to write SQL.

And you'll see that pretty quickly we get a list of requests.

And again, I can open that up and see all of the queryable information here.

So that's just a baseline.

And then I'll ask for a breakdown of costs per model.

So this becomes more interesting when you want to evaluate new models.

Let's say you were using a bigger model, and now you want to use GPT 4.0 Mini, which is a much smaller and more cost-effective model.

So I can see here the cost comparison between 3.5 Turbo, 4.0, and 4.0 Mini.

And you can also use these data sets to run complete evaluations, like comparing the requests and responses as well.

1And then finally, something that's more specific to this app, I can break down the environment, so production versus development, and then also the service, which ties it to a feature.

So let's say I want to understand how much the summary feature cost

compared to something else.

And so I can see that information here.

So this is just one example.

Cost analysis is just like one thing that people want to do, but you can really figure out how to query anything in this system as well.

Our Learnings

So we learned about this process when we started warehousing our own logs and we use it to query things with JSON and SQL and then generate data sets for things like fine tuning.

And then we started offering it to our customers.

And so now we have a ton of customers who are very heavy users of open AI that are using this to automate their entire process.

One of our biggest customers

We launched with them this summer and were able to handle 1,500 requests per second.

So we're really meant for scale and for these companies that have gone past product market fit and really want to optimize their systems and be able to fine tune their own models and evaluate models and really get to that next level.

it's super easy to set up although we're made for enterprise scale it's just two lines of code to get started and we handle all the queuing caching batch analysis for you and every single log will be logged to your own database which is a huge differentiator because you can take that data and go to any model with it or do anything with it long term

Conclusion

If you'd like to follow us, you can scan this code and join our mailing list and also feel free to email me anytime.

And happy to take any questions now or at drinks later as well.