Introduction

I'm also going to dig into custom GPTs.

Exploring Custom GPTs

What we just walked through was how to use, how to set up a GPT that's like a variation on the major OpenAI models, but with special instructions.

Function Calling in GPTs

Next, what I want to get into is function calling. We showed you how, you just saw how

The models can write a little bit of Python in order to solve the problem. That's in an open-ended way, right? Like, I'm just gonna write a little bit of Python in order to execute some math or do a little analysis.

Can we teach a model to write a little code against like an arbitrary other program? The answer is yes.

Dashboard Demonstration

And to illustrate this, I've brought up an app of my own. This is a personal analytics dashboard that my company makes.

This is the Fulcra timeline, and it's got a bunch of my data. What I'm going to do is show you how this looks to a human, then what it looks like on the backend as a series of function calls.

I'll show you how a human developer would look at this backend. Then I'll show you how the LLM can view something like this, and then we'll teach it to execute a little code.

This really is just a live dashboard of my life.

This is my location history for the day. And you can see that I live up here in Dover, New Hampshire. And I drove down here to Boston.

And here we are at... Microsoft New England Research and Development.

So this is just part of me doing self-tracking. I keep a personal log of where I've been.

So all of this, literally these visualizations are running in a web app with a visualization framework that we built that itself is built out of a JavaScript framework called Svelte.

And the data that you're looking at comes from a variety of devices. My location tracking is running on my phone. And then I wear a bunch of wearables.

I got like a watch and a smart ring. And so these things are measuring my sleep and my calorie burn down. It's all running into this data store.

I slept pretty well last night.

Thank you. Yes, I have good sleep hygiene, if I may say so. When we get into bed, it's sleep time and that's what I do.

API Documentation and LLM Integration

Part of the reason why I'm oversharing here is to show you how this data has been broken out of its silo in Apple Health so that we can make it programmatically accessible to an LLM. As I said, this is just, if I may say so, nice responsive web design. against an API. And the API to a human looks like this.

So these are literally the function calls that do everything that I just showed you, right? So there are specific, functions here for how to deal with like the Apple SDK, the Apple methods for tracking location. And then all the biometrics is here.

If you're a professional developer, I would hope that a professional developer would say, wow, Michael Tiffany, this is a very well-documented API. I love your docs. They're very well-organized and they have lots of nice little examples. In my opinion, this is good API documentation if you're writing code manually.

Yeah, hit me. Oh, yeah, totally. Yeah.

I love FastAPI. It's fantastic. Yes.

Yeah, so architecturally, by the way, the thing that I just showed you is running in Google Cloud.

And it's this highly distributed architecture where there's a bunch of ingest that's coming from me and from all of the other users who might be tracking what music they listen to and where they are and what their heart rate is and what's next on the calendar and whether the garage door is open or shut. So that's just all streaming into a cloud.

And then there's a bunch of service workers that deal with that. and then clean it up and then make it available to this API.

Custom GPT Configuration

And I have specifically built a custom GPT.

So I'm going to show you how I've tooled up my GPT to talk to exactly this back end.

And the LLMs, like open AIs, don't want to read an API spec like this. This has a lot of visual layout stuff that they don't need.

So instead, what they learn from is something that looks like this. This, as you can read in my URL bar here, this is hitting the Fulker Dynamics API.

you know, slash data slash v0 slash llm slash openapi.json. 1We, like many companies, are now making publicly facing websites that are specifically designed to be consumed by an LLM.

This file in JSON pretty much replicates the user-friendly documentation I just showed you, but in a way that the machine can parse and understand. And this is an open spec. Sometimes this conversationally is called swagger based on the people who made this spec.

But more formally, it is the open API specification Current version is 3.1.

This is itself a sort of public spec. So if you express something as an open API spec, it's going to be machine readable for a large number of models and even other applications on the internet.

And because of this, OpenAI in one of its least well-documented product decisions has this way, when you're making a custom GPT, of teaching the model how to interact with services.

Actually, I have this tab pre-prepared, but I don't want to I want to navigate to this so you can see how this works.

In my left-hand nav, I've got different GPTs, and this one is literally my personal GPT lab.

We're going to go into edit. And this looks familiar when I click Configure.

Here is what most people do when setting up a custom GPT. They give it a name, they give it a picture, a little description, and the heart of this is custom instructions.

And many people, like me, like to organize their custom instructions. So in my case, I've used a little bit of markdown formatting to just organize my instructions.

But the cool thing happens down here with actions. OpenAI calls the custom function calling I'm about to illustrate to you a GPT action, and here's how you do it. We're just going to click the little wheel.

Here under schema is the exact same file that I was just showing you. In fact, you can paste in an OpenAPI schema into this little text box, or you can just give it a URL like the one that I showed you.

And what happens is the model reads this schema and then outputs this kind of list here. So what this list from get metric time series down to get user info is showing you is basically, yes, I understood.

So the model just read all the docs and is like, oh, I get it.

OK. So to get a metric time series, I hit this endpoint. And to get the calendars, I hit this other endpoint.

OK. So now I'm going to switch to a different tab that I prepared just after I arrived.

That's the user facing experience.

Yeah. Yes.

That's coming straight from your code, right? From the decorators in the... Yes.

Yes. It makes it really easy. It makes it really easy.

Standardizing Documentation for LLMs

And it's not just me like, watch this.

I'm going to open up a tab and then we're going to test Michael Tiffany's real time memory. I think docs.stripe.com slash llms.txt. Is this going to work? Oh my god, it worked. Yay.

OK, so check this out. This is Basically, this file lives at the root of Stripe's documentation site. It's called llms.txt.

This is not exactly a standard yet, it's more like a convention. There's a widespread social convention to put text formatted for LLMs in the root of your web directory as llms.txt. And what this is, is simple markdown informing any robot that shows up here, any AI model, it's literally like here is all the documentation that lives on this documentation site, and here's like the URL, the title, and what you should expect to learn here.

So what's beautiful about this is that you can use it to bootstrap a model. Like if you were using a coding assistant and you want to integrate Stripe, It actually takes real human work to even figure out what documentation to tell your AI coding agent to read to help you. So this is meant to, so the LLM can actually figure it out on their own.

It's like reading the index and it's like, oh, the user has asked me how to set up OAuth for pulling in these transaction records. Okay, I found the website. And so it'll follow the URLs here So importantly, we'll load one of these.

Very importantly, every one of these URLs is itself, let's see if Chrome is sophisticated enough to do this. Oh, good, is itself a machine-readable markdown format file. So all the pretty stuff is gone. Every single one of these documentation pages is not meant to be read by a human.

It's meant to be parsed by the robot. So that first thing that I showed you was the index. And the index is all pointing to pages like this so that the LLM doesn't even bother loading the full web page that's going to have the JavaScript and the illustrations it doesn't care about. It just wants to read the text.

Yeah, hit me. Exactly. It's just a social norm. I literally found this one by trying it.

Here, what I want to show you is, I'll collapse this. I just arrived here and this is my custom GPT. This is the ordinary interface. I just wanted to make sure that things were working, so I asked what time it was.

You can see that I did this half an hour ago. Even this, by the way, My custom GPT answered correctly for my time zone. So what it actually did is it hit the Fulker data store to figure out what my native time zone is and then told me it's 6.46 specifically here.

Then I asked, where am I? And it got it right. I love it when the live demos with ChatGPT actually work. They don't always work.

Potential of AI in Personalization

But I wanna make a few broader points that I mean to merely illustrate with this example, which is that I think that, AI tools such as this are going to lead to a great unbundling of software.

The reason I want to show you this is that until very recently, stuff like this sleep stage data would be buried in the app that came with the tracker. And it would only be there.

And in here, it's getting pulled out via the function calling that I just walked you through. by a model that knows how to speak programmatically to the hidden control knobs even behind a web app.

The first web app that I showed you, honestly, Almost all modern web apps are built as a nice, pretty front end that, if you look at the code, is actually just a bunch of APIs that are getting hit probably asynchronously via JavaScript.

So if you can teach the LLM how to deal with those knobs directly, then you can make whatever UI you want. You can pull out the data that you care about and make the visualizations that matter to you in the moment.

And I find this really exciting because what it means is, in effect, the cost of one-off code is going to zero because you don't have to ask a programmer to do it. You can ask your magical AI robot to do it, which means You can ask for things that it would never be cost effective for a product team to develop for you.

You can have a one-off question involving, in my case, personal data science. Like, look at the past five years of my sleeping and tell me when I slept, had the best deep sleep.

And it's not just my app that's being built this way. The Stripe documentation that I showed you is meant to be an example of how we're starting to describe our own software interfaces in a way that the LLMs can understand so that everyone can be empowered to do this almost like custom AI, this custom extraction, and therefore custom question.

I wouldn't be surprised if we're just passing through a transition phase where the way to interface with these AI models is conversationally as this kind of chat experience. I don't think chat is going to be the forever interface to these AI models.

I think that as the coding models get better, and the runtime execution environments give it, the AI models will be able to basically make a nice little interface for you that is suited to the thing you want to do with it. And maybe some of those designs, some of those interfaces become one of your forever interfaces and you just save it.

You're like, gosh, I love this. Remember this. But in many cases, it can just be ephemeral.

And I think this is going to be incredibly empowering to us all because there's so much, like almost every software company worldwide feels like they don't have enough people. And there's some product manager that's sitting there looking at all their feedback and they're voting on what features to actually put into production because they only have so many developers. And so everything we interact with is quite possibly capable of doing more than the interface is showing to you because that product manager only has so many development cards and only so many sprints that they can lead their team to ship.

That constraint is going to go away. We can do much more edge customization on the software that's in our lives.

That's why I wanted to walk all of you through this because some of those opportunities are only available to you if if you want to bust out VS Code and a coding assistant.

But this kind of GPT customization is, I think, within the grasp of us all. You don't need to write any custom code here to teach the GPT how to use these functions if you're just starting with an app that has an open API specification.

The the tuning step, even if this looks a little intimidating, the fact is that tweaking things like this is always a lot easier than just starting from scratch. And the AI model wants to help you.

So it will help you resolve its own bugs.

If you get into a loop of meta-prompting where you start prompting the model about how I want to create a GPT that makes these function calls, walk me through it, and then it'll guide you in its own direction.

All right.

Yes.

Technical Questions and Standards

I have two specific questions. OK. This one might be a little naive. Okay.

So the first question is that I'm just thinking that what is the mechanism via which this particular application is working? So the way I understand it, and please correct me if I'm wrong, the data that pertains to your health that is stored in your Apple devices, that is sort of acting as a ground truth for the LLM, and it's using that to respond to your prompts.

Yeah, yes. So something that I sort of glossed over is all of the work that my company has done to liberate that data from, say, Apple Health. Getting to this place is sort of not normal.

I just wanted to use my app as an example of then that bridge to the LLM. But in this particular example, It'd be hard to build your own GPT on your own health data without my app, because there is some hard stuff that we had to solve to build that bridge. And that involves installing some software on your phone and getting those pipes flowing.

That's actually perfect, because I have a few thumbnails into my follow-up question. when you were sort of explaining about the application, the necessity for the LLM to understand how the APIs within the front end and the back end are communicating back and forth. Yes. And could you just articulate in what specific scenarios does the LLM need to have a holistic understanding of that?

Because the way I see it is that it's basically a microcosm also of this broader macrocosm because it is a small Right? Oh, man. Okay.

Great question, because it's going to give me an opportunity to talk about OpenAPI and then an emerging standard you guys may have heard of called MCP. So I'm going to go back here to this specification.

This, remember, is the OpenAPI specification of all the backend function calls, right? One thing this lacks is, for instance, an explanation as to why the machine should learn this stuff. So this is just a flat explanation of functions for interfacing with the Fulcrum backend.

There's an emerging standard originally developed by Anthropic that now seems to be increasingly adopted called the Model Context Protocol that uses a function description like this as a starting place, but then adds a little bit more context about intent. So the idea with MCP is to be a broad standard for teaching a model about how to use some third party service in order to get some contextual information that'll be useful for answering the user's query.

I even predict that if I were to give this demo in 2026, it would probably be with MCP. Claude, which is Anthropix user facing LLM experience, now supports third party integrations with MCP and OpenAI said they're going to do it.

So right now, this is the only game in town. If you want to teach chat GPT how to interface with any third party, you've got to go through this GPT action. And all it understands is the open API spec. So that's why we're sort of stuck in this lane.

In essence, what I've done here is taught it the function calling. And then this is on a different screen.

These instructions provide a bunch of the intent and guidance about this basically saying, hi, Mr. LLM. I'm teaching you these functions, and here's why. It's because this is a service that knows about health and location and this and that. And so I want you to learn how to use these tools in order to answer the user's question about this personal data.

MCP tries to unify that so that instructions and how to use the tools are combined. And it's probably going to win. Hard to say.

One of the weird things about LLM engineering is, in some ways, it very much reminds me of being a teenager in the late 90s, where there was just this explosion of innovation. It was super cool.

But in the 90s, there was a lot of Unix hackers doing things from the ground up, and then eventually some frameworks appeared, whereas Frameworks like Langchain and a zillion vector database startups seem to come out of nowhere right away. So this time, we're going through a period that civilizationally feels like the late 90s.

But I guess a whole bunch of people who remember the 90s were like, The framework people made a lot of money. Let's get in on this. And I think they went too fast.

So I thought, a year and a half ago, that building things on Langchain, an emergingly popular framework, was a good idea and would save time. And then honestly, we started playing with Langchain, and we're like, I don't know, man. This seems like it just has it. It has some strengths, but then it has its own set of bugs.

So do we even want to bother with this? Let's just do some API calling directly.

So the uncertainty in my voice is that maybe MCP is going to be like that too, and it's going to just join the graveyard of failed framework efforts. But this one actually seems like it has legs. So it's probably worth learning.

How was that? Perfect. Cool. All right. I was thrilled to share this with you.

Yes?

Consumer Applications of LLMs

I'm not so technically savvy to understand the scripts and such that are used to actually do the MLM here. But from a customer experience side of things, I'm curious as to whether or not you've begun to apply this type of conversational dialogue to consumers of Fulcrum.

The dashboard and that type of data and what that implementation might actually look like.

Yeah, so here's what we're working on.

I've been an entrepreneur for a long time. And something unique that I hope you can see on display here is that I'm trying to make a lovely front end, nice experience. But we really are trying to build a software company in the age where maybe no one cares about the UI because they just want to plug the chatbot into it.

So there's a whole bunch of tricky questions right now about how do we build an open software experience that lets people do custom mashups.

Oh, in fact, look at this. This is some data science that this user of ours named Nicole, she shared this Streamlit dashboard she made of this crazy analysis on her own step count. because she could. Just like an ordinary user, I think she likes the visualizations, but she wanted to just hit the API directly and just do her own thing, which I think is super cool.

What are the Where are we guiding people when it comes to LLM to database backend?

I'm just sort of sick of chatbots being bolted onto every damn thing. So we have not even done that.

Conversational AI Initiatives

The thing that is exciting me the most actually is empowering a personal AI to proactively message you, right? So like if, like when I get home, my,

you can almost uniquely identify because of my data set that I just got home from giving a presentation, which is different than when I just get home from shopping or when I just get home from an ordinary workday. And so it might be the right time to get me to remember one of my relaxation routines.

or to remind me of a habit that I would otherwise forget about because I'm getting home later than I usually am. So I am trying to explore basically conversational AI, but the AI starts the conversation, and it's based on something about how it can see me.

But I feel like... I feel like this is just an experiment.

There's a lot of you just have to build it and then see how it feels. This is the right way to put it.

Challenges of Situational Aware AI

Think about movie AIs, like all the good ones. I'm just going to pick Jarvis for a second, Iron Man's AI.

In the movies, there's this invisible problem that's just always solved by Hollywood magic, which is somehow the AI can see the protagonist. Like Iron Man does not say, I'm in Miami and I'm stressed out right now and I need some help. Jarvis is like, your heart rate is dangerously spiking.

The magical AI just has situational awareness over you. So to what extent is that helpful? Like to what extent can we make an AI a guardian angel that feels like that and doesn't feel like a hall monitor?

And you kind of have to just build it and then vibe check it, right? Yeah.

So this is an area of exploration. I don't have the right answer yet. Cool.

Conclusion

All right. Thank you all. This was wonderful.