We're good for now.

So yeah, I wanted to just talk today about an actual real world practical use case that we were running into with one of our clients. Just kind of get your perspective on it and kind of talk about how we can Essentially use AI practically.

And don't think of this as me trying to sell anything anyway. But think of this as trying to all brainstorm together on how we might try to use AI to solve a real world problem.

And by the way, I'm Natan. I'm the founder of an AI company.

So let me give you guys some context on this. So let's say you're an organization.

And every month, you have proposals or something that you want to fund, like maybe a project. And what you want to do is you want to get the best people working on problems that you're interested in.

So what you do is you'll put out this thing that's called an RFI. And you're like, hey, I'm looking for something related to AI and machine learning or health care, right?

And you're one of these organizations. you get thousands of people around the world who see your RFI and are building cool, unique, innovative ideas, and they'll kind of put in responses.

And these documents, they might be anywhere between five and 25 pages long. There might be like thousands of these applications.

And right now you have people within your organization who are like experts in a field, right? And these experts will kind of get rooted to a proposal and they'll read it. You'll have like maybe three or five people go through these documents and review it.

And then from there, it can take a while. But after you get all the feedback and review it all, you'll get a score. And then the teams who have the best proposal or application might get an award or a contract or an opportunity.

So does that make sense? Any questions on that?

So thousands of proposals, people within our organization that want to review them. Maybe it takes around six to nine months from when someone actually submits a proposal to when you get a decision like, yes, no, we were awarded, which is a lot of time. And maybe the process looks like this.

You receive all the proposals. You want to categorize or route it to the right point of contact. The point of contact will review the proposal and provide a score. You may want to merge the score from different reviewers, and then maybe you'll have some final compliance checks, and then you'll send the results.

And the question for you guys is how might we use AI to help automate this process a little bit to save the time and cost in a way that's really accurate, fast, cheap, and is reliable? And maybe you still want the humans to be reviewing this proposal because you value their expertise and don't want to trust in AI because you need high quality decisions, or maybe in the future they'll just be AI, right?

So first I'm like the business case here, right? Let's say you have an organization and right now they have 80 people going through this proposals and you have a thousand documents and maybe each of these people are paid like 70 K a year and it's taking them, let's say like six months to go through these documents and you want to kind of have a solution that will kind of do this really quickly in like 10 days.

Right? So, The question is, how would you do this, right?

Because before I share this slide, oh, sorry, give me a second. I'm curious if you guys have any ideas on how you might go about trying to build something like this. Yeah.

Start by saying something that I would avoid, and it's asking for scores. because usually this is the area where guests last the most. So I will try to structure the system in a way that I can provide different alternatives and then have the AI to choose one or other path.

And based on the paths, I will build some algorithm to decide on a score where it's mostly aligned to my goals or the criteria that I'm trying to solve. The proposal is supposed to solve some problem.

So I will see the alignment and the potential issues with that proposal and try to evaluate it on that layer. So kind of moving away from score start to goal.

OK. It's an interesting point. I like the idea.

Anyone else?

So let's say we decide that we're like, okay, we receive all these applications. And maybe the first step is we have maybe like a million people apply to this thing, right? So we wanna like filter out anything that just doesn't fit the initial criteria. Maybe there's like too many pages or there's like asking for more money than the award has, right?

Or something that's completely off topic or written by a chat GPT we wanna like exclude, right? And say we do that, the next step is we want to essentially make sure that each person who's an expert is reading the right proposal.

So we wanna route the proposals that we get to like the accurate POC at each organization, who may be someone who knows a lot about like AI or read AI proposals, but someone who knows about healthcare or read healthcare proposals. then we want to somehow give them maybe an AI-assisted score, but we want them to score themselves, and we want them to put the best score on, and maybe we wanna merge it, and then send the final score in a really rapid process.

So that's like a reasonable maybe approach. There could be better ways.

So let's say the first step, we have these proposals and we want to kind of do these filters. So we're like, hey, does this proposal have a title? Does it have an abstract? Are there specific keywords that we shouldn't include here because it's confidential? Is there any information with proposals that are like above a certain amount of money?

that looks like this where we have all these people who are like specialists in specific fields like Dr. Johnson who knows a lot about autonomy and Dr. Clark knows about energy and we have these proposals and we don't know what category they are in but maybe there's like a high budget so there's lots of decisions being made here and we want to kind of be able to give each proposal we see a category prediction And then based on the category prediction we have, we want to actually route it to the person who knows about that category. So they're reviewing it rather than something like rando, right?

Does that make sense? Cool.

So I'm just going to go really quickly because I never really have ever demoed our product here. So I just wanted to show like how like this might work if you use like Anno. So you could just hit this get started button and you go to this like label flow. And you can upload, maybe I have here on my screen, I have some proposals.

And I have these training proposals. And maybe it looks like this, like an example proposal that someone's written. And this could be my training set.

So what I might want to do here is I might want to upload. these tasks. And I'll call this like my proposals or whatever. And this is text classification, because they're just like a few categories. So we're going to upload these six training proposals here. upload files. And I've uploaded these, right?

So cool. And we have these categories, right? So we have autonomy. We have something that's like cybersecurity, right? We have healthcare, I believe, or health tech. We have energy. And we have logistics, right? So these are documents.

These are our categories. We might just try to do some initial heuristics, right? Maybe if there's like the word AI in this proposal, then we think that's related to like autonomy or something like that. We could add like other rules like entities.

And now maybe we see these kind of proposals here. And you can kind of see each document. Maybe you want to label this one as like an energy item. And this one as like logistics, right? And so you're kind of labeling these items. And then as you essentially might label these kind of proposals,

you might train a model to try to make predictions on like how well I can categorize these, right? So I have this history of like the documents I've labeled and I make these labels. Now I have these predictions and maybe a probability score, right? So each of these are kind of with associated category and I could just download this. So now I ideally can do some sort of script where I have these categories. I can essentially merge the classified category and I have this score, right? So I wanna merge this to like the right POC who knows about that topic or field. And so I match essentially Dr. Johnson who's autonomy with the proposal I predicted about autonomy, right?

Now I want to basically maybe email all these people about the proposals I have, right? To kind of give them the opportunity to review them. So you can kind of maybe use something like this where you can use some like automated emailing or you can email everyone individually and maybe you send them some sort of email that looks something like this, right? Yeah, and you maybe you send them like a link to some portal and say, like, hey, Dr. Johnson, here's this autonomy proposal. Are you able to kind of review it in some sort of portal?

So Dr. Johnson gets this link, he kind of goes and clicks on this link. And maybe there's some sort of way that there's like these projects. So you might have a project, this will be called like the proposal project, right? And maybe we'll have like the different people who should have access to specific things.

So this is the Tommy case. So I want to essentially add to this autonomy project, I want to add a Dr. Johnson, who will be like maybe an admin, right. And then I'll have someone who just also an autonomy that will just see the proposal, but maybe not do anything with it. So there'll be like annotator, right. So now I have all this information.

And I want to actually go and help generate these proposal predictions. So I have this data set. I have these documents. I have these proposals.

And what I want to do is I want to see if there's a way that I can basically help use AI to assist with this actual proposal writing. So I upload these documents. And while I'm doing this, maybe I might want to evaluate this in different kind of components, maybe commercialization, which is basically like,

Um, is there, like, a path to, like, making money, right? Or, as Ava's talk was about, like, revenue, I don't know, right? Or, um, maybe I want to know if there's, like, um, like, accept, right? Like, um, should I accept this, like, submission as, like, good or Maybe that's like a categorical question. So this could be like, great, like, yes.

No, and like, maybe or whatever, right? And what I might want to do is I can add these kind of criteria that I might use to evaluate this. And I'll hit this train model button, which essentially will help me go through all these sort of proposals I have. And it will give me like an AI recommendation.

And you can imagine maybe this first proposal, yes, there is a right. And maybe I should accept it, right? And you can have these different people that are in charge of writing these things. And as you kind of do this and you hit this sort of train button, the model trains in the background.

So I actually like ran this ahead of time to take some time. But you can essentially see, like sometimes you can see basically your existing training runs. So you can see... Let's see if this has it.

Training runs

Oh, it should be here. Yeah, you can kind of see the actual predictions from the model here. And as you can see, sometimes the model might not always make these great predictions. But then over time, as you would essentially go through and actually make more items, you can actually adjust and edit these kind of items and do some sort of fine tuning to improve the proposal. And as you kind of do that, you're kind of improving the model's output over time.

Are we doing that?

Yeah, so this is like the person who's currently right now like going through like whatever portion of proposals they're assigned and maybe like like like providing the writing is for like 10,000 and the idea is like rather than having to do that for all these, maybe you have the AI that can help like assist that. Yes, you can kind of think, like, right now there's, like, the actual person who's, like, writing the reviews, and then there's, like, the admin who will merge all of them, and then the admin will kind of decide, like, hey, this will actually... It's, like, a good proposal, and I should, like, give it a contractor award. And so, like, the reviewer basically would only have access to the document and write the reviews.

The admin essentially can, like, see all the things that... Everyone is kind of done, and they can see, like, all the items. And you can kind of, like, get all these scores of each essential, like, person you've done and how well they're doing so you know if, like, it's valid or not.

Why did you choose not to have the... I'm not sure if it's the right term, but that final doctor or so, why not have that person review it? And so if I... When I'm building things, I want to have the training as part of the... workflow so that some percentage of cases I would misdirect knowingly.

So let's say like a boundary, something that scores just under the threshold, feed it into the review workflow anyway and ask for feedback just to keep training the models so that the ones that you've rejected with the AI do get a second shot in case that was rejected incorrectly and there's like a human in the room kind of feedback.

So why did you, again sorry, the question was why did you choose to have separate roles with this for training rather than having the subject matter expert give feedback so that if something is miscategorized, they're catching it in the process.

Yeah, exactly. So I think we basically agree.

So essentially, you would have this AI model that might make the initial predictions. But the AI model might not be super good.

So then you have these subject matter experts that see the documents and then are labeling. These could be people or AIs. And you might take that input and then adjust the model to like learn from that to be better.

Hence like maybe in the future saving it from being like 20 minutes per document to like five or something. And that's kind of the approach.

And then it's like, how many inputs do you need to get to a point where the AI is good enough? And that's dependent on the data set and task, but yeah. So that's the general idea.

And then there's this kind of concept of evaluation. So you only know this is really good if, If you're able to evaluate both the answers and the categories, so there's kind of metrics.

So yeah, I know we're almost at pizza time. So I just want to close by saying why this matters.

I think you can actually think about this in a real world case, where not only is it super time consuming and costly to just be going through all these proposals by hand, manually reviewing them. And this whole process just takes forever. So anyone who will apply to these proposals takes six to nine months.

1But also, you can imagine, because this is super tedious and time consuming, you might have a lot of people that just are seeing these documents, going to page two, taking a snooze, then putting some random score in. So instead, you want to leverage AI to help get more accurate answers and submissions so you can actually get really the best proposals from people that submit things to actually be awarded, which can help solve the problem that the customers have. So very important to not only do this in a timely and cost effective way to help everyone in the process, but to get the best proposals.

I won't ramble on, but thanks for your time. I think we can take questions as we eat pizza and drinks. But come find me, and thank you.