Introduction

So nice to be back. It's been a while since I've been here. I've been a little bit busy, but it's really lovely.

Welcome and Session Setup

Thank you everyone for joining us today on a little bit of a, you know, the weather isn't fantastic, but let's hope that we all learn some things today. Now, I like to start off with, this is a live demo and yeah, caveat, it's a live demo.

All right, so let me just go in and bookmarks and hide bookmarks bar. All right.

Event Context and Tools

Thank you for the lovely introduction and it's great to be here. I think this is the best AI event in London, especially it's free.

We give you pizza and some beer at the end. As you can see, I'm using Gamma here and it's really easy to use. Here we go.

What Is RAG and Why Use It

I'm going to talk about RAG with Copilot Studio. Some of you might be familiar with the tool or not. Some of you might be familiar with Copilot, the word.

It's banded around to quite a number of things. Copilot Microsoft 365, if you've got that in your business, you'll be able to do some of these things because you've got Copilot Studio under the hood.

you'll be able to do RAG and share the agent that you create with your members of staff or select groups. Under the hood, you've got Copilot Studio, but you don't actually see Copilot Studio in all of its glory, but we're going to have a look at that today.

RAG in a Nutshell

I wanted to introduce you RAG, which is very popular. It's a technique that improves the quality of the responses of your chatbot.

There's lots of benefits. We'll have a look at them as well.

I'm going to jump into a demo fairly quickly, so make sure that we get through it in time.

Embeddings and Vectors Explained

I might even touch on a tokenizer, but this is essentially what is happening. comes in, it is turned into vectors, and you can think of vectors, this gets a bit techy but stay with me, we're on a good ride. The vectors can be thought of like how similar a word is to another word.

Your data over here, You've got all these, where is that word in similarity? That's what essentially happens.

For example, puppy is very similar to dog, cat, and kitten, but then you put a human in. It goes into an index, and eventually it gets served up to the end user.

Why RAG Matters for Business Use Cases

Now, this enriches the LLM, because LLMs, which is like ChatGTP, or your Claude, or whatever it is, it's a large language model. And LLM has a cutoff date. So they train it up to a certain date. So it doesn't have the latest information.

Now you can go on and say yes, search the web and that sort of thing as well. But just say you've got a business case where you want to have a HR system or an IT help support system with business context sensitive data. You're not gonna put it on your website.

You wanna keep it and you wanna share it and you want people to go in and self serve. For example, they might want to know how many leave days do I have, or how do I fix my computer? We all know turn on and off again, that's the first thing you do.

But from there, it's something that you can have business context sensitive and keep it secure within your business domain, which is really important.

Live Demo: Building an HR Copilot in Copilot Studio

So we're going to run straight over to a demo. I've got copilotstudio.microsoft.com set up here. If you have a development environment, you may have access to this.

As I said, if you've got Microsoft 365 Copilot, this is actually under the hood. In 365 Copilot, I'm sorry I don't have that to be able to show you here on my machine, but you'll notice that there's a little icon for agent.

you can go in there and create an agent and feed it documents from your HR or your IT system or point it to SharePoint sites or if you really want to get into it, you can create an index with a database such as AI Search with Azure and that can feed a lot larger corpus of data.

Creating the Agent

We're going to create an agent, and I didn't bring my glasses, so I hope I'm clicking on the right things. This is gonna be great. All right, all right.

So the first thing, this is great because we're going to create a co-pilot with a co-pilot, okay? So I'm just gonna call it HR and resource. All right.

I really should have brought my glasses. Anyway, let's see how we go.

And I've got... a predefined, you don't want to see me typing all of that out. I'm going to basically put a description here. I'm just going to paste that in.

You are a helpful agent that Provides information to our staff. All right.

Now, we're going to come back and do the knowledge in a second because I've realized I didn't have it on describe, which is fine. So I'm just going to do that all again. Control V and create. All right.

Microsoft's changed the default over. You should actually just go into the describe business. All right, so the instructions are there. It's actually fed that in. Let me make sure. Yep.

Key Settings: Orchestration and LLM Usage

Use generative AI, test response. Okay. Now, this is a really good thing to leave on, the orchestration. I'm finding that the results with the orchestration is much better.

You can go in and you can create a A traditional chatbot with all your topics and so on, but you've got the use of a large language model, you may as well just have it serve the information. You can get into a bit of a spin with chatbots and you might have experienced this before where you end up in this loop and it's really annoying.

You won't go back to use that chatbot ever again. There's some really good chatbots out there, I think Amazon is one of them, but they've got you know, they've got so much money to throw at it, it's not funny.

So, all right, so I've got the instructions there. That's wonderful.

Adding Knowledge Sources

And this is where I can add knowledge and I'm going to upload files. So I'm going to add knowledge and I'm going to, hang on, here we go. Move this over.

I've got some documents over here. Move this over. All right, and I'm going to have a non-disclosure agreement in there.

I don't know about you, but that's the last thing I want to spend my time reading. I don't want to read these kinds of documents.

All right, I'm going to have to try and upload, select file or browse. Desktop, HR, RAG documents. These are just some sample documents that I've thrown together and we're going to confidentiality agreement, PDF, yep.

I'm squinting at this. This is what happens when you look at a computer screen for about 40 years. I'm really quite struggling. I should have brought my glasses.

But anyway, I think I'm doing the right thing. I've got a number of eyes there. Sometimes this can take a few minutes and sometimes it happens really quickly.

What we're going to do is we're going to add another document as well. Where is my knowledge? There we go.

You can see that that document has been ingested and it's been in progress. Depending on the size of the document will depend on how long that will take, but we can go ahead and add more knowledge here.

So I'm going to browse again and I'm going to do, I think I did, did I do the employment contact? No, let's pop that in as well because people might want to ask about their hours or how many days they get on holidays and we can add to agent two.

All right. Great. And so you can see them in progress. This might take a little while.

Best Practices, Benefits, and Trade-offs

So whilst that is happening, we're going to go back and have a look at some of the benefits and the, hang on, where are you? And things that, reasons why you may not want to use it.

Hang on, there we go. All right, terrific.

All right. Hold on a minute. All right. There we go.

Getting the Best Results

How to get the best results from these sorts of things. Let's do that. That was my next slide.

You want to optimize your data, garbage in, garbage out. It still applies here.

You want to have documents which are unique. Don't have the information crossing over because then you might get a little bit confused, the results, and you may not serve up the document that you had intended.

You want to balance the retrieval to the generation and you want to make sure that your knowledge is always updated in line with your policies and procedures if you're doing HR or just say you get new machines in and you're doing an IT service desk, help desk for example, then you want to make sure that that information is being managed by someone.

And you want to test obviously and you want to have relevant resources too so that it's getting the most relevant information for you.

Benefits of RAG

Then we've got some benefits and disadvantages. Your benefits are improved accuracy.

You can combine real-time retrieval with generation to produce more precise and contextually correct answers. You reduce your hallucinations.

You might have heard of those. It's when the large language model, they're very sycophantic. They will try and please you. If they don't know what the answer is, they will make it up.

And then if you correct it, they'll go, oh, yeah, you're right. I'm sorry about that. I'll go back and do that again.

So this anchors the responses to factual content and content which is business specific for you. So you've got business context awareness there. So it's all about your company.

You might have 35 hour working hours or something like that. It's going to be so much more relevant for your staff and your users.

your smaller model efficiency. So you're talking about model efficiency can be, if you're using the latest and greatest model, you might be using an overkill. And so you could actually use a smaller, more efficient, faster model to serve up better content if you introduce it and you let it and teach it about your own documents it may actually perform better and it's going to be cheaper for you.

The token count is going to be cheaper and when I speak about tokens, it's kind of like words that you're sending backwards and forwards and how many words the person is putting in have to be transferred into numbers and how many words the LLM is returning to you, that again has to be transferred back. So you're getting charged for that and you've got network traffic as well and latency there.

You've got knowledge, as I mentioned earlier, that you're providing knowledge since the model cut off, and you've got transparency and traceability.

Disadvantages and Considerations

Disadvantages, very quickly, retrieval quality, dependency and increased system complexity, slower response time because it's got to go and get the information from those documents, but then it might be faster because you've got a smaller model.

It's about testing these sorts of things out, and you've got also potential for outdated or biased sources in there as well, but if you've got a content manager, you'll be able to ensure that that is all working fine.

Back to the Demo: Testing the Agent

This is still going. I'm just going to refresh on this, and hopefully it's just cached on the image. All right, we've got ready.

So that's telling me that the documents are ready to go and we can go now and ask it some questions. So what are my working hours? All right.

Now, this is the Copilot Studio testing suite, and you can deploy this to websites. I've deployed this to chatbot sites for professional companies, and it's actually using the content that's on their website to serve up documents, but you can add content in as well.

and it's telling me what it is and it's giving me little links there as well. Now as to why it's giving me two, I'm not terribly sure, I'd have to look into that but that could be resolved quite easily.

But you know that the documents are reliable, the information is reliable and it's not making that stuff up and it's telling me my working hours. If I went into those documents, I would be able to actually find that text in there.

Q&A: Document Sizes and Data Sources

So, are there any questions? Yes. How big is your document, or are you doing document directory searches?

Depends what it is. Not particularly.

I've used it with SharePoint locations. Now, you've got a limitation there. You've got about 200 documents you'd get around that.

You can put your documents together. Wouldn't recommend that because then it gets a little bit confused and you probably don't want to have those documents concatenated anyway.

The other thing you could do is deploy, and this is where you start to get a little bit more tech, is deploy something like Azure AI Search. You can even hook up your Dataverse if you've got CRM, the Microsoft tools. There's lots of other connectors there.

and you can get these to talk to other agents as well, but that's out of the scope. I'd love to come back and do another demo where we actually set up some topics and maybe an agent-to-agent interaction, but it'll probably take longer than the 15 minutes that I have today.

Does that answer your question? Yeah, so it's going to depend on the document size. Yeah.

Is there anyone else? Yes.

Q&A: APIs, Connectors, and Deployment Options

Does it provide any API, web service, to be used, I don't know, on the website as well? Not that I'm aware of.

Generally, what you get is you can go into it. I can't do it with this particular account, but you can actually go to a demo website, generally. It's not working. The last week, I've noticed, it's coming up with a JavaScript error, so I didn't want to show you that.

the central store of OpenAI. There is an option in OpenAI to upload your documents, but it's in terms of API. And just, I was curious that if GoPilot also provides a service Oh, to pull your information in, I wouldn't be surprised if that's possible, and I'd have to have a look into that to see.

But you do have a lot of connectors in there. So if we go in, and if we go to add knowledge, you've got those as standard. Now, there are the, I think, suggestions. Is that what it says? Yeah, see suggestions. So you've got suggestions there.

We could go to have a look in advance very quickly. So there's a few other things there, Confluence. Interesting, yeah, it'd be interesting to have a look at that. That's the Azure AI search that I was alluding to, so if you could get it into that, definitely, or if you could output the data into SharePoint, but then you're kind of dealing with the data a few times and you've got the overhead of getting all those pipelines set up.

So what we've done today is we've set up a RAG-based chatbot that we could go and ask it more questions as well. And you can deploy it and share it with your users. As I said, Microsoft 365, you would notice if you've got the full version with your company that there is the word agent over on one of the sidebars. Go and try that out because it's got studio underneath and it's really easy to use.

Okay, are there any more questions? Oh, there's loads, great, yep.

Q&A: Guardrails, Moderation, and Models

Are there any guardrails? Ah, now, fantastic, awesome, thank you for asking me. Yes, you've got 8,000 characters of guardrails in this section.

You've also got a conversational boosting area that you can change the moderation level from low, medium to high. When you're starting out, leave it at high, I would recommend. I would really highly recommend putting in your guardrails and that's essentially your instructions.

You can say it has a content filtering on it automatically from OpenAI in the background. You can also change the models. You can actually change the models and go and use other people's models as well.

There's a lot of stuff in here, but I did want to just show you the RAG part. This is really new, this extra functionality that we can see on screen right here.

Your guardrails are really important. Guardrails are essentially your extra information parts. For example, if I was setting up a HR help desk, I don't want it going and looking at my competitor's website for the HR information because it's all about internal. 1I would tell it, only use the information that I am giving you right here.

If they start swearing, generally the content moderation would kick in anyway, but someone might say, I'm having a mental health issue, and you may have a department or a caregiver to particularly filter or triage those people so they can reach out, or they might have online services for them to get extra help. You'll probably want to put that in.

You can do topics as well. I can speak about this for ages, but... I've actually probably just overshot it a bit.

But love to chat with you after if you've got any specific use cases.

Oh, and there's another question. Yeah. Yeah.

Q&A: How Much Is Too Much? Scaling and Vector Databases

How do you know how much is too much? You test. Okay. You test.

It is so tricky because I don't know how the large language model has been trained underneath. I've written it, I've deployed it.

The most successful that I think is actually a deployment where I've got it asking over about 2,000 plus, 2,000 to 3,000 code pieces. So it's actually, I've got descriptions about the code pieces, and you can go and ask it how many times, where is this table being touched in this code, and it will surface that, the PDFs. But that's actually using AI search. So once you get above that 200 mark, you really need to start thinking about a proper vector database, which, as your AI search is.

You're gonna be around afterwards as well, right? Absolutely, yeah. we can go for more. Yeah. Okay.

Wrap-up and Closing

So a round of applause for Mel.

Thank you. Thanks so much.