Introduction

Hi, thank you for coming here and thank you for having me. My name is Finca and I will talk a little bit about conversation design in the wake of LLMs. And if you wonder what conversation design is, that is a good question that I also often get. And yeah, I'd be very happy to tell you a little bit about that.

My Background and Path into Conversation Design

let me also mention my background first so that some of the stuff that i'm talking about maybe also get a little more clear i actually studied polish philology and then i went to be a regular polish teacher at the almost regular polish school but you know it doesn't pay well and i decided to just you know like switch careers uh and i ended up working because i always love grammar and a little weird i understand The thing is that if you want to teach grammar or do something with it, the only place you can really do it is, you know, at school or at university.

From Linguistics to Voice Assistants

And suddenly someone told me, hey, Samsung is looking for linguists. It's a tech company, obviously, but, you know, there is something for linguists to do there. And I went there and I spent five years working on Bixby, Samsung's speech assistant.

So I understand... some people don't know exactly what Bixby is. It is like Siri for Samsung devices. So for mobile, but also for fridges and TVs and whatnot, all the devices that they have.

Discovering NLP as a Bridge Between Tech and Language

And this is also where, you know, I got a little bit more of my technical knowledge. I managed to understand, like understand maybe that's too much of a word, but I dive into the field of NLP, of natural language processing, which is just this amazing, you point of connection between the technical part and the linguistic part. So it's awesome.

Entering the World of Voice Bots and Conversation Design

And I ended up working for a company that creates voice bots, mostly voice bots. But that's when I also got to know a little more about the whole field of conversation design.

What Conversation Design Is and Where It’s Used

So where conversation design is used mostly is in bots, in various bots. It could be a voice bot chat, but also in what we call now agents. And I understand that this division is also, you know, the names, the terminology here is, quite changing because when I say chatbot one might already think about something like ChatGPT so about an LLM whereas when I say chatbot I rather mean like a specific instance for a specific for example institution or a company that interacts with people and can do things with on behalf of the user so

Why Many Bots Feel Frustrating

Overall, creating a chatbot or a voice bot or an agent, this is something that an engineer can do. But, okay, I don't want this to sound like cocky, but like, can you... Think about the last time you talked to some kind of like an AI or not AI, even chatbot of like a company. So I don't mean an LLM.

I mean, like, I don't know, you called an institution, you called a company and you wanted something. And then there was a chatbot or a voice bot. And I... would like to assume that this was maybe not the best experience at least this is my experience that the experience of talking to them is not always good and now i can also like i see why it's not always a good experience and this is because it was probably done by like one engineer so there was a voitec just sitting and you know being told hey just can you

I don't know, we need on our webpage a chatbot, like a little thing, and someone can type in a question. And then poor Wojtek just did that, even though he's an IT admin. And then the chatbot is there on the page, but it's not necessarily working.

Designing for Goals, Not Just Chit‑Chat

And what conversation designers is like this whole field. that teaches you and shows and also creates principles and guidelines so that those chatbots, voicebots or agents are actually conversational. But when we say conversational, it's not just the fact that it's pleasant to talk to them.

Because when you think also from the linguistic perspective of what the goal of proper communication is, it's not always to have a pleasant talk, but also to actually accomplish something. So even from the linguistic perspective, 1A conversation is way more than just the exchange of information. There is usually some goal to be achieved.

Scope and Context in Real Conversations

We are also talking in some kind of a scope, so that what we want to accomplish is still within something. For example, if I talk to a chatbot or no, not even a chatbot, let's get a real person, right? If I talk to a real person in a bank, of course, I can start with, you know, like some small talk and pleasantries like, oh, the weather is nice.

But, you know, I will probably not talk about, I don't know, the implications of I don't know, slavery as depicted in Kunta Kinte, something like that. This is not the place to do that, right? So we can also assume that chatbots are built with the same in mind.

Why Teams Need Conversation Designers

And to do that, it is good to actually have the support from a conversation designer that knows the linguistic implications of a conversation, but also the technical part, right? And also it gives me a job. So that's, you know, that's also quite important for me. But actually having someone who knows those principles of conversation design will ensure that the product that you are creating is actually useful.

It's not just, you know, I don't know the English expression when something is just for decoration, right? Like just like a little flower for decoration. But, you know, it accomplishes something. It's useful.

So

Identifying Automation Opportunities

Overall, in different companies or institutions or whatever you have in terms of an entity, you can identify specific areas that can be automated and they can include, they can be both customer facing and internal. So imagine you have a big company where employees often forget or, I don't know, their password resets, right? You might want to have an AI agent.

Agents, Automation, and When They Make Sense

Now I will use the term agent to, or AI agent to show that it's also quite versatile and it is AI based. So you can have this agent in order to solve those issues. It is,

it makes sense i'm using also this word automation here because mostly we will be using those kind of solutions in order to make something faster and not do it manually i assume that you are here also people who are um from like the technical background or with a technical background So, you know, there is this joke of like spend six hours trying to automate something that you can do manually in six minutes. Been there, done that.

Natural Language Interfaces: Old Idea, New Power

But if you actually identify something, some kind of process, some kind of communication, some kind of exchange in your entity that can be automated, it can be made with a chatbot, voicebot or AI agent. But what is also quite important, when you use an AI agent or a chatbot or a voice bot, whether it's AI-based or not, you can use natural language.

And this has been happening now for quite some time. So for example, Siri has been there for ages. But of course, now with AI here in place, we got quite some more possibilities of developing those agents.

but I would like to also argue that it comes with a lot of issues and I would like to also present you this

Static Bots vs. LLM‑Based Agents

the main differences of having a chatbot, voicebot or AI agent solution that is more static and strict and also one that is based on LLMs. And I said it now an AI based or LLM because there is also a distinction that should be made whether something is fully LLM based or if it has different algorithms that do not necessarily have something to do with LLMs like ChatGPT, right?

Start with Use Cases and Predictable Flows

So imagine that you are a bank. And you know that there will be people calling you, for example, if you have a customer service, or they will write you because you have a chat box. And they want something specific. So your first thing that you have to do is actually identify those use cases.

Because if it's something very complicated, why would the AI agent do that? You should actually use your resources, your people that can solve those issues. And some of the stuff you can automate.

So once you identify those issues, you can kind of predict what the conversation about this issue will be like. For example, if you're a bank and someone calls you because the card got stolen, the conversation will probably go in a quite predictable way.

Intent Recognition and Handling Variability

So my task as a conversation designer is, first of all, design this conversation from the technical perspective as well. depending, of course, of the technology that I'm using. I need to understand what the initial sentence is, what the initial request from the user is, which usually we call intent recognition, so that the technical solution should recognize the intent of the user that's calling.

The thing is, though, that it can be uttered in many different ways. Someone can say, oh, my card got stolen. Someone can say, I lost my card. Someone might say, I don't know where my thingy to pay is, not even using the word card.

Deterministic NLP Pipelines and Control

So the static solution is to have NLP solutions, sorry, I will repeat myself, to identify the sentence, match it to some correct action, and then move forward with this action. Why it is nice? It is nice because if someone says something that's outside of the scope, we can still delegate it somewhere else, right? And the thing is that with those NLP technologies that have been in use for quite some time, this is quite predictable.

So what happens is you have full control over the whole conversation because you are designing it step by step. If you design a proper business case and you define it as well, then this can be all covered by the technology that you are using. And I will come back to it in a moment when I talk about LLMs.

This is also a place of security. We actually had a little exchange here with Jasper where we said, well, yeah, like you can, for example, dox a chat bot, right? I mean, this can happen with most of the technologies.

The thing is that it is slightly easier to actually take care of a static bot and make sure that things like that don't happen because it is less versatile. And I will say in a moment, What does it actually mean in contrast to LLMs?

So I wrote also full control and full control because it's really the usual conversation design as a technology choice gives you actually full control, not only of the conversation, but also on each step what is happening, whether you are making an API request in the background, what kind of data you are sending and so on.

Where LLMs Shine

If you use an LLM, the big advantage of it is that it's always generating something nice. And I have to say, it is, to me, it was quite surprising seeing LLMs the first time how, or generative AI to be precise, and see that what I've been doing before, which is painstakingly just write all the proper responses that the bot might give, that this is taken care of. This is already done by the LLM.

I don't have to worry about that. Another part though is that the, oh, and also error handling in, and what I mean by that is if something is out of scope, right? So if someone says something that is not predicted by me, the LLM will actually respond to it in a really nice way.

The static bots, they will usually say something like, I did not understand that, or I cannot help you, can I help you with something else? a LLM-based bot will probably handle your weird inquiry really gracefully. And even though it says, I cannot help you with that, it can also say something that is more meaningful, like, oh, I cannot help you exactly with talking about this film, but I can help you block your card if you need to, right?

Trade‑offs: Predictability and Reliability

The thing is, it's less predictable. How? If you employ an LLM to assess what someone has said, There are sometimes errors that you cannot reproduce.

And I am not using here the word hallucination because hallucination is just something that we assign a negative value to, right? Like the output of an LLM is just an output. So it is just us then. who look at this and say, this is false, right?

But an LLM is a prediction based technology, so it will always spit out something. And the thing is that we don't always can, we cannot always trace it back and say why.

I will give you an example. I worked on a voice bot that was based on an LLM and it was supposed to get a user input and then and then let me know whether the answer was positive or negative. So if someone said yes, the bot would say the answer is positive and it would go to a path in the conversation. If someone said no, it would go to a negative path.

On 10 responses that were a plain yes, one of them, the LLM, assigned as negative. And that's basically what you can do. There is nothing else than just blow a raspberry because that's what the LLM did.

And we don't usually have much possibilities of tweaking such a giant model unless you have actually a custom model built yourself. But if you're using some of those models that are just out there, like from OpenAI, this is not really something that you can trace back. This is something that with the static bot will not happen because it's more deterministic, right?

And the same goes for, because it's less predictable, there is also less reliability in the terms of the conversation is always nice, but you might say yes and the bot will understand no. Or you might, because when I say it, I can be also misheard. There might be something on the line.

But even when you write, and I know that this is extreme, but it actually happened also to me, right? So even though it seems like this technology is so advanced that it should surely understand that a yes is a positive answer, sometimes it just doesn't. And when you work in tech and you have something that you cannot really reproduce, that is really annoying because I can send it as a bug report or something, but there is not much that we can do.

Although there is a little solution that I will tell in a second.

Security, Ethics, and Data Protection

The thing is also there are ethical concerns. I will not dive too much into it.

I think here mostly about environmental costs and the fact that if you have a solution like a static bot that's more deterministic, you can also apply more security to the data that's being transferred. This is, of course, something that this risk can be mitigated.

Trust Layers and Mitigations

For example, I now work with a bot that's being created by using the Salesforce platform. It's called AgentForce and AgentForce has a like a layer an additional layer to everything that's happening under the hood it's called the einstein trust layer so what it does it actually prevents data from being sent straight to an llm because it's masking it before it even reaches the llm so there is a possibility like that you can still do it although then there is also the question how you how do you do it and if you a

that if you do it, if you analyze the input before sending it to an LLM with an LLM, then you kind of get into this loop of, OK, but then at which point are we not sending the data to an LLM just to analyze it? You have a little more control over it when you have a static bot that does not necessarily rely on LLMs to send out data, to create API requests and so on, but

The Conversational UX Advantage

When you do it, you often lose this conversational part of what makes actually the bots based on an LLM so much more fun to interact with. And this feeling of having a good conversation, it seems it is so important that uh those static bots are less and less used because it seems that no matter if there is an ai hype so to speak or not that the advantage of having generative mostly generative ai

to generate the responses to the user and an LLM also to analyze the conversation and when you give away the parts of the design to it so that it's more versatile can handle weird input better and so on that this is the price that you pay and it seems that this is now the reality like people are actually doing it this is what the current technology is so

A Pragmatic Hybrid Approach

1What I really like and what I usually use in my work is this hybrid approach. So what I do is I still prefer to have a controlled flow. So I design the conversation.

Design the Flow, Let LLMs Handle Style

I design the different points so that if someone says something positive, I go this way. If someone says something negative, this way. If they provide me an information, they go this way. So my job is to design this giant tree of this decision tree of a conversation.

But what I do is I also use LLMs in order to to generate those nice responses. This is what I like to analyze the user request so that I have this technology that, you know, is actually good at understanding the user. So I don't need to use NLU natural language understanding technology in order to do that.

Teaching LLMs Conversation Principles

But what I also do, and this is something that I gladly present in like three months, I try to explain to the LLM what conversational design principles are this year because This is something I said in three months because this is my master thesis that I'm working on now currently.

I want to see how an LLM is behaving if it's used for a voice bot when it has no guidance, no guidelines, nothing. I'm not giving any instructions on how a conversation should look like. So what it usually does is it looks very LLM-y, right?

It has all those emojis when it lists something. It says, that is a great question at the beginning. This is not natural.

So my thesis is to check what happens and how people are perceiving those bots when they are made with my design, when they are made with just LLM, no design at all, and when there is a mix of it, when I use an LLM for parts of it, but also keep the conversation design, and I also keep the principles.

Examples of Practical Design Guidelines

Those principles are, for example, I don't know, when it's a voice bot, so it's, you know, you hear it, the question should be at the end of the sentence because if not, if the bot will, you know, ask a question and then speak after it, someone might already start answering the question and then you would talk over the voice bot, right? So this is one of like a very easy,

Key Takeaways and When to Use Agents

uh examples of a conversation design principle something that chatbots and llms usually don't know so this is something that you still have to have in mind and the main take here of um of my presentation is basically to make you aware first of all that there is something like conversation design that it is good to have someone actually look at this from a conversational design perspective and by this I mean agents that you might be creating because even if the agent is not supposed to talk to a customer it is supposed to only, I don't know, do something for you, right? I guess that it will be also talked about later.

It is good to have someone look at the quality of the conversation so that you know that it will actually do what you want, especially if it's not something that's just for you, but it will be facing other people, other stakeholders, especially if those stakeholders are actually clients, right? Like your customers.

Do You Even Need a Bot?

And there is this one question that you always have to ask yourself, do I even need it? Because from my experience, there are many things that you can solve, not necessarily with automation, not necessarily with a bot, Although bots are nice. I mean, this is what I also do and I'm excited about that.

So, of course, the answer for me is quite often, yes, you might profit from it. That means from actually using an AI agent, a bot, chatbot, voicebot, whatever that might be.

Define the Use Case and Design Holistically

the thing is we need to also define a proper use case for it and this is something that conversation designers will also help you do to properly plan and design not only the conversation but everything that comes with it and around this and That's it.

Closing and Contact

That's me

Just as a little treat, I will also say out loud my email which is finka at grammaqueen.me and you'll see later why I said it out loud.