Hi everyone, thank you for the introduction.

Introduction

My name is Eva and I'm going to tell you about my startup Pandria that I run with my co-founder Omar who isn't here today. So indeed our goal is to help people leaders in organizations build a better growth culture. I'll tell you all about what we mean by that. In the world of HR, this makes sense.

So what I'll talk you through today, so one of the prompts for the talks here was what I've learned building with LLMs. So I'll hope to kind of talk you through what we've learned over the past, say, three months or so building a conversational product.

So firstly, of course, the problem we solve and the actual solution we are building. Then our kind of design process, what we've learned through that, how we work together as a team, indeed wearing the multiple hats.

I feel like we're very synergistic tonight. And then, of course, look into the future a bit. And I'll also really look forward to your thoughts on that as well.

So excited to jump in now.

Identifying the Problem

The problem we solve is performance management usually sucks. I don't know if you guys have much experience working in larger organizations, but what usually happens is the company grinds to a halt about two times a year because people have to fill in irrelevant forms about each other that they kind of fill out with whatever they happen to remember. And what often is the result is people have these awkward conversations that are, as I say, irrelevant at best, but often biased and demotivating at worst.

I see some people nodding, some very young people thinking, why would you ever go through that? This is a thing at work, I promise. Now, as I say, it's a common thing as well.

So actually less than 20% of managers in organizations is content with the performance review process. in their organization, and under a third of evaluations feel fair to teams, exactly because of the biases I mentioned. So people just fill out what they remember, it's often irrelevant, and people nominate people to give feedback based just on who they work with, right, who their friends are. So this perpetuates a lot of inequity in the workplace as well.

Now, that's not just an annoyance. It's actually a cost driver. As you can imagine, talent is expensive. And unfair biased performance reviews cause a lot of decrease of well-being. They even cause burnout. And of course, they cause people to leave because they are demotivated.

1On average, that costs about $3,500 just in wasted time alone, let alone the kind of decrease in performance and motivation. But on top of that, a disengaged employee costs their employer about 18% of their wages. So this is an expensive problem.

The Cost of Ineffective Performance Management

And what's at the root of this, and this is exactly what we are trying to solve, is that people don't talk about growth and performance throughout the year. If people just gave each other frequent feedback, this wouldn't be a problem. There wouldn't be a surprise at the end of the year, and there wouldn't be a need for this peak in kind of activity in your annoying HR tool.

So despite this tooling being in place, there are buttons to push. Under 19% of employees say that they actually receive the timely feedback that they would want. And you can probably guess why this is, but of course there's a lot of research into that as well.

People simply find it awkward, right? It's difficult to approach someone and tell them about their performance. It's also really annoying to switch when you're working against a deadline to go into some tool and suddenly enter your notes about someone else's kind of behavior. People often lack the skills to have constructive conversations around this, and there's also not really any accountability built into the system about having those conversations and then following up.

So those are kind of the friction points that we saw through personal as well as kind of more advisory experience and that we aim to tackle.

So I was going to show you this in Slack, but you probably have all seen Slack and you can't read the letters on the screen anyway, so I'm just going to show you here.

Our Solution: Pandria

We aim to make feedback an easy habit. So how do we remove those frictions? As you can see, we actually take the initiative. So the first hurdle is asking so that your employees don't have to.

So instead of filling out this form, what about, say, once a week or twice a week, you're approached by your coach, Pandria, who says, hey, Ling, you just had your product kick off with Mike. I see your schedule is clear. Do you have a moment for some feedback? And you say, sure. You have a moment, right? Okay, great.

Now Mike is working on his goal setting skills for the team. How do you think that went this afternoon? And you enter into this conversation.

And we do that right in Slack or MS Teams or even Google Chat, wherever people are chatting at work. We also aim to be maximally relevant and personalized. So we know who to ask, when to ask and how to ask for feedback at your individual level.

Secondly, We add in this micro-coaching to actually build that habit, right? So we help you to formulate feedback well, to build your skills, and we actually create accountability.

So let's say we nudge you to have a conversation with your manager. Afterwards, we say, how did it go? Before it, we say, do you want to practice? Let me pretend to be your manager. I know how he talks. I know he's not very confrontational. So let's try and nut this out. How can you be better?

And then finally, of course, we help save time at the end when formal evaluation time comes around. So connecting to your HR information system, we can summarize everything we know so we can draft up your performance review. Of course, here a human is always in the loop. This is sensitive information. As well, we provide a lot of insights exactly around that inclusion and equity, as well as kind of engagement that I mentioned before, that wasn't really accessible at that level of granularity before to HR and people leaders. So that's the solution, we think anyway.

Technical Aspects and Integrations

So how does that work under the hood, right? Some of you may have a pretty good grasp of this, but I'm going to just talk you through it quickly.

So how do we get to the right time and the right person? We need a bunch of integrations. 1We connect to your calendar as the main trigger point for relevant feedback.

In the future, we can see that there might be other ones like, for example, your GitHub when you're in a technical team and someone basically releases, I don't know the right word, a bunch of code, right? There's a pull request or something like that.

We connect to your HRAS so we understand your org chart as well. So we know who is in the team with whom basically.

And then we use a proprietary machine learning algorithm to continuously learn and improve what the best time is to actually reach out.

What about the right topic? So as you can guess, we use an enhanced RAG system to retrieve relevant skills and behaviors to ask about. We do that from information that the customer supplies to us. So most larger organizations have a pretty mature competency framework related to every role with expectations for each individual.

But we also, of course, have this growing feedback history. So we know, look, Mike, he keeps doing this or that. Do we actually see growth in that? And we can think, reason about who should we ask about that.

And then finally, the right tone. So how do we ensure that we reach out in a way that resonates with you? we feed our bot with kind of best practice frameworks to give feedback. And I'll talk more about how we actually pour that into a conversational shape, right, step by step.

We also have the system reason in the background around what is your personal communication preference. So as I say, we know that Mike isn't very confrontational. That's something the bot can build up knowledge about in the background as well. And then finally, also, we can train on company tone. But this is all definitely not in the MVP that we are rolling out with pilot customers right now, but this is certainly on our roadmap.

The Art of Conversation Design

So what I want to talk you guys through in terms of learning to build with LLMs is actually how do you build a good product when you have no interface, right? When all you have is conversations.

Of course, very important is monitoring and safety. Luckily, we've covered that topic, so I don't have to go into it.

But what is best practice UX when it comes to conversation? And there's a whole burgeoning area of conversation design. Conversation design has been around for a long time.

Who of you is kind of familiar with this field? Great. Some of you are. That's fantastic.

But of course, that field has really taken flight with the current wave that we're seeing now. I have to say, I am very much learning by doing, and I have had a lot of great input from someone who has been doing this for many years. Her name is Maaike Groenewegen, and she's been advising us on this as well. So credit where credit's due. And she will probably come for me because I'm getting things wrong, but there you go.

So what is conversation design? It's all about, as is with normal kind of UX design, about centering your conversation around the user. So basically that means talking like a human, right? Using natural, easy, and intuitive conversations.

The second is context and flow. So how do you ensure that what you're saying makes sense in the context? And there's a logical progression that allows you to actually attain your goal with the conversation. And finally, of course, always improving by learning with input from your users.

So I'm gonna tell you a bit about how we have designed our conversations at Pandria. As I say, we're three months in, so we're still very much in the throes of it and improving this process as we go.

Crafting Pandria's Personality

Step one, who is Pandria, right? You're basically building a personality.

Step two, Designing the happy flow. What does Pandria sound like when things go according to plan, right?

And finally, building in logical steps in a chain so that you can accommodate for all the kind of quirky things that humans do, all the edge cases, as well as, of course, making sure that all the logical kind of connections to your architecture makes sense.

So firstly, the personality. I'll go through this briefly. What are the goals and values of your bot? So for us, that would be trustworthy, for example.

Then how personified is your bot? If you're an ordering bot at McDonald's, you don't want that bot to say, hey, man, how's your day? I'm so happy to help you. Whereas if you are actually a coach, that might make sense, right?

You want to build up a trust-based relationship. So what are the character traits that you're going for? You know, for us, what's really important is that our body is disarming. It allows people to open up.

And then fourth is tone. This is more of a linguistical issue. So is your boat formal or casual? Do they talk like an expert or a novice? Like, are they there figuring out with you or do they talk from a position or authority? Are they very warm or are they more cool? Very practically, do they use a lot of exclamation points, for example? This is also very geographically sensitive.

And then you design your personality around key behaviors. So how do you keep the conversation on track? When someone veers off, are you stern or do you make a joke and say, okay, let's get back to it? And of course, how does Padraig behave when someone is abusive? Do they shut down or do they refer you to a human? All these things are part of someone's personality.

Then we designed the happy flow. Now this is really a kind of the product design hat coming in, right?

Designing User-Centric Conversations

Thank you. So we did this basically by just simply creating a mockup in Figma and talking to a bunch of end users about how it felt to use that, right? So we basically wrote out a particular conversation and tested it.

And then finally, of course, designing the actual logic and edge cases. Now this, you cannot see what this says, but what I can tell you is we basically designed swimming lanes saying these are the actual kind of prompts. These are the logical steps, the evaluations and the connections that need to take place. This is what happens, say, in our dashboard to data. And these are things that are related to various kind of user personas or human roles. So this is how we then port that into a formal design of a conversation.

Building the Conversational Product

So how did we actually build this then? Great, we have a huge Miro board with all these conversations. Excellent.

Now obviously, there are tools for this. We are not reinventing the wheel. So we started with off-the-shelf tools.

One of the ones that we were really excited about in the beginning was called Voice Flow. I can highly recommend this tool. It even allows you to build in steps like RAG, your own knowledge base, and this kind of logical chaining.

It allows you to preview things. However, we quickly realized that it wasn't going to serve our purpose.

We of course want enterprise grade security. We had these complex iterative steps in mind and we had a lot of integrations to take into account as I've shown you. So we built something very simple for ourselves.

So how we now build together is, My co-founder and tech lead has basically built a little tool that I use to generate kind of conversational flows myself. So I insert the steps, the evaluators and all that. I iterate on that and actually play with it in that particular playground that he has built.

And then when that is done, I export a JSON, a simple file that is then replicated and imported into our production environment. And this allows us within our team to wear these various hats and build very quickly without kind of breaking and losing time in our actual production environments.

So of course, and I would very happily show you what that looks like. So come find me and I'll be there to show you.

Looking Ahead and Inviting Collaboration

So what do we want to do? We obviously want to get this in the hands of users.

So if you feel like this could be something that might be interesting for your small team or organization, I'm very keen to hear about it.

We want to... So we feel like this certainly adds value for HR people. Now we have to prove that human users actually enjoy using it, right?

So we need to optimize our algorithms, obviously iterate on the prompts and the flows, and of course, prioritize our features. That is what's next on the roll for us.

Conclusion

Thank you.