Introduction

So a very quick introduction to me. I'm a nerd and I spent a long time working at IBM and I'm now the CTO for Squadify and my main focus in that role has been that process of building AI into the platform.

So I thought I'll start off my little explanation about Squadify with a question. How many people in here are members of a team? Almost everyone.

Fantastic. So this is brilliant because this is for you.

Importance of Team Performance

Team performance is really, really important in every organisation. And McKinsey, so this is a technical talk, I'm starting off by mentioning McKinsey. But McKinsey released a report last month in which they basically said that team performance is actually more important to organisations than individual performance. But if you look at what happens, I see quite often that individual performance is where the sort of measurement and focus happens in a lot of organisations.

And the problem is that current solutions such as executive coaching and leadership development or engagement surveys, anyone filled in an engagement survey? One or two people. They're not all that effective at getting the best out of teams.

And that's what led Squadify to be formed.

Introduction to Squadify

And Squadify, as it says here, is a SaaS platform. And it's a services solution that rapidly and continuously builds engagement and performance through data-driven coaching and AI-led development. So that's the sort of top line.

And we work by we survey your team and get data. And then we can use that to clearly pinpoint the areas where teams are doing really well and other areas where they need a bit more work doing.

And in addition to that, Spotify looks at key dynamics from alignment through to psychological safety. And each metric shows you where you are, where you need to be, and how you compare to other teams. And you can drill down into more detail.

Benefits of Squadify

And we know this works. So using Squadify data, a lot of teams from startups and enterprises and education to health care, all these different sectors have made, in some cases, 30% gains in their performance.

But you knew there must be a but coming. While Squadify was able to give really clear data to teams, turning that into actionable points requires more expertise.

And a senior and experienced coach can look at that data and the way it's changed over time for that team, and they can really help the team. They really know what to do with that to help the team.

But not all teams can afford or have access to that kind of level of coaching. And that was the driver, the really powerful driver, because we needed to make that accessible to every team.

Incorporating AI into Squadify

Enter AI from stage left field.

I wrote that because when I was thinking about this talk, I took myself back to March 2023, which is when we started thinking about this. ChatGPT had been out for four months. And OpenAI, in the month of March 23, launched both GPT-4 and their API.

And anyone who used GPT-4 back then, you knew immediately that is a massive change over what was possible with GPT-3.5. And so we sat down and did a lot of thinking. And we sort of came up with the usual sort of how might we statement. How might we use AI to support a team in making their squadified data actionable? And that was where we set out from.

And we sat down and we played with GPT-4 just in chat GPT, assessed can it do coaching? And it was kind of primitive. and very slow, but sort of miraculous at the same time. And this was when we kind of thought, well, we can do this.

And we had chat GPT. We had it in front of us. It was too slow, and it was unreliable. Do you remember? It still does it sometimes, but it used to be, oh, there are too many people using it. Please come back later. And the API, when that arrived, was likewise a little bit slow and a little bit unreliable. And there were no other credible options at the time for doing this.

Initial Challenges and Solutions

So to cut a long story short, we decided to start by using AI to enhance the dashboards that we already had for teams in order for them to see their data. And the reason we decided that was we can run that asynchronously in batch. So what that meant was the fact that it was unreliable and slow didn't matter. We ran it, stored it in a database, and then they could just look at it when they arrived at their dashboard.

brings me to the the second challenge which is um we we realize that we've got to tell this ai a lot of stuff that we had kind of internalized and forgotten that we knew and for example we have a lot of knowledge and experience of what the scores mean in squadify but we didn't initially spot oh the ai needs help to know what the numbers mean and we had um was one team that was in a particularly good situation and the ai was giving us a strong telling off about its uh its least good um attribute but actually that least good attribute was really very good and the ai should have been positive about it and so we developed quite a lot of boilerplate that went in our prompt so this is an example uh explaining the scores and of course we've got a lot of data from from teams now so we're able to calibrate the the explanation so that we've got the various quintiles of what's good and bad.

And you can see that less than 3.2 is really quite bad. So that's not what you might intuitively believe from something that goes from 1 to 5.

We made a load of other prompt boilerplate for the summarisation stage, particularly we said what tone we wanted. We said we wanted British English. And we... We told the AI to refer to the team as you, because it kept inviting itself to join the team and saying, we need to do this, we need to do that. Well, no, GPT-4, that's not right.

And we also found that it was advising teams to be more happy. The other one, the good one, was get more resources. And of course, most people who work in teams find that getting more resources is not quite as simple as just doing it and being more happy. So we actually had to sort of tell it, don't give advice to that effect.

But after we did that, we found that we're starting to get to a really quite a decent explanation of the data that was user-friendly.

So there's that clarity ring and here's the beginning of the paragraph. One thing we did was we had the AI create these teasers at the top because one of our coaches gave us feedback that nobody's going to read all that. So we thought, well, we'll try and tempt them with that.

But we had something that can get useful commentary for a team on their dynamics and 3Cs and all the rest of it. But And of course, this comes from 20 data points. We haven't put in 4.2 and said, write that text. It is more nuanced than that. But a coach will be helping them, taking this knowledge and insight into how the team is and making it actionable.

Enhanced AI Capabilities

And that's challenge number three, was how can we go from this useful, very nice explanation in the report to actionable advice that would equal that, that would be as high quality as something that a reasonably good coach would do. And when we started with this, we basically found it made a lot more sense to actually ask for things like team focus areas based on what the explanation was of the three Cs and the dynamics and all of that. Because that meant that it was much more likely to be consistent, so the advice would be to fix something that the other advice had said was a problem.

And from these... From the focus areas, we can then generate coaching questions and content recommendations, which was based on Squadify's excellent We Not Me podcast. This is a bit of a plug. I've been on it, so it's obviously very good. But we... We used a retrieval augmented generation pattern for the podcast episode, so we chopped them up into bits. And the player, we then were able to include a player that when you press play, it started from the right place in the podcast to get the exact advice that you needed.

So, we're now extending this to be able to recommend more varied resources. So, our enterprise clients have got learning management systems, stuff like that. So, we're wanting to integrate with those so that it can recommend learning content available there.

We also had AI produce a three-word summary of the team status, which was kind of hilarious because lots of teams were aspiring and quite a lot of resilient, and resource strained, which is two words. So we solved that by giving it a list of words it was allowed to use and saying pick the three of these that best describe this aspects of this team.

Testing and Validation

So this led on to a very important topic for us. And during this process, I worked with Squadify's chief data officer, Julia Owen, who has a very deep understanding. And she really had a great in-depth knowledge of what the numbers were that explained the team's data. So iterating between us, we kind of got it up to a level where we thought the words explained the numbers.

But then we went into a testing process with six of Squadify's top coaches. So I realise there's 10 people there, but, you know... artistic license and we started off our initial score was 5.2 out of 10 and we had to work and some of those were scores of 3 out of 10 so we had to iterate a lot and then some way through we knew we were making progress because one of our top coaches was about to dispute something and then realized the ai had actually spotted something that she had missed so that was that was pretty satisfying and then in the last round of testing we got the score up to 8.3 out of 10.

Production and Monitoring

1And at that point, we were ready to go into production, which is obviously really simple and easy, or not. They did create a load of concerns. And of course, a lot of them are the same concerns that you have in any application going into production. But we had some particular AI ones that we dealt with.

So we used something called LangSmith. So the LangChain developers built this. for people to use to sort of monitor what was going on inside the AI interaction.

So you can see this interaction had a lot of steps on the left. You can also see it cost just under half a cent. So it kept track of the cost, which was very helpful as well.

This makes it very easy to filter and examine results. And if the AI did something wacky, which it hasn't, but if it did, we would be able to go into here and find out exactly what happened. So really, really, really useful.

Also, we wanted end user feedback. So we put the usual thumbs up, thumbs down, and also a comment box. And we decided to put that into the LangSmith as well.

And it would tie the feedback to the exact interaction that generated it. And that means, again, we can go in there and filter on the feedback. And then if we need to iterate prompts and things like that, we're able to do it easily.

And then the final thing we did with LangSmith was we put our prompts into it. And this was a game changer for us because we started off, as a lot of people probably do, with our prompts in the code, albeit in their own file, easy to sort of manage. But of course, we found that when we wanted to change the prompt, we have to go through a release cycle, which disturbed the kind of flow of our offshore dev team.

because they do lots of things, not just this. And it added a delay into the process. And so we didn't want to do that.

Putting the prompts in here means we've got this versioned and testable. And there's a lot of benefits from doing this, where you can easily test, try new versions, and then push them into production without having to go back into the code to do it. So we really love that.

Current Status and Future Plans

Where are we now?

Well, every team gets expert analysis of their report and a set of actionable recommendations and with recommended coaching questions to consider resources to help them learn how to make their teamwork better. And we've got a system from a technical point of view, flexible, maintainable, and powerful.

So have we finished? Hell no. We've got loads and loads of things to do.

We've got a load of these kind of upcoming plans, time-based analysis. So rather than just a point in time, we can have the AI comment on... on the team's progress through time, guided journeys where the AI can handhold teams through the whole process, and the LMS integration I talked to earlier, and the chatbot, which is where we had started. So that will come back, but a chatbot that's informed about where the team is, knows the circumstances, and can be directly helpful to team leaders and team members who want to use it.

So we still have a ton of work to do, and we are currently talking to investors to help us do it faster. We're quite a small team at the moment. But yeah, I'm delighted to say that we're on this journey. We're staying on it.

Conclusion

It's been hair-raising at times. But thank you very much for your patience.

Listen to me. If you've got any questions about anything I've talked about, like I said, come and find me at the end.

If you want to know more about Squadify, then Dan Hammond, who's a co-founder of it, is sat there. He would love to talk to you about that. And he loves to talk about teams. So beware, if you get him started, it may take a while.

But all good. So thank you very much.