Introduction

So hello everyone, and welcome and thank you for coming.

So I'm Anne, I'm the CFO here on the EDGE team here at Qualtrics, which as Stephen said, is like an in -house market research agency.

EDGE does everything from full -service projects to panel -only projects, so we do survey builds, design, fieldwork, everything, work very collaboratively with clients.

Today’s Focus: Synthetic Data at Qualtrics EDGE

And today we're going to talk to you about synthetic data.

AI Capabilities and Tools at Qualtrics

So before we do that, I just want to give a quick little highlight of kind of AI and Qualtrics so aside from the day -to -day work that we do and creating gems which MindStone helped us with and prompts and things like that that have like really improved our processes we also have a researcher

agent coming we've insights Explorer Qualtrics assist common summaries ORM AI generated ticket replies conversational feedback and response clarity so see these are all links so if you are interested in any of them you can just Google them followed by Qualtrics and a page will come up and all the information will be there.

Defining Synthetic Data (EDGE Audiences)

But today we really want to focus on synthetic data which what we call internally is edge audiences.

How EDGE Audiences Work

So our inputs to this are of course LLMs, so that's open AI, perplexity, then we have the XM survey data, so anonymized rows of Qualtrics survey data, and then as well as that there's additional training data, so that's purchase training data from other players in the research space so edge audiences kind of today tomorrow where we're at

Inputs and Training Data

Current and Low-Risk Applications

what we're looking at first of all this column on the very far left there low risk and no risk applications so this is for clients who are a bit unsure don't really trust synthetic data and they're kind of like what is it about you know they just want to test it out see how they feel about it which is completely fair I think it's new to everyone everyone's a little bit scared a little

bit skeptical so what they can do is say if you build your survey and you can use synthetic data to kind of test it, see how it'll flow, see how the results will come out, see how the data looks and then you can go okay yeah I like how that looks and then you can field that with say human responses and you can use it

for example if you have a long list of ideas from a lot of stakeholders you're like how am I going to get this down I need to just like whittle it down so it's like digestible in a survey for example you can actually use synthetic data to bring that down and it's data based results so the list you will get will be backed by you know the research in the background of it rather than people just kind of sitting in a meeting for an hour and picking and hoping that

it works and hoping that they're picking the best ones and another thing that we do is pilots so some people are actually using parallel testing so they will send out their survey to human panel and then they will also run a synthetic data and and then they can combine them at the end and they can look at them,

they can compare them, see where the pitfalls are maybe on the human sample, the pitfalls are in synthetic, and equally where the benefits are in the synthetic and whether the benefits are in the human sample.

Building Toward Advanced Use Cases

And then finally, building towards, so this is kind of where we're going in the future, so this might be, for example, so boost existing human responses.

That'll be if you're looking at a niche target audience and you can't find the human sample for that, you can use this trained data so to sort of boost that sample and it's based on real people so then it just kind of like fills in gaps that might be hard to reach normally and so this is

Roadmap and What’s Next

our audience roadmap so today we have us gen pop we are very close to having international so that's kind of English speaking UK Australia Canada and also us b2b synthetic sample and then looking forward we want to amplify we want

synthetic personas so being able to have a conversation say with someone who it's synthetic but it's someone who would have taken the survey and has the views

of a respondent then we have user trained adapter and survey extent I'm

Client Use Case: Dollar Shave Club

gonna present with you like a use case that we did with one of our clients dollar shave club so for those who are not aware because they are at the end of the day like a US based company right now they are a personal health brand

like they do subscription boxes where they will send shavers and razor blades and any kind of like accessories for that on a monthly basis so you pay subscription fee and then you get every month you get like one of these boxes

Business Context and Objectives

and this is a very good model in that like it's a really disruption in terms of like how the category normally works and it's very different so a lot of this is based on online and Dollar Shaped Club wasn't really that aware of what

kind of segments or what kind of customers they're speaking to so what they wanted to do was do a segmentation analysis and we thought well this is actually a great way to test audiences as well because we can get like a view

Why Test Human vs. Synthetic

on the one hand of how the human data lines up with the synthetic data but then also we can explore how the data relates to each other how synthetic responses and human responses will coexist with one another and then maybe

Project Workflow and Speed Advantages

even more interestingly we can deep dive and do more advanced analytics like we could do a full segmentation and factor analysis to see can we actually replicate using synthetic data what we see in human behavior so this is what a normal project lifeline kind of looks like for

us we do a product kickoff we design we build a survey and then we do a sample collection the big difference that you will see between a normal survey that has to feel for an extended period of time is that synthetic responses on Qualtrics that's instantaneous you will

just press a button you give in your quotas what you want to feel for and then once you have it it's nearly instant anyone who does survey research can tell you this saves you so much time because sometimes you're doing research

for very niche audiences you're interested in b2b and it can take weeks to collect what those responses so I'm gonna focus for the rest of this on the

Findings: Human vs. Synthetic Alignment

analysis site where we're comparing, starting off with humans versus synthetic. So here's an example of that and what we actually found for the majority of our survey is that there's remarkable alignment between the humans and the synthetic responses.

For instance, we'll ask them on an agreement scale, I would rather spend money on experiences over things and we'll find that there's near perfect alignment in those responses in terms of how the scales are used, in terms of who responds what.

But yeah like we see that even further going into like category questions into more specific questions and even when we find that like the questions don't perfectly align usually the key takeaways are there.

Where They Align Strongly

But to just then jump into a bit more nuance if you go to the next slide there. So oh sorry previous one yeah sorry yeah so we're gonna jump into a

Where They Diverge and Why

a little bit more nuance because the synthetic model doesn't always perfectly align and so if we show the first example there this is a great example of that so here we see that the model currently as it is doesn't quite perform as well on abstract or subjective questions so a question

like I like to have fun with my shaving that is really abstract and in fairness if I ask humans in this audience like oh what does that fun with your shaving look like for you that's probably going to be fairly different for everyone so what we see is that the model kind of replicates that

and it's those nuances it can't quite catch it does much better when it's objective facts when it's like frequencies when it's future intent but subjective things and morality it's not quite there yet hopefully with the future iterations like the model is fast developing and every two weeks we get a new iteration of it, we will see some improvements on that.

Category Behavior Nuances

Another example that I found was quite interesting. So here the model shows like nuanced differences in personal care products. So we're asking people, where do you buy these products from?

And when we look at the human data, humans will say they go to Target, to Walmart, to Sephora. But what you see, there's a pretty big mismatch there. if you go to the synthetic data where they say more for CVS.

So this kind of highlights that synthetic can be different in some unexpected cases, but the interesting way to think about this and the way that we envision this as well

is that the synthetic data picks up on something slightly different, namely the objective reality in this case, because when you look at the actual shares

and where people go shopping in terms of the raw numbers and the revenue, you, CVS is much bigger than Sephora.

So it seems like in this case, people are just kind of showing the reality they wish they live in, because Sephora might just be more prestigious than, say, walking into a CVS and buying something from a pharmacy.

From Alignment to Segmentation

So this brings me to the next portion

of this, which is the actual segmentation.

Approach to Factor Analysis

It's something that I'm really passionate about, which is data analytics.

So I'm so sorry. I'll try to make it relatively clean and simple.

not go into too much detail but so when you think about a segmentation analysis what you're doing is you're taking this really big survey you have demographic information you have behavioral data and you have these wide -ranging dimensions and then the question is well what are the themes that we can raise up among them and how can we reduce that to like the actual exemplars

so one way to do that is using factor analysis and that will look at the correlations between the different items and that will give you factors that are the archetypes of the different survey questions based on those relationships and the real

question is as well with respect to synthetic well do we arrive at the same

Overlap of Themes and Added Richness

themes and the answer is very straightforward yes we get a near -perfect overlap between the factor structure between the humans and the synthetic responses so when we look at these perceptions of shaving they're nearly nearly identical in terms of experience and necessity, the same values are seen again.

In fact, sometimes synthetic is even better than human. I'll just show this here.

So these are the factors that we see for both of them, and this is the human side. What we see is that there's even more items and themes, items that load, and the light ones there are the synthetic responses, and they're unique to the synthetic responses. So it seems that while both audience produce the kind of the same theme, the synthetic is able to capture more of that richness.

Why Synthetic Can Capture More Variability

And that actually kind of makes sense in a logical fashion when you think about the survey taking experience. Oftentimes when people take surveys, they tend to have like a lot of cognitive biases. They try to reduce their own cognitive load. And a lot of the time they get anchored.

Like if I get like 10 items with a seven point scale, at some point I'm just going to go six, seven, seven, seven, seven, six, seven, and there's not really that much space for variability.

So what's really good with the audiences and with the synthetic responses is that we're seeing that it takes the question at the questions level. And so when you aggregate it, it might appear similar to a human, but on an individual level, there is much more variability and much more richness, and the model can catch that.

Clustering Results and Implications

So that brings me to the kind of conclusion from this. When we take those factors, we can then translate those back into clusters and descriptives. So for that, we actually took together the synthetic responses and the human responses.

And what we see is that we get four clusters in total. On this side, you're more likely to be human respondent. On this side, you're more likely to be synthetic respondent.

Combined Clusters: Human vs. Synthetic Tendencies

respondent and what is interesting is had we done this exercise with the data separated these first three factors will be the only thing we detect and if we just use the synthetic responses these ones are the only ones that really come

Key Takeaway: Synthetic as an Amplifier

up so 1one way to think about synthetic responses and perhaps the best conclusion to this story is that it really amplifies the differences existing differences in these data sets and it also gives you more of a leeway to see smaller segments pop up and that's kind of the story that we brought to dollar shave club

Conclusion

and who's now going to use this for their own marketing so yeah thank you for your attention