So hello everyone, my name is Regina Duarte, I'm a PhD student in Instituto Superior Técnico, it's near here. A bit of my background, I'm a technical student, so my background is on applied mathematics and engineering and then I move on to AI, let's say, but not really just technical AI,

more on the human AI interaction part. So in the interaction about humans and how we cooperate with the AI tools.

So I think after these two interesting talks, let's have a bit of a perspective shift on how we see AI and how we cooperate with this. So I want to bring the impacts of ASS decision making and why we should care, which is something that I am currently working.

So let's start with something that you already know, which is, you know, all the that we are working on ChatGPT. I think this is not that updated, but DALI 3 and all the image generated models, even the copilot.

You know, this kind of, and even the deep research that we just heard of, like all the tools and they are tools that are great to generate a lot of stuff that supposedly became more, we became more productive. We spend less time in, you know, working and thinking as our previous speaker just said.

but there is kind of several caveats right and maybe i'm here on the a bit of a negative side which kind of you know this was a is all something already talked about a lot which is a lot of bias and ai generation generation topics like machine bias and some misclassifications, misgenerations and really several people are like really wondering if this is really a benefit you know so here when working with AI we have two options

we have first option which is the AI tools just be automated tools and replace humans and this is something that sometimes is a bit concerning like okay should AI replace workers in the workplace? Should we just go to the employment?

But then the second option, and I think most of us are currently going to that one, is just not replacing, but really having this collaboration, right? So we just work more productive, harder, more intelligently, and so on.

So OK, let's focus on that part. Let's focus on the part of not replacing but collaborating.

Here, today I will try to, let's say, prove that this might not be the case for always. So this assumption that might be general common sense might not always be true.

introduce three examples that maybe we can think of our mind while I'll go through the presentation. And in this presentation, I will just specify AI-assisted decision making. So for sure, human-AI collaboration is a spectrum way larger.

But the idea here is just to focus on AI-assisted decision making. And I will give three examples.

First, clinical diagnosis with AI predictions. So for example, your doctor has screening tools and AI tools that can diagnose, for example, breast cancer or skin cancer or other types of diseases.

This is something that sometimes is already being deployed. So think of these examples. Your doctor is the final decision, final decider, but has access to these tools.

1Another example is credit risk assessment, where a risk analyst in the banking sector has to assess the risk of a client and has an AI tool recommending, oh, reject or accept these clients. And yet the final decider is the is the risk analyst and then even a maybe not so dummy example is fake news detection imagine you are in your home passing through several news sites and then you have a AI chat bot or a bot saying oh I think this is a fake news and you have to you know by yourself decide do I think this is a fake news yes or not

Okay, having this in mind, what are the three challenges in human AI-assisted decision making? So the first is AI should complement human's abilities. This seems trivial, but it's not really all the time the case.

Imagine the medical expertise. We know that, for example, skin screeners and AI tools that predict skin cancer, they are great, like 90% of cures in 95, which are way higher than medicals. But maybe the medicals that are most expert are better than the AI tools.

And these kind of things that is the AI really complementing the human abilities are not just checks. What is checked is the AI abilities, and that's it. The complementarity is not given. So this is a challenge.

Second challenge. humans should understand AI capabilities. This is also not guaranteed. Do we know the boundaries, the error boundaries of the AI?

So in this case, deep research, we know that there is even 1% of hallucination. But do we know the examples that these hallucinations occur? I don't think so. And we should just to have a great cooperation in these settings.

And then effective interaction between AI and humans. And what I mean by interaction is just that it's not just AI recommendation. We need explanations of the recommendations. We need a two-way interactive environment where the humans can understand why the AI is giving these recommendations.

So these are the three challenges. And I argue that these are really hard to combine and really hard to, in several settings, exist. And I say, don't exist yet.

But what I want to also point out today is a bit of a framework that we are developing, which is OK. 1So depending on the type of task and depending on the type of person, we can have several factors that affect decision making outcomes. So I know maybe this is a lot, but the idea is, depending on the decision making task,

So remember the three examples, medical with recommendations to diagnose, the risk assessment, and fake news detection. The decision outcomes such as task accuracy, task efficiency, AI reliance in the way that if the person relies or under relies or overlies on the AI system, these outcomes are really heavily dependent on several task related components. For example, expertise of the decision maker.

So different expertise, imagine you have two medical doctors, one which is kind of a newbie just newly graduated or nearly after the residence program and one with 40 years of experience. What is the decision output and the decision making collaboration with an AI assistant, given that one is expert and the other is just really freshly graduated. It's completely different.

The reliance pattern will be different. The accuracy will be different. Maybe the expert will understand the recommendation of the AI system, and maybe an expert will not understand.

Even personality, our personalities sometimes we have people are more sceptic to use AI technology, other people are early adopters and this kind of different traits of personality also affect how we use and how we rely and in consequence how decision outputs are affected. Let's also see task factors.

Imagine that you have someone say, OK, you have to decide something, like fake news detection, and you really don't know anything about fake news detection. This is really a hard task for you. If you have an AI recommendation, like an AI tool, what is your go to. It's just to rely on them.

Even if you don't know anything about this AI tool, if it's accurate enough, because it's difficult to you, you will rely on it, right? So depending on if the task is difficult for you, the stakes, so if it's really a high stakes task, if you are stressed to do this task, this will all affect the outcomes.

And after, so we have kind of three components that may affect this decision making. We have human factors, task factors, and then AI ecosystem factors, which is really if the AI is accurate enough. XAI type is if this AI system will provide an explanation and what kind of explanation it will provide.

So if it's features that were important to the recommendation, if it was counterfactual rules, this kind of stuff. Even the AI design is important. What I mean by AI design is really, I mentioned that a doctor has a platform of checking.

You are talking with your doctor and say your symptoms and so on. And he is just recording all the stuff. And one design, he already has a prompt writing, okay, I have an AI recommendation saying this is, I don't know, sepsis. Or the medical doctor can open another design, completely different, open the AI recommendation if he wants to hear the AI recommendation.

They are totally different designs, just one which is the one that and the doctor has to see completely or other one that he just clicks in to see the option. So these are really different choices that really impact the outcomes.

So I think I made my point clear. Let me just pass to two or three, depending on the time, user studies that we did just to show you how this is real.

So in the first study, this is just a dummy study, but kind of have implications for general decision tasks. And here we went to the mushroom edibility assessment task, which is really just we ask people to say, oh, do you think this mushroom is edible or poisonous? And people were thinking, oh, I think it's poisonous. I think it's edible for a lot of sorts of reasons. And then they have a recommendation, an AI recommendation saying, oh, I think it's edible, or I think it's edible. So what we manipulated was if the AI recommendation system had explanations, presented explanations to that reasoning, so with or without explanations, and if the decision was made by one person only or by a team of two. So just for the sake of illustration, so the decision could be like there is the mushroom, there is some characteristics, and the persons could say if it was edible or poisonous. And in the later page, an AI recommendation would pop up and they have to decide again. Okay, based on this recommendation, it is edible poisonous. And so there could be one condition, one person doing this, another two persons, with and without XAI.

What we really found was quite interesting. the effect of presenting explanations, so for the people that were given explanations of the AI system, the explanations had an amplified effect on teams. What this means is that if the explanations were not present, okay, if the explanations were not present, teams relied much heavily, over relied much heavily on the AI recommendations than individuals. And our idea is that You are two people trying to decide something, a mushroom edibility assessment, and you have an AI recommendation without an explanation. Your idea is just to try to think and discuss between the two why the AI is giving that recommendation, and you will try to argument in pro of it. So you are biased to assess that. However, when the explanations are present, teams were much more quicker to detect incorrect recommendations from the AI system. So explanations really improved the critical thinking and the discussions. This is just one example that different settings of the decision-making task with AI assistance really affect how the decision outcomes will behave.

Another example, and this I think it's fun, it's still the mushroom edibility assessment, but here we manipulated the modality of the AI. So instead of being in just a computer interface, so we have two conditions. One, the recommendation was just in a computer interface. In another, it was held by a robot. Just a robot saying, oh, I think it's edible. I think it's poisonous. And then explanations, yes or no. Sometimes yes, sometimes no. And the thing is, what do you think were the results?

Yeah, exactly. They trusted much more. They relied much more in the robot. And the decision performance increased because we manipulated so the AI was more corrected than the humans. So this is just two examples.

I have here a third one, but I don't think I have time. So this is something that we conducted with medical students. So this was really a clinical assessment. We presented medical students in the fifth and sixth year of their studies with two clinical cases, just came up cases, but real ones. And we manipulated, so several students did the clinical cases without any AI tool, several did with AI recommendations and recommendations were recommending exams prescriptions and diagnosis and other ones with AI and explanations. And what we really found is that the clinical score, so we had a score of how the clinical case was assessed, and really improved when AI was present, which is great. But it was also a first thing, because they were medical students with few experience. reliance was much heavier when explanations were present and not when AI recommendations alone were there. However, when explanations were there, critical thinking increased and so the efficiency, so the time to resolve the clinical queries decreased.

Compared to the baseline? Compared to the baseline. So several takeaways.

Conclusion

I know that, you know, There is a lot of hype on the AI, and it's great. I think it's a great tool.

But we also need to assess how we are interacting with the systems and how we are, you know, cooperating with them. What are the consequences? What are the outcomes?

And we have really to assess if we are better off in all the cases with these tools, okay? So it's really important to understand dynamics between us humans and these AI tools. And I think we should really remain critical thinkers.

And this is not to like don't use AI, not that the point. It's just to say that I think critical thinking is really a huge important skill, a human skill that we have.

So one question that it's a foot for thought question is that should we optimize AI tools like generative AI and other AI tools for decision performance, like outcomes and accuracy, or for human critical thinking? Optimize these tools so that we enhance our critical thinking and not just the output of decision performance.

So this is just a question. I don't know. We can discuss later. So yeah, thank you.