Introduction

Today I'm going to talk about something which is happening now as we talk.

The Hype Around AI Agents

So how many of you have heard that 2025 is the year of AI agents, autonomous agents? Anybody?

Thank you. Thank you for that show of hands.

Anybody who would like to just tell us what do you understand by AI agents? In very simple layman language, if you were to explain to the person sitting next to you, what is an AI agent?

Understanding AI Agents

Absolutely, so essentially these are AI systems which can act which can assimilate insights, inputs from multiple sources, take actions as you had programmed them to, and arrive at certain next point, which again you had programmed them to autonomously.

So they are autonomous agents.

Then there are different grades.

You will find there are these agentic workflows which could perhaps, let's say we were talking about recruitment.

Now that the recruitment cycle is over, you as the HR manager would like to send an email to all the candidates who had applied. So you can create an agentic workflow which will have those customized email being sent out to them.

let's say you want to send out to everybody except top four candidates because you want to keep two main and two reserve that can happen so you can set up those agentic flows and then there are these if you remember anybody who has worked in RPA

Applications and Evolution of AI

robotic process automation, which was a big thing for many decades and did a lot of, yes sir. So yes, so now those process automations have become more intelligent, there's a lot of AI, but I'm not going to talk about that because I'm sure you already are getting a lot of feeds on agentic AI.

Challenges in AI Adoption

What I'm going to talk about, a lot of people ask the questions about the bias, fairness, and a lot of other stuff about use of AI for, let's say, recruiting, use of AI in healthcare, use of AI deciding, should I get a mortgage or not? And there are issues in that.

We haven't seen that. So AI has been biased. There are various reasons for it, why AI has been biased.

So again, not going there.

But if anybody would like to talk about, what do you think is the biggest problem in AI right now? What is the biggest problem of AI right now? Is it compute data people?

Ethics. Yes. Very important.

Data. Yes.

So all of this would come under, like for example, I'm sure there are a lot of senior folks here in the room who have seen all these, right?

AI is another general purpose technology. Before this, we had the cloud transformation. Before that, we had mobile internet. We have gone through that as human race. We have gone through all that.

Ethics and Governance

So the biggest problem for AI is neither data nor compute because you can always manage that.

It is like what all you mentioned about ethics, about fairness and how do you take care of it which can be put together in the larger umbrella of governance. How do you govern AI? AI has to be governed by humans because you want it to achieve a particular goal.

Practical Implications for HR

output you wanted to achieve a particular mission because you want, let's say, if it needs to be used for HR, there are a lot of functions in HR which are very, very time consuming for the HR team. Onboarding.

Or when you have, let's say, whenever you join a new company, perhaps you may take six to eight months to understand what is where, where it is. And you can't every time ask people because you feel a little shy about it. People will think, OK, you are coming from such a big organization to this organization, and you don't know this.

For that, you can definitely have a lot of AI supporting what to find where.

So coming back to this point, we realized most of the time, governance is put as an afterthought. You have built the model, you have created a product on top of the model.

For example, GPD 4.5, GPD 4.0 are the models. You made chat GPD on top of it, then you made custom GPD on top of it, another product.

And then you're thinking about governance, that okay, is it fair? Is it having any kind of bias against certain set of people, certain gender of people? That is a clip on that is too late.

Governance By Design

1So the idea here is, as agents become mainstream, can we have a concept where we are trying to bring in governance by design?

So by design means that when you start thinking of an AI product, or you start thinking of building an AI project within your company, you may be either buying it or building it, you start thinking about it. So the engineers and data scientists in the room, they would know that when you start the lifecycle, you will start

thinking about which data set to use, which data set to engineer, subsequently which model to select, what are the features which you'd like to have, can we build in some governance there? And that is the main subject of this, that we want to bring in governance by design at the initial stage so that what comes out as a product, especially when it's agentic, it remains truthful to what humans would like it to be.

It remains aligned. It remains safe.

By the way, my name is Himanshu and I do AI research and I also enable corporations, small and medium businesses, and amazing leaders like you to adopt AI. This is my LinkedIn if anybody wants to connect.

The idea is that how did we come to this particular problem statement and what was the challenge?

We realized most of the governance, as I mentioned, is done on completion of the product being inducted. It's too late. It has already done the damage.

There are issues in terms of compliance. There are issues in terms of regulations.

Let's say, I'm sure there must be a lot of entrepreneurs here. Anybody who is building an AI product who's sitting here?

Yes, just presented. So you would like to build a product which you, if you want to ship it across Atlantic, it should be accepted, right? You would like it to be confirmed into EOI Act.

I'm sure Stefano will help you to do that. But for that, if you have a team or you have certain practices which allow you to build in governance from scratch, you will create a product which has more feasibility of being shipped out of Ontario or of Canada or wherever you want to ship it, right?

So those are the issues.

The Concept of Governance

So what we thought about is that if we bring in a concept like this, it will allow us to have some very specific considerations for agentic AI because agentic AI is little different than traditional and generative AI. And we'll see and we'll consider it separately how in the next few slides.

And then I'll try to give you one or two examples in different domains. If we try to build in this kind of design, how it will be helpful.

And what all could be the matrices for evaluating? Anybody, especially my friends from engineering and data science background, anybody who would like to talk about why it is important to evaluate and have evaluation matrices in your model and your solution? What is the importance of evaluation?

Yes, sir?

Absolutely.

And any one of the business leaders here, would you like to talk about the business matrices which you will use and why do you use business matrices when you develop a product or create a service? Yes. Absolutely.

Addressing Trust and ROI

So after governance, the next biggest problem of AI, we solve so many questions, is trust. Trust of your users, trust of your internal and external stakeholders.

And then the third piece, which a lot of business leaders would talk about, is return on investment. If I invest x amount of time, effort, money, what do I get in return? So it is important that we get all these aspects grounded, and we have those metrics which help us evaluate our solution better.

So what is the idea of having governance by design? If you see, there are issues in agentic AI in terms of governance.

There are gaps in expertise. Not everybody is right now up to the required level of expertise.

There are regulations which are yet to be promulgated. But still, we need to build these solutions. People are using them.

So there are gaps of what is aligned with the regulation, what is not. Then how much is the limit of your innovation and how much do you conform to the regulations or compliance?

And especially those of you. So how many of you are working in fintech, finance, health care, hospitals, medical, medtech?

I'm sure you know how difficult and important it is for us to align with those regulations and compliance because it's all private data. There are a lot of private health information, personal health information, the personally identifiable information which we need to confirm because we do not want any kind of breach of data privacy, right?

Still, in spite of this, we would like to have the product realize the competitive advantage over our competitors. And that is why it is important to have an idea of a design-centric approach, which makes sure whatever we are building, it's built right from the scratch with the governance in mind.

Autonomous Agentic Systems

So let's talk a bit about agentic systems. As we were talking about earlier, a true agent tech system would be autonomous.

You just have given an indication. If it has to move from point A to B, it will move from point A to B. After that, take action and produce that result. Whatever the next step, we'll undertake that. That's a totally fire and forget kind of option.

it'll tend to achieve a particular goal, right? And it'll interact with the environment, seek inputs, update itself, deploy a lot of concept of reinforcement learning along with machine learning to achieve that particular desired state, and adapt on its own, and take decisions, which are very, very, all of these, when you read these five characteristics, what are they?

When you read this, and I don't say that this is agentic, what does it sound like? What does it look like? Doesn't it look human to you?

It's only humans who do this, right? But now we are building systems which will be able to do this and they're doing it right now as we talk to you.

That is why it is important if we do not ground them to safety and alignment, they can go haywire.

Risks and Governance Practices

So I'm not trying to be alarmist here but I'm just trying to bring this to your notice that that is why we require a very sound governance practice and those of you who are thinking of pivoting their careers in your respective line of business, risk management of AI, compliance of AI, AI governance is a very very important consideration and a job role, which is coming up in a lot of new organizations.

So that's something which you can look at. If you are passionate about making AI totally human-aligned, human-centric, it's a very good thing to dwell deeper into.

So far, any questions, any thoughts by anyone? Anything which, yes, please. Sorry. Yes.

So there are few matrices which are available. So you have some matrices which have been promulgated for agents.

Then there are frameworks. Previous to this, we worked on a framework which we called joint evaluation framework, where you have a combination of human experts, because there are not so many. You don't get human experts so readily available.

Then you use Language models as judge. You would have seen these tools available. There are a lot of research on that. And agents as judge. So you have a combination of these three so that you share the load and final responsibility of clearing that particular system will remain on the human.

But a lot of things which are repetitive, which can be handed over to agents. Let's say, for example, you want to check, you want to red team, and you want to do a lot of adversarial attack to see how safe the model is, how safe the product is, what is its water tightness or air tightness against cyber attacks how safe it is for your privacy of data how safe it is for cyber attacks so these things can be offloaded to the agents right they can do a phenomenal job in this a lot of other aspects like for example which are very good with the language models let's say summarization of of whatever has happened or extracting of certain context you can hand over to the the llms

When it comes to humans, the ethical part of it, should I use this particular product for this? Let's say, should you use a particular model which is doing triaging in a hospital setting or a spam filter? And to what extent, what should be the matrices for that?

I mean, the traditional question which generally people would come across in terms of using precision versus accuracy for spam filter versus triaging the patients for ER. Those kind of things. So there are these frameworks now are available where you are having a joint evaluation team of these three.

Anybody else? Any thoughts? Yes.

Human, large language models, and agents, autonomous agents. Yes, yes. Because you don't have so many human experts available and it's too expensive and too time consuming, right?

And you want to ship faster. So there are these options available. Again, none of them are foolproof. But then again, this is something which is a good gap. I mean, good way to bridge the gap till the time you bring something else.

Lifecycle of AI Systems

Okay, I'm just talking about different stages of life cycle and I'm sure you're familiar with it. So I'll just use this particular slide here where you are initially planning and designing a particular AI product or a solution.

Then definitely you have to have a good data pipeline. You have to acquire data and see how data, a lot of your time will go in this.

And I'm sure data scientists and engineers in this room will talk about it. How much time do you kind of consume? Almost 75% of your time in an AI solution goes in data pre-processing.

And then you do a model development, model training, select the model. And test and validate, very important point.

If you don't do this, I think there will be issues later on, both from compliance as well as complaints from your customer, and then you are not building it responsibly. And this becomes a very important part.

Once you have deployed it, you will have to continuously monitor it because you need it to remain where you want to. And the biggest problem with large language-based models, generative AI-based models, they have a lot of shift. So there's data shift, there's model shift, which will cause you to deviate from what you expected the model or the product to behave.

For example, last November, FDA in the US had to revoke license of almost 26 healthcare startups, which were Gen-AI based, because they were all trained on synthetic data. Synthetic data, easy to access because, again, getting access to healthcare data is not so easy. It has a lot of PHI, and a lot of builders will resort to building it where synthetic data has its own issues.

And when these products start interacting with real-life patient data,

they start drifting that is why you as leaders will have to keep this in mind that where the data is coming from what is the data lineage does it have any kind of bias somebody asked about it so there are tools now available which allow you to understand where is the data and this that's why as very responsible leaders i will urge you both When you are talking about building a solution, give sufficient time to actually designing these tools, these aspects of your product to your teams so that they do due diligence that where the data is being sourced from.

It is not just a public data set, which is only representing one subset of the ecosystem. So it represents only one demographic.

And when there is somebody who does not belong to the data set, they apply, they use it, it rejects it, saying that invalid. It happens. We have seen it, right? There are examples, live examples, which are there.

Mod gauge being rejected from a certain particular locality because there is a... intrinsic leak of data by zip code and you just the system just negates mortgage to those people who otherwise qualify because they have let's say double income both both both the partners are working they are earning they all have university degrees just because zip code A few decades back was a red mark zip code and the model was able to actually link it up.

It causes that and definitely it has a lot of reputation risks and it has happened. You would have heard about these stories. There are stories about these mortgage agents being used by financial institutions which have caused a lot of grief.

Ethical and Responsible AI

So it's important that we ground whatever we build in responsible and ethical practices. And the bottom line is, whatever is not safe for me, you, and all of us cannot be safe for anybody else.

When you're testing it, this is a litmus test. You may be a data engineer. You may be an AI scientist. You may be a business owner or a business leader.

Do this, and then you will think, OK, let me take some more time. Let me take 10 more days to build a data lineage and check Am I leaving somebody? Am I leaving one particular subset of data?

And then it'll do a good thing for us, right? I know we are short on time, so I'll just move ahead.

Reskilling for the AI Revolution

and this is where which is very very important and I would like to bring your attention to this doesn't matter what is the tool doesn't matter what is the technique today we are talking about these tools these techniques tomorrow something else will come what is important is you and I would like to urge you strongly all of you all of us we are part of it together the skill cycle across the globe is shifting right

Like somebody mentioned rightly, autonomous agents can do a lot of work, which otherwise would require a huge army of human talent.

So how do we reskill up ourselves? How do we realign? And pivot is very, very important.

1So wherever you think you can leverage and upskill, you do not have to be a computer scientist and a data engineer to use AI.

Usage of AI is like for example, we all are right now using electricity. Everything which is happening here is on electricity. None of us are building electricity in a power plant. We are just using it.

We are consuming it for our respective usage. We are using our phones, internet, everything is on electricity. But we are using it for whatever personal and professional achievements.

So perhaps it's the right time to upskill and pivot whichever way you think would be good for you to use AI for your own personal and professional growth.

Upskilling and Tools

and how it will evolve is that we will continue to work on this and we are trying to work on a toolkit which will be available which will be a plug and play toolkit which could help you use it along the life cycle of whatever you're developing especially in terms of agent tech tools and these are few matrices i think somebody asked about them so this is what we have been considering but there will be more of these

Conclusion

Again, if one thing you want to focus on is AI governance, is responsible AI, everything else will fall in place. And it's a very good career stream also to get into if you would like to. And there's a lot of openings in this field and less competition because we need more and more people in this.

Again, if we design the systems by keeping this particular idea in mind, we will have a lot of safety and alignment, which will be helpful, which will make sure whatever is the end result, it is aligned to human safety and our well-being.

Aligning AI with Human Safety

This is it.

If you have any questions, please ask. Happy to answer if we can't answer all the questions here.

And if you want to connect online, this is my LinkedIn QR code.

Thank you.