From the event: Mindstone Singapore June AI MeetupWhat I/we learned building with LLMs
View event

What I/we learned building with LLMs

Introduction: Enterprise AI from a Microsoft CTO Perspective

Charlie Harrison I am a CTO at Microsoft so I look I don't know everyone knows Microsoft from PowerPoint probably teams Microsoft is also a huge hyperscaler like Amazon and Google we make servers we make data centers we manage huge

enterprise business applications okay and I'm here I I work across Asia and I work across all the different technical solutions that Microsoft sells I'm in a a small -medium business segment, which means we have a lot of customers.

We have 6 ,000 customers in Asia, and I'm talking to customers probably 10 -20 a week about their AI journey and transformation, and how they're approaching everything we just talked about. The themes are pretty consistent with what you've heard already from Marcus and the Polynes teams, but I'm still going to talk about them.

Ethics and Workforce Impact: AI and Job Change

I do want to address the ethical question because I've worked in big tech and I've worked in big multinationals for my entire career. Before I joined Microsoft, I was at General Electric for eight years in a CIO role, so I was there involved in enterprise change, transformation, tech adoption. Before that, I was at Motorola for 20 years, and again, it was technology and tech change and tech adoption.

AI is not the first time we've introduced technology that makes jobs obsolete. How about the ATM, bank machines? When they first came out, everyone thought, oh, that's the end of the bank teller. We're not going to need them anymore. We still have bank tellers. They just do different things. They do things better. the quality of service for the customers gone up significantly.

So it depends a lot on the company, the ethics, their business vision. Are they going to try to grow with the extra manpower they have, or are they going to try to cut cost? And that probably depends on the state of the business they're in. I worked in GE when they were in a terrible situation. We had to let a lot of people go. When business is good, you're investing more and you're growing.

So let's count on the good companies to make good decisions for their people and invest in us overall.

Have I got a clicker? Yes. Okay.

So I think, and I've got a long history of transformation, so even though I'm in a technology role, about 140 engineers in Asia that work for me to help me convince customers to buy Microsoft products, we always try to sell the right products for the right problem, and we're trying to sell a lot of AI at the moment.

How do I move forward, Marcus? You can click. I'm going to move fast.

Well, I'm not doing very well so far, am I? Okay. Right.

Why LLMs Aren’t the Main Story in Enterprise AI

So it was interesting when Marcus said, hey, Charlie, can you come and tell stories about your experience with LLMs? I was like, what? Who talks about LLMs anymore? We don't talk about that.

But a lot of the press loves to talk about LLMs. Models do matter. Yes, that's for sure. But they're not really what makes the difference.

The models are getting better and better. In fact, in the last three months, there's been more than 60 new announcements about models. Cloud 4 .7, Cloud 4 .8, Opus 6 .7. In fact, Microsoft, we just announced seven new models on the market. So they are important.

They're critical, but they're not what's going to make a significant change in the enterprise. By the way, I am going to talk about enterprise today. I'm not talking about you as a person, your own personal productivity, how you do your daily traveling and so on, how you manage your finances. I'm talking about a company and how a company gets the most out of AI and their strategy.

So the model has very little to do with it. it. It's really about people and adoption and organization.

The Adoption “J-Curve” and Lessons from Past Tech Waves

The J curve, you've probably heard about that before.

As we introduce new technology into business, we find there is a lag in adoption. It takes a long time for people to adjust.

When we first started using PCs, no one knew what to do with it. It sat on your desk. You played with it.

I played Donkey Kong on mine a lot when I first got mine. Didn't really move the needle on my

work for a few years until we got networked and we started sharing files and then we got got email. And then, wow, there was a massive boost in technology change.

So similar situation for AI, except I think the scale is faster, the impact is higher, and the way we need to approach it needs to be much bigger.

So I think an important measure is when we get to success and where we want to get to with AI in the enterprise is if we take a look at how we really change the way we work.

So been through a few tech transformations.

I used to work at Motorola when we were building cell phones.

From the Web to Amazon: Transformation Comes from Reworking Processes

When did, let's go back to the internet, that's the Amazon example. When the World Wide Web first came out it was so cool.

I remember Gopher, you could go around, you could find cool sites, new information that really transformed the way you lived or the way you worked.

It didn't really. It was interesting, it was more information, but you get the information already from a book.

You can maybe find out about a company faster, but it didn't really change the way we operated.

But then when Amazon completely redesigned the way you you could buy products online. They re -gutted the fulfillment process, logistics process, the sales process.

So you got, as a consumer, you got a single experience. That's when the big kick and transformation happened.

From Prompting to Skills to Agentic Workflows

So as we look at AI, where we are right now, which is prompting, you can take courses on prompting, and our sponsors probably teach courses on prompting, so I won't say anything too bad about it, it's important.

Where we wanna get to is, the next level up is a skill. If you've played around with co -work, you know what a skill is. that's actually kind of a permanent prompt you teach the agent how to do the

same thing over and over again well actually we don't want you as an employee writing prompts we want the agent managing itself it's all written it's already done it's operating in the background just like when you clicked buy on Amazon it all happened in the background that's where we're headed for

Five Requirements for Successful AI Adoption in the Enterprise

but in order to get there we think technology and we're a technology company the technology is a really big part of it but there's actually five five things that I see consistently need to be achieved for a business to be successful in the adoption of AI.

1) Define a Clear Business Strategy and Desired Outcomes

The first and foremost is they must have a business strategy that says they want to use AI. What do they want out of it?

How much money are they going to invest in it? Where do they want to go? Is it they want to transform customer experience? Do they want to completely change their manufacturing process?

Do they want to come up with a brand new product? Or they just want to fire people and save some money? So they've got to define what it is they want to do. Be very clear about that.

It's got to be top down. If you don't have top -down buy -in, you won't make any progress.

2) Build an Execution Strategy and Operating Structure

Second, you need to have a strategy for how you're going to do it. How are you going to do it?

Are you going to break down the processes? Are you going to do it organically? Are you going to have a center of excellence? Are you going to have an AI officer?

You need to build a structure within the organization. There's a few different ways to do it, depending on the type of business that you're in, but you need to build a capability to adopt AI, change management, organizational design.

3) Create a Culture That Enables Change

You also need to create the right culture. You've got to get that into the culture. The culture has to be ready for change.

sure, employees are afraid they're going to lose their jobs, so they're not going to help you with your AI transformation if they think they're going to lose their job.

If you help them understand that actually by helping us design this new process, you're going to get to go do new and exciting things, they're going to be much more happy to help you.

Even if it's an old company, and I talked to some companies that have been around for a long time, you've got a lot of old -fashioned employees, even they are able to actually make the change and start to get employees to

4) Put the Right Tech Stack, Data, and Security in Place

adopt with the right leadership and the right pace fourth yes you need a tech stack absolutely I love you to buy Microsoft's we have a great one you need a tech stack there's a lot of technology involved in this let's be very clear you

need to have a data strategy we talked about data that allows the most current not just any data the most current organizational data that is needed specifically for the task that the agent is going to do you as an employee can

spend an hour searching through your computer yeah that person sent me that file I know where it is I can go look for it the agent needs to know exactly what data where it's going to use to make decisions and act on things so you

need a very clear tech strategy data strategy security is absolutely essential as well how do you want a AI to deal with ethical situations in the company how do you want it to manage your data how do you make sure that the the data that the agent has access to is data it should not share to other people in the company.

I think a lot of people use Microsoft product called SharePoint. You can share a lot of files in it. There was no security on it.

When agents first sort of had the ability to search across those, CEO salary. What do you know? Here it is. It just shows up in a file.

So putting all these controls in place, absolutely essential, has to come down from the top.

So then the model just is one part of the tech stack, right? Models are very important, but they aren't everything. This is what you need to be successful.

Avoiding the “POC Graveyard”: Why Pilots Stall

Okay, the POC graveyard.

So what we see in a lot of companies is everyone gets really excited, let's go do a demo, let's go do a POC, let's build something, let's get some agents running. And then they just kind of stop.

And I used to think the reason they were stopping is because the employees weren't trained up, they didn't know how to use it, they weren't adopting AI fast enough. Actually, that's not really the problem.

we're discovering it's the organizational challenge. The leadership structure's not there. The change management capability's not there. It's not the employees.

Even though there's probably lots of examples of people not taking advantage of it, that is kind of built into the company culture as well.

Driving Adoption Internally: Microsoft’s “Customer Zero”

So Microsoft, as a big AI company, we have an initiative called Customer Zero.

We all have to use AI. We get measured. How many days a week did I use AI? Did I use AI to do this?

Used to be, you came to a meeting and you used AI, you were kind of embarrassed. you kind of hide hid the fact that you researched this with AI now if you come

to a meeting and you've done it yourself like you said gave me what why did you do this yourself this is ridiculous use your use your free time for something else get the agents to go and do it so getting the company to think about how they're gonna get AI and how they're what the employees to do it is really

What the Work Index Shows: Organizational Factors Beat Individual Mindset

what's gonna move the needle as well and so Microsoft did this big survey we do it every year the 2026 work index survey because we have a lot of customers we can survey a lot of customers because we have a lot of agents we can survey that we also have this thing called

many people here i'm sure use outlook and teams and so on we have all this data we can farm from that it's all anonymized don't worry we don't know anything about you but we can see your work patterns we can see how you operate with the systems and the tools that you use and what we

were able to find from these studies is that actually one in three employees are held back by organizational issues we could do much better with AI if we fix the organizational challenges same actually same same statement here this is when I was going to take out 67 % of the employees are limited by their organizational challenges not themselves yeah here we go so this this is kind of

the results of the study the Microsoft Work Index trillions of anonymized data sources 20 ,000 different AI users surveyed 10 global markets what we found is the number one impact on success with AI was organizational culture, talent and support from managers. Individual mindset important but not as big as the other two.

So leaders setting the targets, leaders leading the way, sending the right incentives, creating the right environment. To do it is what is going to give you success in your company with AI.

So now you've got the right culture, you got the right change, you got everybody fired up, we're gonna go do this.

Turning AI into Business Value: Choosing Workflows and Building Focus Teams

How do you actually get success?

I think building what we said earlier, you need to pick the right workflows from the right outcomes. Not every workflow is suited for AI, certainly not in early adoption.

Pick one that's important, one that matters, one that is consistent. If it's an inconsistent workflow, it's going to be very hard to teach an agent how to manage it.

If you find out halfway through the process there's actually a step where somebody takes a piece of paper, they walk across the hallway and they get a stamp on it, that's not going to be work for an agent either.

We need to really dig deep on the workflow, pick the right teams to do it with the right focus and the right skill set, and then create a team of people to go and do it. Focus is what really makes a difference here, and this is where we see the POC graveyard I talked about.

Often it's because, even though the employees had the best intent, senior leadership's not bought in. There's no investment. There's no push to go and say, I want to make this process an agentic process, and I want to make it happen in the next two months,

and I'm going to set targets against it and I'm going to give you money and I'm going to give you time to go make this happen because actually as you guys have discovered it's really hard to do this it's not as easy as everyone thinks it's

it's easy to have an agent check your email it's easy to have your agents summarize meeting minutes for you but to actually change a purchasing process give that entire process to an agent requires a lot of thought a lot of

investment in analysis and data and decisions about who's going to own what in the process which is another point that came up earlier if you really want to have success this is where the senior leadership comes in you need a focus

team the focus team needs to have all the right tools so you need a tech guy on the team that can understand the tech stack the data can help build all that

for you but you need people who understand the process the people who actually do the work know what needs to happen know why we do the work need to

be as part of the team if you don't have the leadership vision you will not get the kind of results that we're expecting and you would hope to get out

Accountability in Agentic Processes: Assistant vs Teammate vs Operator

of this kind of a process Marcus talked about accountability and this is absolutely key as well so I imagine most of you work on the left with an agent as an assistant at the moment right maybe checks a few emails for you maybe it summarizes your meetings maybe it helps you with the research you ask it to go do something to find something else it helps you accountability is a hundred with you.

Maybe an agent's a teammate, so you've given it a task. You said, I want you to go check all my sales numbers, do a complete analysis every day, send me a summary of how my sales team is doing. It's now in support of you, but it's still working for you. You're 100 % accountable for any decisions that get made.

But when you run into an operator agent that's actually been given full responsibility to make decisions by itself, run 24 by 7, change the way, act on decisions, act on information that you've received in the organization, you still need to decide who's going to be accountable. This was a big point that Marcus was making. Without this, the agentic process will go nowhere.

It needs to have an accountable owner that's going to own the entire process, going to manage the performance of the agents in that process, going to calibrate their inputs, inputs, their outputs, going to manage change of that process. Accountability is absolutely key for managing this, and it fits right into the model that you mentioned, Marcus. So that was the lesson four.

Lastly, so you built a strategy. You made the investment. You got a team. You went deep into a use case.

You've assigned accountability to the person. Great. You built your first agentic workflow. Now what?

Scaling Beyond the First Use Case: Build a Learning System

Well, actually, you need to build an entire learning system. It's an operating model.

You said that earlier. This is going to be running for the next, until the next major technology change, right? Forever.

We're going to be changing business processes. What business process doesn't change? Constantly changing.

The market changes. The skills of the employees changes. Donald Trump starts a war in the Middle East, so all of a sudden oil prices go up. You need to change your business model.

It's a learning system. How do you pivot? How do you change?

You need to build a system around this. It's not a one -time, let's build a few agents.

Continuous Improvement as the Engine: Lean, Kaizen, and Change Management

to go solve a few problems and because Marcus knows me I'm a transformation guy it was a GE I did Microsoft I mean sorry Motorola Digital Six Sigma I'm an absolute lover of lean anyone ever heard of lean mindset lean practices right it's a fantastic tool to consider adopting in an organization for AI he

doesn't have to be lean and Kaizen there are other organizational transformation tools but to me the biggest thing that you need to be successful the AI and and enterprise is a change management process, continuous process improvement capability, because that's what's gonna drive the change.

You need the tech guy, you need the data guys, you need the agent builders, but you need the organizational change capability, and that's what's gonna make all the difference. Build it and then don't stop.

How to Get Started (and Repeat): Practical Next Steps

So how do you get started? Go and see, find a work process that gets stuck, figure out why it's getting stuck,

where should we have humans, where do we have humans doing too much work, where could we save capability where could we free up bandwidth what if an agent did this

pick the right team pick the right tools and and go build some processes but i think and then do and then repeat and then repeat and this is the whole continuous improvement process related to kaizen and lean it is a process operating system you're going to need to build

and use again and again it's worth the investment because we see customers that get fantastic results that do this we see like double -digit percentage changes and improvements and productivity it's real we see amazing changes not just on those

that are building brand new products come old -fashioned companies like Rolls Royce completely changing their production line saving 40 % things like

Conclusion: Invest in Change Capability to Realize AI Productivity Gains

that so if you're running a small company if you're part of a big company encourage this kind of an approach it's gonna make a big difference in the way the company operates and you as an employee will have a better experience and that's it

was i close to time no it is always 15 minutes yeah okay any questions

Finished reading?