Introduction

I'm going to talk about reinforcement learning agents for autonomous decision making.

I've, you know, in the past I've talked about, I think the future of AI is narrow and I think the future of AI is really about reinforcement learning, I think.

And so I'm going to do kind of mathematical kind of reinforcement learning decision making with some examples of

of work that I've done over the last 20 years.

Speaker background

So just a little bit about myself.

I have a master's degree in aerospace engineering.

So when I was going to school in the 80s and 90s, I got exposed at that time to some of the most powerful parallel computers in the world.

I also got exposed to Geoffrey Hinton.

He was a professor at U of T, and he used to give all these seminars on machine learning and neural nets.

So I kind of got interested in AI, ML from

from Hinton and all his lectures.

I still have some of his old notes and stuff from the lectures I attended.

Early industry experience and leadership

I worked for the Air Miles Reward Program, built a data warehouse for them and built a consulting practice, was one of the first worldwide users of IBM's machine learning technology in the mid-90s.

I trained about 100 IBM consultants worldwide, wrote an AIML book for IBM, one of their

technical red books then i ran ibm's machine learning practice for several years and then i founded daisy intelligence a company that i ran for 20 years that was acquired a couple years ago and we had what daisy was focused on was kind of no human in the loop decision making using reinforcement learning what i'm going to talk about today and it was for merchandise planning promotion measurement fraud detection and medical diagnosis those the different areas we worked at and we managed some

some massive data warehouses like Walmart was like a 200 terabyte retail data warehouse.

And I have several patents in my name in those disciplines.

Current advisory work

What I've been doing recently is doing advisory consulting, working with startups, helping with just general management, doing fundraising, shred claims, grant proposals, just giving them general advice on how to run a business since I've done that for 30 years.

For kind of enterprises, I've been helping them build AI roadmaps.

Everyone wants to integrate AI into their product roadmap.

So I've been working on that.

And for investors, I've been doing due diligence on investments before they make them in technology.

So if you need help in those areas, feel free to hesitate.

Don't hesitate to reach out to me.

Why reinforcement learning and agents

So I'm going to talk about today is kind of this reinforcement learning, a bit about the motivation, and I'll define what I think an agent is, workflows, the results achieved, and talk about the infrastructure behind some of that.

Emergent complexity from simple rules

So in terms of some of the motivation, first is that emergent complexity arises from simplicity.

So all this talk about intelligence, I think, is really

it's really meaningless i think 1complex system behaviors arise from very simple agents networks of simple agents so if you've ever seen ants ants will always find the shortest path through a complex maze to food just by a very simple behavior their agent behavior they've ants will move along the strongest pheromone trail when they walk they leave like

pheromones and if they find a food source they leave stronger pheromones the pheromones decay with time because they evaporate so they will by by definition if you follow the strongest pheromone path you're going to find the shortest path to where the food was and so i've actually observed this sitting in a hot tub with a glass of wine putting a piece of cheese you know 10 feet from where i sound a little found some ants and water and put a bunch of obstacles in the way and you can watch

very quickly over time, they'll find the shortest path.

And that ant is very simple.

All it does is follows the strongest pheromone trail with like a high probability.

Some percentage always searches to see if there's other food sources, so there's some randomness in there.

But a very complex system arises from very simple agents.

kinetic theory of gases if any of you have taken gas dynamics you know if you treat air molecules as simple billiard balls banging into each other and conserving momentum when they crash into each other you can derive temperature and pressure and all the laws of thermodynamics from that kinetic theory again just a simple model of treating the atom as a billiard ball with elastic

collisions and conservation of momentum, you get a whole rich field of science out of that.

Cellular automata, if you've seen those like the game of life, these computer games where you have little simple rules for flipping pixels on the screen, all of a sudden emergent complex behavior arises.

And Stephen Wolfram is working kind of on his new physics paradigm where he believes the world is a hypergraph.

And simple hypergraph updating rules will derive the laws of physics.

And he's been actually proving that general relativity and quantum mechanics seem to be derivable from his hypergraph model of the world.

1So complexity emerges from very simple things.

It doesn't need necessarily any grand intelligence.

We're so wrapped up in this discussion of intelligence.

Reinforcement learning as optimization and control

The other motivation for this is reinforcement learning.

I think this is an analogy for how living beings or computers can learn.

I don't like the word learn for computers.

I think more optimize, you know, by doing random trial and error repetition.

You know, human beings, you know, as children, you learn shapes,

by putting shapes into a block with holes in them and you randomly pounded and eventually you figured it out and then through repetition you kind of got better and better and better but it's mostly kind of random trial and error and you know that's how we learned first order closed-loop control is another reinforcement learning you know you try this see what happens

Then you try again, you kind of iterate the input to get the desired output.

AlphaGo Zero was a deep reinforcement learning.

It started with no knowledge whatsoever except the rules of the game, and two bots played each other, played more games than all of humanity in three or four days, and then went and beat AlphaGo, the first version of AlphaGo, like 100 to nothing, the one that beat Lee Sedol.

So again, these are examples how reinforcement learning can do amazing things.

Defining an agent

So what is an agent in those examples?

I think it's just a simple snippet of algorithmic logic that creates some output based on input.

So the example of the ant following a pheromone trail.

So where the pheromones are strongest, it just looks around and follows the strongest pheromone.

That's the simple rule.

With like 90% or 10% of the time, it'll randomly go somewhere else so that it continues to seek and look for new food sources.

In the game of life, there's some simple updating rule that you look at the three pixels above you.

If you see dark, light, dark, then you are dark.

If you see dark, light, light, then you're light.

If you see light, dark, light, then you're dark.

These simple operating rules, you apply these iteratively, sequentially through a pixel screen, and you'll see emergent behavior happen on the screen.

And as I said, the kinetic theory of gases, conservation, momentum, and elastic collisions.

arises complex behavior.

So I think the agent is a simple local rule applied to some input generates an output, and then when you assemble those agents into larger systems, complex behavior can emerge.

Designing robust decision systems

Desired system characteristics

So what type of system characteristics do we want?

Obviously we want smoothly varying output decisions.

So in the case of if I worked in retail a lot, you don't want to, and we're helping retailers decide what to promote and how much product to order.

So you don't want one week to tell the retailer to order a million tons of bananas and the next week two tons and the next week seven million tons.

You don't want it wildly varying.

You don't want this hysteresis that occurs in kind of signal processing.

You want kind of smoothly varying output decisions period to period, right?

You want it to be fault tolerant.

You don't want the system to crash and then, oh my god, what should I do?

You want to have some reasonable, safe decision logic to be applied, even in case if the system crashes, what should I do?

So it should be totally fault tolerant.

And we see that in the Mars lander.

And that Mars lander is screaming to land on Mars.

There's no human with a joystick.

There's, I don't know, a 14 minute delay or something to signal.

It's got three computers deciding, when should I pull the parachute?

pull the chute now, two say yes, one says no, what do I do, oh my God, I gotta decide, so you have to be fault tolerant and no matter what the situation, decide what to do or things will go sideways.

And we need to be controllable, so there's this,

observe, orient, decide, act control loop that's existed in control theory.

So if you're getting a message here is that a lot of the things we talk about in AI, it's really we've already invented these things in systems control theory and we should be leveraging what we already know in systems control.

So if we want a fault tolerant system, we want it to be well behaved, we want to control it, all these domains have been invented.

So let's not reinvent the wheel.

Let's go back to these engineering domains and

Agent architecture and memory

leverage some of that knowledge so an agent architecture along the lines of what uh what i've helped develop with the companies that i ran and some of the patents i'll reference the patents at the end where these kind of arc agents so if you have a software agent that's a simple algorithm some algorithm it does you have input data coming in you have some memory you want to log previous executions of that agent

so that you can do some quality control and i'll talk about that this you know statistical process control how we make sure that the system is this agent is behaving sanely you then update the memory based on the output and you generate output data so this is kind of a simple api and you can assemble this so you know we've assembled this in software on a single server platform or a

or a multi-server architecture, but you can build platforms like this.

And I think that's the modern world.

There's a lot of AI platform services out there.

And so if platforms offer simple APIs that you can call, you can assemble those APIs in unique ways to get really interesting behaviors.

From calibration to control

Calibration via gradient descent

So here's an API for equation calibration.

So a software agent that does gradient descent that will

find the calibration weights, let's say, for a partial differential equation.

We've invented the dynamics of a system, or we built a predictive model that was a linear equation, and we want to calibrate it now.

So we have a simple agent that does gradient descent.

You feed it some input data that has your dependent and independent data.

That's what's required for all of history.

So you can do your gradient descent and find the model weights.

You have previous memory.

You have all your previous model weights there.

And this is important.

This is where you can now do statistical process control, which I'll talk about, is that from week to week or successive iteration of your model training, your model weights are totally different.

then something is going very wrong.

Your model weight shouldn't change very much if your data's not changing dramatically from iteration to iteration.

Then you update the memory with the new weights, you send the output data, which is a new calibration, right?

And so here's an example of a linear PDE.

It looks messy, but it really says that this year's sales pi is equal to last year's sales, that function in the middle, plus

the difference in the items I promoted last year versus the items I promoted this year times the prices I charge times the quantity I expect to sell PQ times the associative effect because when you buy pop you buy chips when you buy you know different products have these have these associations so this equation says that the betas in that equation are just the linear weights that are by category of products so we have a different weight for category of product and

and so we can stick this equation into the into the agent and we can give an input data of this year's sale or your last year sales last year's promotional quantities and i can train the weights to try to fit the data just a standard linear fitting with gradient descent right

Statistical process control for sanity checks

And then I want to use statistical process control.

This is what's been done in manufacturing and SCADA systems and electrical power plants and nuclear power plants for decades.

So you take the parameters of interest.

So in this case, my model weight.

So let's say my model has 10 weights.

And I've been training it now for every week for a year.

So I have 52 data points.

So I know what my average value of my weight is.

I know what the plus one and a half standard deviations above and below the expected value or the average value are.

And if I train my model, and as long as my weights appear within those control limits,

then everything's good.

You know, I don't have to worry about it.

If all of a sudden my model weight appears outside of its control limits, I've never seen this before, it's an outlier, then I expect my system was not going to behave well.

It's going to perform poorly.

It'll maybe be insane, right?

So now what should I do after calibrating?

If the model is out of range, I can ignore and continue, which is not a good idea because you'll get maybe some insane results.

I can stop and I can diagnose well that's not good if you have an autonomous system because now I got to stop my process and go in there and figure out what's going on or you can adjust to some previously known safe value maybe I'm out of range why don't I just set the model but parameter to last week and and then I'll figure it out without slowing down the autonomous system or I can set it to the average value so the statistical process control allows you to

Make sure the system stays within bounds and adjust the system back into those bounds if something has gone wrong and allows you to revert to the previous known state.

So we can do this.

Again, you should do this for every model parameter, every input variable, every feature, every data element.

You can have billions of elements in your LLM.

You should track those as you train and say, which are the ones that are changing more than I have ever seen before?

And that should give you concern

if they do change, that things aren't going to work well.

And then you won't be surprised when you get hallucinations or other odd behaviors happening with your output of your model.

Optimization and fault tolerance in recommendations

So then I could run an API for a reinforcement learning recommendation.

I can take the input data in the previous example where I calibrated.

I can take a specific period.

I can take the weights that were output out of that previous model.

I can run it through a genetic algorithm to simulate to say to some of the other features.

Now I'm trying to, if you go back to the mathematical equation, I'm trying to solve

x i x j y j for this year compared to last year i want to say how many products should i promote this year versus last year in every department so now i could try different solutions i want to find the optimal number of and the optimal items to promote this year compared to last year so now i'm doing an optimization so i'm going to run a genetic algorithm to optimize and try out all the different combinations of products

using my previous recommendations, what I've done previously, and then my agent will come up with a set of items to recommend.

I'll compare those in history.

I'll say, geez, I've recommended that the same as last week.

Well, that's kind of dumb.

I don't want to do the same promotion as last week.

Or I'm recommending items I've never recommended before.

Okay, I'm applying my statistical process control thinking.

So we've now assembled two

two of these together to get some more complex behavior.

I'm training my model.

Now I'm using my model to optimize and make a decision.

Then to do fault tolerance, maybe I should run five versions of my optimization.

And does my model generate the same answer five times?

If it's the same process, it should come pretty close.

If you get five completely different recommendations, then do you believe any of them?

If you get four the same, one slightly different, okay, that's giving you confidence that you're doing the right thing.

You need an adjudication decision maker to say, okay, great, I've got five answers that are the same, I feel good, I'm going to go with it.

Or if I've got five totally different answers, I should stop the bus here and

and decide what I should do because my system is insane.

It shouldn't be giving me five completely different answers running with the same input data, the same input logs.

So it's important to have this fault tolerant wrapper around it.

And this is how, again, we can protect our systems from doing stupid and dangerous things that will harm the business.

Workflows and reusable agents

Retail merchandising workflow

So you can build, this is a workflow we did for optimizing item selection for retail.

So everything with a star was kind of like an agent.

We saw we had data processing agents that just created features, all the feature sets.

We had different types of features.

We had last year sales, that was one feature, a piece of code that could calculate that.

Affinities doing association rule mining, calculating when you buy pop, you buy chips, et cetera.

So there was different agents that did different tasks.

Then we did the calibration task.

We have a QA.

The QA wrapper was the statistical process control.

Then the outputs from one agent became the inputs to the next agent.

And we strung these agents together to get a complex system behavior, which was selecting items to recommend for promotion, selecting prices, selecting order quantities.

And so now these agents could be reused in different paths.

If I want to build a predictive model to predict fraud, I might use the same feature creators to create features for another model.

I can assemble these agents in different ways.

And if I give you these agents in a platform or allow you to download it, you can build your own workflows that do different things.

You know, this is really...

And innovation is taking what people have built and extending it and doing something new and unique.

And I think the way technology is going, the previous talk about how you're packaging up algorithms is a great way to package up innovations and share it amongst different people.

And you can come up with new and innovative things with capabilities without having to build everything from scratch.

Insurance claims and pricing workflow

So example of a reinforcement learning workflow for insurance was, you know, we haven't built an agent for policy pricing and underwriting, so coming up with a risk rating for a policy quote, and then once that policy has claims being applied, I can then adjudicate those claims, should I pay them or not?

If I decide to pay them or not, do I first check is it fraud or not?

So I can

pay a claim if it's fraudulent i'm definitely not going to pay it if it's not fraudulent i might pay it if it's not fraudulent i might go subrogate it mean i can go sue the other insurance company and no fault insurance and get my money back or i can just automatically process it so here's kind of a workflow built from four different agents that achieves many of the tasks in claims processing

Quarterly-to-weekly retail planning loop

In retail, as an example I show, we did kind of quarterly promotional product selections, so you're planning your promotions quarterly in advance so you can go negotiate with vendors and get deals on products.

Then weekly, you're selecting those products, then you're selecting the prices, and you're selecting the order quantities.

You're kind of iterating those three things, item, price, order quantity, to get the optimal mix of promotion and non-promotion.

and then you're putting them out to inventory allocation how do i allocate that order quantity to store so again i get some complex merchandising behavior by assembling four or five agents that do different things right the one key thing that in in any one industry or in any one company you should have one mathematical theory that applies that all the agents use so that mathematical equation i showed back here

A common mathematical theory for an industry

This is the mathematical theory for retail.

Here we can do item selection, we can do pricing, we can do forecasting, we can do assortment planning, all with the same mathematical equation.

So all your agents are pulling in the same direction.

They're not working at odds to each other.

Some of the challenges when you buy vendor tools that do different tasks in the same domain, that they might not be optimizing the same metrics.

So when you have the same mathematical theory that underpins all your agents,

then you know that they're all working towards the same metrics and they're all using the same decision-making logic.

So that's an important feature that we built into the agents that we designed and the workflows that we designed.

So here's a common theory of retail.

These are the differential equations that govern how retail works.

Differential equations for retail

i won't spend any time on that but there's a macro theory to decide where to put stores relative to your competitors if you have bricks and mortar or where to where to build websites for online retail it's really a gravity model if you recognize kind of a the you know laplacian equation there del squared phi is a gravity model for doing site location then there's the

the performance the change in sales over time as a function of the difference in promotional quality prices and orders year over year and then there's how do you lay out your stores where you put the products in the store on the shelves impact how sales happen and so you can iterate the set of

equations, you embed solvers to these equations and those little agents and you assemble complex workflows to achieve the desired outcomes with one theory that governs all the decisions you would make in an industry.

Why theory first, then data

So the approach we used at Daisy was to, the company that I ran and founded before, was to create a mathematical theory for each industry and then we built workflows around that mathematical theory.

And so the proof is that this works.

The mathematical theory is the way science is done.

You come up with a theory first.

You don't get data.

You can't learn complex dynamics from data.

So retail is a business, if you have to choose 2,000 products to promote out of 50,000, there's 10 to the power of 3,600 possibilities in the 50,000 choose 2,000, the combinatorial math.

So you can't do supervised learning when there's that many

data points or that many states that the system can be in.

So you need to do something different, which is why we invented PDE.

So I think that's the scientific method.

Come up with a theory, then go get the data that encompasses the theory, the features in your theory.

So in retail, it's simple.

The theory was, you know, products have affinities.

If you buy pop, you buy chips, there's relationships between products.

If you promote a product, lower its price, people will buy more.

Products are elastic.

There's seasonality.

Different products are bought at different times of the year based on holidays and weather, right?

And there's promotion.

If you promote something on television or on the front page of your website, that'll drive more sales than if you don't tell anybody about it.

So these common features you assemble into a mathematical theory.

You go then get the data second.

Results and impact

Retail outcomes and patents

calibrate, iterate, see if your theory works and then you apply that to the that's that's the scientific method and so this works these are examples from the daisy company in retail the blue the graph shows year-over-year sales so the blue line

prior on the left was prior to using this reinforcement learning system retail sales year-over-year sales was declining which is bad and almost all the examples the red lines were either flat or declining after starting to use the reinforcement learning system to decide what the optimal combination of products prices and computer ordering we were able to turn around sales and grow company sales by up to five percent which is massive you know for a

largest retailers about 30 billion in sales.

We grew their sales by more than a billion dollars a year by optimizing merchandise planning using these no human in the loop agentic kind of workflows based on reinforcement learning.

There's a bunch of patents.

I showed you some of the contents of the patents.

If you look at the USPTO.gov, you can read in more gory detail if you're so inclined about what I kind of this example I showed here and some other examples.

Practical infrastructure and cost

We did this in the cloud in GCP.

Simply had a Hadoop cluster to do all the data processing.

We used a GPU cluster to do all the reinforcement learning and iteration to find the optimal outcomes.

then we presented their solution back to the clients via the web so a very standard simple infrastructure to do this massive computing Walmart we had like 200 terabytes of data Albertsons we had 100 terabytes of data cost about 10k a month on the Hadoop cluster and US dollars to run this on an annual basis for clients this was kind of using committed use discounts so not even kind of

ephemeral we kind of had committed use server always on we did we had one single gpu vm so this need for computing power is massively overstated we're doing the most harebrained mathematical calculations using old generation nvidia cards

Four cards on a server cost less than $1,000 a month.

You don't need to build a nuclear power plant to achieve amazing results.

And we ran this every week, so retailers do decisions on a weekly cycle.

Took about two days of processing to do on a weekly cycle for insurance.

We were doing fraud detection and claims automation on a nightly batch, cost a fraction of that, maybe 10% of the cost of that to do insurance claims adjudication and fraud detection.

So you don't need to blow your brains out with massive infrastructure farms like these open AI guys are talking about a few

use the scientific method, do narrow point solutions, doing complex mathematics, science.

Conclusion

You can do very amazing, accomplish amazing things from very simple tasks.

You know, complex behavior emerges from simple agents doing one very specific task.

So thanks very much.

Happy to answer any questions.

Feel free to reach out.