Introduction

Hi everybody. I hope everyone's had some good pizza. Normally I end up doing this talk before pizza, so I have to rush through. So bear with me.

I kind of scrapped this presentation together this afternoon, so it might be a bit like here and there, so just roll with me for a minute.

So I'm Drew Steele, two -time founder, operations and strategy consultant, and things that interest me are space and physics and the future and unanswerable questions. So that's kind of like why I think about this stuff a lot. That's my credentials.

Axioms: energy growth, AI energy use, and efficiency

So just ahead of this kind of thought train, I think we should just lay out a couple of axioms just to kind of like roll with this. So like, let's just say global energy production increases over time. We should just say that that's a thing because if it doesn't, then none of this is relevant.

I think we should also just say energy deployment to AI will also increase over time again, because if it doesn't, this is irrelevant. And I think we should also just say that and or energy efficiency will also increase over time.

What the trendlines suggest

So these are just some graphs showing that kind of we're using more energy. We're giving more of that energy to AI and the energy that we produce is used more efficiently over time. So, you know, gigaflops per watt is improving and so on and so forth. So we're getting more power, giving more of it to AI, and it's better at using that power.

Possible futures and why alignment determines the path

So then I kind of like, I thought about this process and like, there's a couple of different ways we could go. There's kind of like a good pathway ahead, utopia. Maybe there's a bad pathway ahead, like dystopia. Maybe we kind of merge with AI in some sort of symbiotic thing. being Symtopia, maybe. 1And I think the choice of pathway that we go down ultimately boils down to AI human alignment.

A quick audience vote on the pathway

So I wonder, just by cheer vote, maybe, which pathway should we go down? Should we go down the good pathway? Yeah.

OK, maybe. Bad pathway? OK, a couple of people want to go down there. And then the unknown pathway, we'll ignore that.

Okay, so I think there was slightly more for good pathway, which is good, because that's kind of the way I've learnt this one. And I think the bad pathway probably is good to explore maybe another time. Okay, so yeah, good pathway.

Generally, I think humans are future cats, and I'll kind of like come back to this statement in a bit, but like that's kind of the overall premise. Slightly before we're cats.

Automation and the march toward near-zero marginal cost

I think as we are now on our journey to becoming cats I think that we're gonna

Which costs fall (software, extraction, construction, entertainment, labor)

see a couple of kind of things happen I think software production cost will fall close to zero I think material extraction cost will fall close to zero construction cost will fall close to zero entertainment cost will fall close to zero and labor costs will fall close to zero and by zero I kind of mean like

marginal cost of energy plus a bit and so we've had some chats earlier today from people using AI in the trenches and we're already kind of building software quicker than ever before and using less people to do it and we can make eight times as much stuff in one times as much time and then it's eight times as cheap.

So you know like if everyone uses Slack and you're paying for Slack how far are we away from just hey Claude Code 7 make me a free version of Slack for my team right. So those costs, I think, will maybe approach close to zero.

And as AI sort of starts to embody into the real world with maybe things like humanoid robots, hey, can you go and extract raw materials for us, please?

And we're already starting to do that with mines in Australia where robots are doing some of the work over there, and that's going to increase.

And we're already starting to do that with mines in Australia where robots are doing some of the work over there, and that's going to increase.

Same with construction, same with entertainment. Hey, Netflix can have a movie that's kind of like Avatar, are, but a bit better, but has a happy ending or whatever.

So I think all these things will trend towards lower and lower cost over time.

Retraining vs. replacement (and the idea of “merge rate”)

And I think as we do that process, as we go along that journey, something we've got to think about is the rate of retraining everybody, versus the rate of everybody getting replaced, versus maybe the merge rate.

So I kind of put the merge rate in there. But it's less just kind of ignore all that for the second, it's not a very long presentation, we've only got 15 -20 minutes.

But really, we've been through technological revolutions in the past, we had the rise of steam power, and you know, the weaving loom that everyone used to use rapidly replaced with more modern machinery.

And so the replacement rate there was whatever that rate was, and the retrain rate was quicker than that replacement rate. So it's a

bit annoying, everyone has to retrain, but no one, no one's, you know, doesn't matter too much. Same with electrical power power coming out, compute power as we develop chips.

We're kind of now in the cyber power type of thing where we're starting to build systems with AI and we're starting to get AI to build systems for us.

And so I think the problem we'll face is that the retrain rate will be slower than the replace rate. So we'll replace more people quicker than we can retrain those people.

There'll be new roles and new jobs and new things coming up. up. Which one will be faster than the other? I'm not sure.

Slightly after with cats, let's just jump ahead. Let's just go through the bad bit. And then let's just get to cats. Great,

The “good pathway”: abundance, post-scarcity, and what humans do next

we could maybe have AI solving kind of diseases for us and like improving medicine and maybe with sort of free abundant kind of material extraction, we've got an abundance of materials maybe there's automated agriculture so we could have an abundance of food

and we could kind of like again probably needs a longer talk but you could fast forward this process to like an age of abundance where well why why would you need to go to work to earn money to buy the thing that you can't just have when actually in the age of abundance you could just have that thing so you don't need the money so you don't need the job so and so would would

From scarcity-driven society to abundance-driven society

humans then end up with kind of an abundance of time as well and so that might lead us to maybe like a new type of society we are currently in a society driven by scarcity there are not enough things for everybody so we have to fight over them or we have to work for some sort of currency to buy those things

maybe maybe we you know what do we do if there is no scarcity I don't really know know the answer, but maybe we'll find out.

I think if we don't have that scarcity problem, and we have lots of time, you know, what do we do?

Meaning, purpose, and exploration in a world with more time

Maybe we're trying to find what our meaning is and what our purpose becomes. A lot of people's purpose today revolves around work or time spent acquiring resources or deploying resources.

What are we going to do? I don't know, maybe we explore reality, maybe we become more creative, maybe we explore the universe and so these kind of like futuristic things that we could start to sort of get into maybe in this in this environment where where we don't have to sort

of struggle so much okay so like back let's go back to humans of future cats so I've spoken through the cats thing in a previous talk I don't know if anyone was here for that but

Humans as “future cats”: the knowledge horizon metaphor

A cat and a laser pointer: understanding within a horizon

let's just look at a cat's understanding of a laser pen so a cat can see the laser pen pen and it can touch it and it can move it around and we can shine a dot on the wall and the cat can interact with it using its five senses.

And so this is the cat and the little circle is like the cat's knowledge horizon, how much knowledge the cat has. And the laser pen sits inside that knowledge horizon, the cat can see it and it can smell the laser pen and it could touch it. So as far as the cat is concerned, it understands everything it thinks it can know about that laser pen.

And maybe maybe the cat could expand its knowledge horizon a little bit to make some sort of correlation. When I press button, red dot happens, you know, so it could maybe expand its knowledge horizon somewhat.

But no matter how much we try to teach the cat how the laser pen battery works, that's fundamentally beyond the cat's knowledge horizon. It cannot understand that.

Scaling the horizon: humans, ASI, and the unknowable

So then if we kind of think, okay, well, do all intelligence systems have a finite knowledge knowledge horizon. Maybe there's some equation to figure that out. Neuron density, brain volume, transactions per second, how much time you're thinking about it, how much energy you can consume, I'm not sure. But I think all intelligent systems must have a finite knowledge horizon as opposed to an infinite knowledge horizon.

And so maybe this is us, this is a human, and we have a much larger knowledge horizon than the cat. And perhaps our knowledge horizon actually encompasses the knowledge horizon of the cat.

And things that we know about cooking great whatever loads of stuff sits inside our knowledge horizon and similarly we can also expand our knowledge horizon a little bit maybe we could put something in that expanded bit like quantum gravity something quite complicated but we could still probably figure it out maybe you know spit got to expand our horizon a little bit but it's still in there but maybe something that sits outside of our knowledge horizon uh we would we cannot know what that is because it sits outside our knowledge horizon so there'll be things that we cannot cannot understand.

And maybe that's like, what is time? I don't know. Maybe questions like that might permanently live beyond our ability to understand.

And if we also kind of then look at like an artificial super intelligence, they probably also will have a knowledge horizon, but their knowledge horizon, I believe, will become larger than our knowledge horizon and therefore will be fundamentally unknowable to us. So maybe that's the ASI's knowledge knowledge horizon, which encompasses our knowledge horizon and the cat, that's the dot there, the little tiny dot, cat's knowledge horizon.

And so quantum gravity, tricky for us, maybe it's easy for an ASI. But maybe something else over there is knowable to ASI, but long, no way for us. And then who the hell knows what's over here? There could be things that exist this beyond the ASI knowledge horizon? Who knows?

So then, if we kind of take take that kind of that kind of logic with us,

Core AI alignment when rules fall outside our understanding

and we think, okay, well, how do we travel down the pathway of towards utopia or towards dystopia? Or how do we how do we force that pathway?

I think we have to look at fixing AI alignment, like core AI alignment. And so kind of what I

Objectives, constraints, and alignment inside our horizon

I mean by alignment is we can give AI an objective. So let's say this objective is inside our knowledge horizon. So we say, Hey, I, your objective is to get from A to B, use some sort of legs come up with a leg design. Cool, fine.

Yes, you have to start a Yeah, you can't, you can't not start a No, you can't just be one massive leg and fall over. That's not allowed. No, your legs can't have wheels, right?

So we can set kind of boundaries to this objective, so we can give it an objective and we can provide like a ring fence of boundaries so that the objective that it achieves is actually kind of what we want, it's aligned to what we want, i .e. if we didn't set the boundaries it could just make one massive leg and fall over, it's got from A to B, but it's not what we wanted, so we'd set it boundaries.

And this is this is kind of easy when it's inside our own knowledge horizon because we can think of those rules because we can understand those rules and we can then define those rules and it gets a bit complicated okay well what about if we said an objective maybe this is on the edge

of our horizon okay keep the occupant safe in this driverless vehicle okay well but here's some rules ah yes the occupants should be prioritized uh but no you shouldn't take evasive action if if you're going to injure someone else oh but wait yeah you should actually take evasive action if it if it saves more lives than if you just let the driver die like we're kind of getting into like

weird territory now where we're trying to give the AI an objective keep the occupants safe but the rule set the rule boundary layer that we have to provide is getting a bit fuzzy and this gets even worse than if we give it an objective where some of those rule sets will be definitely outside

“Make Earth a utopia”: where boundary rules get fuzzy

of our horizon so for example we could say make the earth a utopia let's go down the utopian pathway uh but don't kill anyone please uh and i was like cool right promise i won't kill anyone and we're like oh wait don't freeze anyone either don't don't freeze anyone okay i won't kill anyone or freeze anyone okay but also let us do what we want to do like we want to be free we don't want

to okay but unless it is bad we shouldn't do bad stuff okay but then also make sure no one's unhappy and also let us be individual and so on and so but these rules that we've given it you know don't don't kill anyone, don't freeze anyone, still sit inside our knowledge horizon.

And so what rule set might the AI use that is outside of our knowledge horizon?

The cat analogy: solutions we can’t anticipate

And so kind of if we go back to the cats, like let's say we wanted to kill all cats on Earth, the cat might look at us and go, okay, how are they going to do this? They're going to maybe like fight us, they're going to like scratch us, they're going to like, you know, cat ideas.

But we might actually go, we're not going to do any of that, we're just going to edit it your genes so that you only ever have female offspring and then we'll just wait we'll just wait for all cats to eventually become female and then we've won but so that solution was inside our knowledge horizon but outside the cat's knowledge horizon so it could not ever possibly have like put a rule around that to say uh you know don't change our genes so the same problem i

think will come with us um how do we give an ai an alignment boundary so that we can give it an an objective, but it is aligned to what we actually want it to do, if we fundamentally don't know what rules to give it because they're outside our knowledge horizon.

And so this is kind of like core alignment, maybe as a problem. Some of you

might know this film.

Toward deeper alignment: principles that might transcend horizons

And so I wonder, like, could, could we try to give, give AI, like, alignment rules that transcend transcend knowledge horizons.

So we could say, hey, your alignment rule is to be empathetic or to be compassionate. And do those rules then transcend our knowledge horizon so that when the AI wants to do something that is outside our knowledge horizon, it actually goes, ah, am I

being empathetic when I do this? Am I being compassionate when I do this? And so much like a parent is to a child, they might have tough love.

Hey, make Earth a utopia. But our rule was don't kill anyone but actually maybe sometimes you know the parent has to be mean to the children

sometimes so i don't know can we solve deeper alignment maybe to to sort of somehow uh be able to keep the alignment beyond our knowledge horizon anyway so that's that i've kind of ripped through

Conclusion and how to follow up

it pretty quick uh i'm drew steel i don't really use linkedin but if you want to chat to me on

linkedin there's my link and then um i made like a quick website like real real quick called core Coraline. You can scan that QR code.

Not really sure what to do with it at the moment, Coraline, but I just thought it might be useful to have a group of people that think that there might be some sort of value in aligning AI with humans. So yeah, hop on there if you want.

That's me.