Introduction

So actually, it's interesting.

Himanshu's presentation was all about security kind of on a per developer basis.

I'm going to talk a little bit about security more at an organizational scale.

So we didn't plan it this way, but this ended up being a really kind of nice micro to macro security view.

Why Packaging Determines Security and Speed

Specifically, I'm going to talk about speed and security, and I'm going to speak about how you package and why the way you package your application may actually be, or your AI, may actually be the most important decision that you make in your AIML toolchain.

It's probably not the one you're spending the most time on, unfortunately.

So first, quickly, myself, I noticed some familiar faces from last time, but I'm the founder and CEO of a company called Jozu here in Toronto.

I'm also one of the maintainers.

We originated actually the KidOps open source project, which now has over 150,000 downloads.

I think just last month when I last spoke to you, it was only over 120, but we're growing fast.

It's my second startup.

You can connect with me on LinkedIn there.

I'm always happy to connect, happy to discuss people there.

From ad‑hoc builds to containers

All right, so as I kind of alluded to, project packaging is, in a lot of ways, kind of the hub of the wheel.

And it's something that I think, certainly for most developers, most engineers who are used to developing in an application world, not so much an AI world, it's something you've probably taken largely for granted, especially if you've been developing for less time than I have.

oldie timey times there wasn't as much kind of standardized packaging and you would have to get your jar files and your gem files and your wheel files and your make files and your tars and you'd have to figure out ways to kind of package them all together and send them out to machines and oh man if it's running on Windows that's going to be different it's running on Linux which is different if it's running on a different flavor of Linux which is different it's running on Mac it was a huge

Pain in the butt.

Then we came along with containers, and everything got a lot simpler.

What people tend to focus on with the container is, oh, man, it made my job as a developer or as a DevOps person much easier.

Isn't that awesome?

Security gains from standardization and immutability

What I think gets missed sometimes is that the actual security posture and security level of applications rose dramatically after containers came into existence and use.

And the reason for that is fairly simple, that if I'm building a security analysis tool, and I need to build it to work with jar files, and I need to work with it to build the gems, that's totally different.

And then if I'm going to add tar files to it, again, totally different.

If I need to make it work across different platforms, huge amounts of effort.

So security companies would tend to focus and do it for one specific area.

Once containers came along, it became fairly trivial for me to build and maintain a set of security tools and approaches that would work largely for every application and every target.

And that just meant that it was easier for people to get access to these tools, to use them every day, to make them work quickly and easily, and therefore everything gets better.

The second huge thing that containers brought was immutability.

Once you create a container, it is immutable.

It doesn't change.

You can't just go and tweak it.

You have to tweak the recipe and then rebuild a new container from it.

And that is a hugely important differentiation because it means that if I get hold of a very specific container, a built container from you or you or you,

and I run it, it's gonna run exactly the same as it did for anybody else because it didn't change.

It can't have changed.

Even if I move it through a very unsafe pipeline,

Nobody can actually tamper with it because it would change what's called the Shaw digest, and that Shaw digest can't really be fooled, can't be tricked, can't be made an imposter.

And so I would see right away, wait a second, this has a different digest than when she gave it to me.

This is not okay.

I'm not going to use this.

It's obviously been mucked with.

Again, huge leap in security since so much of security depends on seeing things which are meant to be hidden.

All right, so I went a little long there.

Apologies, I get excited about this stuff.

AI/ML Packaging Today: Gaps and Risks

But it's a way of saying that the way you package something has a profound impact on everything else you do with that thing.

1The other thing about packaging is that it impacts how you version it, how you secure it, how you govern it, how you make collaboration work, how you audit it.

So this is all a long introduction to say don't build on a shaky foundation.

And I say that with urgency because in AIML,

there has not been a standard packaging format the way there is with containers for applications.

And so you tend to have very fragmented tools.

If you look at how MLflow packages something, it's different than how Kubeflow packages something.

It's different than how Weights and Biases package something versus Domino versus DataRobot versus et cetera, et cetera, et cetera.

That gives gaps for security attacks, for security compromises.

And it doesn't have to be attacks.

There was a question earlier about if I have a firewall, am I not safe?

And of course, you're safer.

But people make mistakes.

We're human.

And some of the worst security breaches I saw at AWS, for example, when I was running API Gateway, were not attacks, even though we were under attack literally all the time.

It was from people making well-intentioned mistakes internally.

Most of your security systems are probably looking out.

Many of your issues are gonna come from looking in.

I read a recent report, something like 68% of AI vulnerabilities are from internal error, not external attack.

So the majority of what you need to worry about is behind, already inside your doors.

What counts as an AI artifact

So let's look quickly at how people store AIML artifacts today.

I think one of the other tricky things to remember is your AIML model is not the only artifact that matters.

Your model, not only does it need weights, it needs training data, it needs validation data, it probably needs embedding code, it needs prompts if it's generative AI, system prompts, base prompts, template prompts.

All of these things can determine and change how the model is going to behave.

And it may behave better or worse if you change any one of those things, if you tamper with any one of those things, whether you mean to or not.

Common storage patterns and their pitfalls

By far the most common place that people store these things is in S3 or a file system, object store, whatever it is.

And it's because it's scalable, it's super simple, and it's cheap.

Unfortunately, it has versioning, which depends on you naming something typically.

Dataset for training underscore final.

to be for real this time.

So it's not maybe the most robust versioning.

You don't get a lot of metadata.

It just kind of is when it arrived, how big it is.

That's not metadata you can really use in most cases to make deployment operational security determinations.

Oh, and it uses MD5 hashes, which, by the way,

You can spoof an MD5 hash with a laptop and about $100.

So definitely not really as secure as people would like you to believe.

Git and Git LFS, or large file system, way better, of course, because it uses a SHA digest.

But your storage costs get really high if you start dumping big models in there.

And it is still not really tamper-proof.

It is still possible to tamper with things despite that.

DBC uses MD5, so again, not tamper-proof, not terribly secure, but it's super ML-specific, it's very easy, lots of people love it.

It doesn't mean that these things are bad, it means that you have to be conscious about how you're using it, right?

Experiment trackers, MLflow, weights and bytes, lots of people think of this as the, oh, this is where it all kind of comes together.

This is my package.

This is where I'm gonna hand off from.

But again, not only do you have vendor lock-in, you typically get MD5, because they're typically using S3 or object store under the hood.

Plus, they can be brutally complicated for non-data science teams.

If you have a DevOps engineer that just cares about which model is ready for production, not obvious at all in general from your experiment tracker.

Reusing What Works: Registries as a Single Source of Truth

So one of the secrets of security and speed, how to make things faster in a big organization, this is something I definitely saw at Red Hat and at Amazon even more so, is reuse.

Don't reinvent a wheel.

Use the wheel you already have, especially if it works.

So interestingly, the container registry is something that people already think of as a single source of truth, not typically for their AIML work, but for pretty much everything else that goes out to production.

If you want to know where is the stuff that goes to production, check your container registry.

It'll be there.

It's battle tested.

It's got RBAC, signing, security.

It's got global distribution, caching, dedupe.

Compliance and auditing can be done there.

It's got integrations with pretty much every tool under the planet.

A better question: can AI artifacts live beside images?

Three-ish years ago now, when myself and my co-founders sat down, we were playing with a lot of AIML at the time, and we were pulling our hair out about the number of different packaging formats there were.

And it just honestly felt like we'd gone back to doing DevOps work in 2010.

We were like, why?

Why do we have to go back in time?

So this question we asked was, wait, can we store ML and AI artifacts

beside container images in the registry that we already have, because that would make a lot of things easier if we could do that.

And of course, our first thought was, well, let's just jam them into a container.

And yeah, you can, but that's not really what containers are for.

It's a massive misuse of them.

It doesn't work out very well.

You end up with things that are in a technically runnable container that are never meant to be run, which just kind of is a weird model to try and force people to understand.

OCI Artifacts 101

Luckily, we have OCI artifacts.

How many people have heard of OCI?

OK, let me explain.

So when I say container, how many people have heard of containers?

All right, much better.

OK, so all of you who have heard of containers have heard of OCI.

You just didn't know about it.

So OCI is called the Open Container Initiative.

It's actually the specification that defines how a container behaves.

So when we say container, what we're really saying, if we want to be nerdy, is OCI image.

Images vs. Artifacts: recipe vs. pantry

And so the OCI specification, though, has a whole secondary section for OCI artifacts.

Images are for running.

They're like your recipe, right?

If you look at a Docker file, it's do this, then do this, then do this, then do this, then do this.

End up with your container, end up with your pie.

I happen to love pie, so I'm gonna use a pie as an example.

But just like a recipe, you can't just go and do step one of the recipe, and then the last step, and then the third step, and then the seventh step, and expect that your pie is going to turn out nice and delicious.

It's not.

It's going to be a big old mess.

So you've got to go sequentially, and you've got to do every step.

You can't just skip them randomly if you want to end up with the same thing at the end.

OCI artifacts are different.

OCI artifacts are more like your kitchen pantry.

They've got all the stuff you might want to use there, all the ingredients for your pie and your stew and your whatever else you like to eat.

But by itself, it's not a recipe.

You can look at the pantry and you might have ideas, but you're not going to end up being able to make a pie just by staring at your pantry.

So the way an RCI artifact works is it's more like a set of layers.

But each layer is distinct.

They're not sequential.

They're not meant to be run.

It's just a holding tank.

So you have everything you need you could put in there.

Layering models, data, code, and prompts

And that was our big realization.

We said, wow.

So you could have a layer for models.

You could have a layer for data sets.

You could have a layer for code.

You could have a layer for prompts.

And then everything would be in one place.

And then you could take that package and move it wherever you wanted.

That's more like a shipping container

but not one you run, just one you kind of store things in.

Introducing KitOps

And so that's why we created KitOps.

And we did it to create kind of secure and honestly kind of friendlier packaging for AIML projects that was open source, openly governed, and vendor neutral.

What a Model Kit contains

So when you create a KitOps model kit, as they're called, it is an OCR artifact.

And just like I said, it contains a layer for models, data sets, code, configuration, prompts, and documentation.

And we don't care what kind of model it is.

It'll work for LLMs, computer vision, predictive ML, anything you want.

You don't even have to have a model in it.

If you just want to store your data sets in it, awesome.

Have at it.

Or only your prompts.

Selective pulls and modular reuse

One of the cool side effects of using an OCI artifact for this, of using a model kit, is that you can put everything that everybody might need into this so that anybody in your team or company can reproduce this project, but it doesn't force everybody to take everything when they go to use it.

So if I'm validating the model, I may grab only the validation data set in the model

If I'm only working on the prompts, I don't need to grab any of that hundreds of gigs of stuff.

I just grab the few K of the prompts or I push back the few K of the prompts.

I don't need everything else.

It's not important to me.

So it's highly modular from that perspective.

Security properties out of the box

The nice thing too is that this gives you essentially out of the box

Awesome security.

You've got SHA-256 verification, which means you have that tamper-proof mutability.

If anybody goes to try and change a model kit, you'll know.

The tool will actually know.

It'll tell you.

It won't let you use it.

But the point is that it can't be fooled.

You can sign it so that, you know, you have DocuSign, like you have the same thing for binaries.

You can sign the binary and say, hey, Brad created this on this date, and this is proof that it really was Brad, not somebody pretending to be Brad.

And so now if there's a problem with it, you know, okay, well, I'm going to go bug Brad about it because, look, his signature's on this thing.

He signed off.

If it's no good, that's his fault, and it would be my fault.

You have RBAC.

You have audit trails, you can kind of get everything you need from that.

1Turning your container registry into a model registry

So kit ops was really about turning your container registry that is already so much the center of what most development teams do into a secure model registry as well.

And one that would work with any of the tools, not just one particular vendors.

Walkthrough and Demo

So I'm going to quickly walk through this, and then I'm happy, obviously, afterwards to answer more questions, because some of this I know is a bit technical.

Defining the kit and versioning components

But just to give you an idea of kind of what this looks like, we have our overall package.

I can version that.

I have an author there.

I can give it a description, a bunch of metadata about what is this whole project about?

What am I doing with it?

Then I can drop in a model.

This is my churn predictor.

It happens to be using scikit-learn.

There's a path to my particular model itself.

And I can version that model independently of the whole package.

So this could be package version 1, but maybe I've been iterating on the model a lot, which is pretty common, pretty likely.

So my model may be up to version 2.1, even though the actual project version, as it moves forward out of a milestone, is version 1, or version 3, or version 17.

Doesn't matter.

Datasets, immutability, and auditability

I can add as many data sets as I want.

Here's my training data set.

Now, the nice thing about doing this is remember this whole package is immutable.

So especially if I'm in a regulated industry, health care, government, financial,

When I take the training data that was used to train my model and put it into a model kit, it means that even 10 years later, if an auditor comes by and says, prove to me there's no PII, prove to me you adhere to HIPAA regulations in this data set and in this model, you just hand them off this thing and you go, there it is.

This is exactly the data in exactly the state it was in 10 years ago when we trained this model, or 10 months ago, or 10 weeks ago.

It doesn't really matter.

You can't do that very easily if it's in a data lake or if it's on a file system, because guess what happens in a file system?

Some well-meaning intern will come by and go, gee, we're almost out of space on this file system.

I'm just going to delete some of these old data sets that nobody's used in three years.

And there goes your audit trail.

There goes your compliance proof.

And that makes people very, very unhappy later on when they find out.

Including code and provenance

And then you drop in your code.

We're not suggesting you don't use Git, by the way.

Your code, continue to use Git.

We love Git.

We use Git.

But again, if you need to prove the provenance or the lineage of your model, then adding the code base at the moment it existed when that model was trained or built is a really good way to make compliance happy at a later date.

this can also by the way be just a link or a pointer to a particular git commit if you'd rather handle it that way all right now let's try and do a live demo and see if i can make it all blow up can you read that is that large enough make it a little bit bigger all right uh okay so

Packaging the project

First of all, this is what my directory looks like right now.

You can see I've got a data subdirectory with my white wine quality.

I enjoy wine.

Data set, I've got docs, which include my licenses and my README.

The kit file, which we talked about before, that's kind of the manifest or the listing of what's inside the model kit.

I've got my MLflow artifacts.

I used MLflow to train this.

So I've got things like the metrics and a plot showing kind of the accuracy.

I've got my MLflow run Python.

I've got my model as a pickle file.

I've got my requirements text and a couple of Jupyter notebooks.

So you can see I've got kind of everything that I've used, every artifact as I've kind of developed this model is here.

So now,

Let's take a look at what the kit file looks like.

And I'm not going to go through this in massive detail, because we just looked at it.

But it'll give you a sense of what's there.

So you can see I've got description.

I've got an author, all the model information.

Oh, framework is interesting.

So you'll see framework here, XGBoost.

That framework is really important metadata that you can embed in this because now your DevOps person, if they just even look at the manifest for this model kit, they can know, oh, OK.

So it's going to need this type of container in order to run it in production because it's an XGBoost.

If it was an LLM, it would need something different.

If it was computer vision, it would need something different.

So that can be a great way to either automate through a pipeline or even manually just give people hints of this is how you need to actually run this.

This is what you'll need to do that.

You can see I've got my hyperparameters in here, my metric scores, the specific run info so people can always go back to MLflow and look at the details if they don't get what they need here.

All my code, my data sets, and my docs as I described.

OK, so all my stuff's here.

I'm a happy cat.

Now what I'm going to do is use kit pack.

So now that's going to say go to that kit file, go grab all those separate artifacts, and pack them into that OCR artifact, that model kit.

And you can see I've called it wine quality and tagged it with today's date because that's a nice way to do that.

And there you go.

So we can see we've got, what is that, four or five, like maybe a dozen layers, no, ten layers maybe, in there with all those different artifacts.

Configuration is saved, the manifest is saved, and the model is saved.

You'll notice that every single thing has its own SHA digest.

So not only have I meant that there is a kind of proof of...

Proof of tamperability for the package itself, but every single thing inside has its own as well.

So I can verify that nothing inside has been tampered with the same way I can make sure that the whole thing hasn't been tampered with.

Listing and pushing to a registry

Now, Kit List is going to show me all the model kits that I have on my machine right now.

Because this was a demo, I think I probably, yeah, I only have one.

Normally, I would have a lot more.

I wanted to make it a little bit clearer for everybody, so I got rid of the old ones.

But you can see there's my repository, wine quality, my tag, my maintainer, the name, the size, and that digest.

So kind of at a glance, I get some good metadata.

Now I'm gonna push it.

Now the push has two parts.

I'm gonna basically say take this thing, which was right here on my local machine, and push it to my registry.

In this case, I'm using, of course, my company has a hosted registry designed for these.

I'm gonna use that because, of course, why not?

Now we're gonna push it.

I didn't think about the fact that we're on probably not the world's fastest wifi, but okay, cool.

That worked anyway.

So now it pushed and it sends that confirmation SHA so that I know exactly what it did.

Inspecting, pulling, and deploying

And now let's flip over to this and I should see, yes, there is my,

One quality.

If I look at the contents, I see all that information from the kit file just laid out in a way I can read it a little bit better.

All my ML flow info, my data set, my code bases, my docs.

Kind of cool thing you can do here, of course, I can pull this to my local machine if I was somebody else.

If I only want the model, I just click this and I get the terminal command I need to just pull the model.

That's kind of cool.

But the thing I find even more interesting is deploy.

So if I was going to run this locally,

It actually auto-generates the container that I need.

So I would just copy this to my clipboard, drop it into my terminal, and I'm going to have my project running on my laptop.

I don't need to understand what Docker does.

I just need to know how to control C, control V, which I'm going to guess everybody in here knows how to do that.

Now, if I want to deploy to Kubernetes, it's a little more complicated, obviously.

In this case, you get a little snippet of the Kubernetes manifest.

This is obviously not enough to just deploy to Kubernetes.

You need more here.

But this is all your DevOps person would need to know, OK, this is the bit that I don't understand as a DevOps person.

Now I know all the other stuff that I'm going to insert this into.

So they can generally take it kind of from there.

um we also run security analyses of these that takes a little bit more time you can do diffs with different versions so you can see the changes i've made since march 5th quite a lot and we get the model card i don't know oh it does have a model card yay yay me um anyway so that gives you some idea of kind of the different information you can get from from this stuff uh okay i showed this we'll skip past that especially in the interest of time and move along here

Automating with SDKs and Policies

Python SDK and CI/CD templates

You can automate the creation of these.

This is kind of important.

You can use something called PyKitOps, which is our Python SDK.

So for example, if I want to do this in MLflow, this is what we did.

Drop in some credentials.

We have a model kit manager.

I log in, I initialize the kit file, and then I pack and push just like you saw me do on the CLI, but now this is all in Python.

Nice thing is I can template this into the code base if I have like an IDP, an internal developer portal, and I have golden paths from my developers.

This can all just be embedded automatically, so they don't even need to think about this.

It just happens, and then that's nice for everybody.

Policies, scans, and attestations

Josu, my company, whoops, sorry, we add policies and securities around this.

So the model kit is that kind of immutable, secure container or that package.

Then we do all the things around it.

And there are some of these actually related to what Himanshu was talking about earlier.

Not only do we expose that model lineage and provenance really well, not only do we allow you to do all the

all the kind of visualization of that and reading the metadata nicely.

But we also run, we have about five, I think, I forget if it's five or six at this point, different security engines that we run against models.

And so we look for different types of vulnerabilities in the models.

And then we provide a report that says, this is vulnerable, it's not vulnerable.

That covers not just the model itself, but the code and the data sets as well.

So you can look for data set poisoning, for example.

We have policy gates, so you can define policies and say, for example, if this model kit is not signed, it can't be deployed to production, but it can be deployed to dev.

If it doesn't have an attestation that proves that the security scan was run and that the security scan was clean, it can't go to dev, but it can go to pre-production.

You can have human in the loop approvals.

Brad has to provide his manual stamp of approval or it doesn't go to production.

And then we have full auditing and compliance.

Deployment topologies and integrations

Even better, because we work a lot with regulated industries, Jozu and KitOps, of course, itself, can be run in an air-gapped environment, on-premises, self-managed, or in the cloud.

What else have I not talked about?

Oh, yeah, it's trivially easy to integrate with your CI-CD pipelines, your experiment trackers with Kubernetes, KSERV.

whatever it is you're using, because it uses that same format as the containers do.

Conclusion and Next Steps

All right, and that's it.

If you want to learn more about KitOps, you can scan that and do the tutorial yourself.

It's a little bit richer than obviously what I did here, but it's quite cool.

If you want to learn more about Jozu at jozu.com, scan the other one.