The Case for Training Smaller, Specialized AI Models

Okay, so the basic idea is that you know that the amount of people who, the count of people, I'm not the best in English, who climbed Mount Everest is higher than the number who created their own one billion parameter language models from scratch.

And our ambition is that the math is like, this is like absolutely crazy image that you have to put into your brain is that the more compute you put into the model the better results you have the more the larger data set you have the better results the more parameters as well better results but it has a very nice catch is that a model which is twice the size cost for many times to us to train it's the quadratic relationship between one of the other So the model which is 10 times smaller will be 100 times cheaper to train.

And this life hack allows us to train models that because okay this model is 100 times cheaper but it doesn't mean it is 100 times worse. In many cases in a small model specialized in some specific use case can be as good or even better than the frontier model at this task. task. And I think Jeff proved that.

From Base Models to Practical Applications

So, what we do, we do base models. That is like basic model which you can fit into your use case.

We do a lot of experiments. And then we do task models which, for example, OCR, decision models, and for example, we have Pollock.

Yeah, David. Yeah, Polak. Yeah, you know. Hi.

Teaching More People to Train LLMs

We've created a training program for people to learn for them how to train their own LLMs.

And this guy, he just came and he went, he got, I think it was sixth place on the leaderboard of small LLMs at first approach.

So it's like, what I'm saying to you guys is that don't take OpenAI or anthropic propaganda that there is no way to win with them. There is a way to win with them because, you know, quoting the Ender's Game, the enemy gate is down and they bleed. They are humans just like us.

And there is like no secret to doing open AI, it's like literally it's like this. And yeah, there are some hidden receptors and hidden things. things, but recently DeepSeq just gave the Recyp how to serve tokens 100 times cheaper, like maybe 1 ,000 times cheaper than OpenAI, and then we'll have tokens too cheap to matter.

And it's very nice being in the place which is like a libertarian place, because the thing is that the capitalism delivers what the communists promises. And it means the almost unlimited consumer surplus. It means that the tokens will be cheap and AI will be available and in many use cases we will actually have this thing.

Scaling Up Through Competition and Investment

Okay, so we are doing the training run. We are doing the scaling from the smaller models like 8 million parameters, 16, 22. We are just like here, almost here, we have trained our half billion parameters model model and each steps cost us for many times as much.

And being absolutely great and showing that we are winning the previous competitions allows us to get more compute and get more VC capital and so on. So it's literally like Formula 3, Formula 2, Formula 1. You start as a go -kart and then if you are best at the small scale, you have sponsors to give you to get you to the next level.

An Invitation to Build and Share Models

And I have actually like there is a thing called Hugging Face. Go there. Who of you have a Hugging Face account? Hands up. Yeah, those real people.

And who of you have your own model there? Yeah, I see one guy. You see?

And my point is that in the next few months I want to see at least half a room raising their hands on Do you have your own model there?

Because we've been called Slayer previously, but now it's Fabryka .ai.

Open Models, Data, and Training Recipes

We have models that are open. You can not only get the models, which is of course like base level. You can also get a recipe of how it was done. I mean, like we are sharing almost everything. Like you say, see this like description of how it's done. And also what's the data set it was trained on. And guess what? But the data set used for that is also public.

So you can just plug your agent into this showing like, okay, I want to make models better than those guys and compete with me. Like I'm literally asking you to compete with me because I want to have more people training this.

I see that we have a lot of, this is like small part of what we are actually doing, but we make a lot of research. research.

Fabryka AI and the Polish AI Ecosystem

And we have something called Fabryka AI. Let's switch to English. We build models, then put them to work.

And we recently partnered with like greatest Polish model developer, Remigiusz Chinas on Bazal Omni that we are like official API inference provider for him. so if you want to track his model and actually use it in your business go to us because we have his blessings to like offer this api and you see that say okay something is still polish so like

like we have like kind of ethno nationality vibe in here that we want to make this models in poland polish so we promote polish creators of polish models and i want you on this list and And let me help you become a model creator. That's my push for you.

And also, I have a presentation on how actually this model was created, but I just translate it in on the fly almost. So let me, how many time I have? You've still got about seven minutes. OK, seven minutes.

Wait. Maybe I will use the Polish one, because I and I will just say it in English, never mind.

How Language Models Are Trained

So how to come from the data sets to model is that you start with the data and then you have tokenization.

And the first thing that we teach in our courses is that you understand deeply how it works and why it works. And it's like very technical stuff.

Pre-Training, Continued Training, and Fine-Tuning

Then you have thing called pre -training, which is training model to learn this whole data set and the pre -trained model is just like a piece of clay like which you can then shape into specific purpose and then you have continuous pre -training so you can take

somebody else model like continue pre -training and giving him other facts because there is one thing called like fine tuning and using fine tuning you can teach the mother specific specific behaviors, like steer it a little, but if your model needs some specific knowledge, it usually has to get this fact like 20 times during the training. So this is like very costly part, but it's required.

Learning by Building

And we are doing this because we believe that actually getting your hands dirty, actually doing the development, actually training those models teaches you so much more details on how it works. works.

The work is no longer a black box for you after you join this community, join this course or join the AI community of people who are implementing archived papers for fun. This is a very important part. We are doing it for fun out of the love for the craft. Yeah.

So unfortunately, ah, white agents are still working on this. Uh, okay. Open it.

And usually this funny thing is that a lot of people don't even, don't even use the computer. Many people are using the terminals all the time, like it was here, sources, it doesn't work, never mind. Let me use the Polish versions, sorry for that.

The Technical Foundations of Language Models

Okay, so basic thing is that the rain is falling or something. think and today is raining and like either snow rain or like hail and the

model on basic level is just it just multiplication of large matrices with some nonlinear function which is like activation function and I don't want to get too technical but yeah hey if you want unlimited amount of like meat like Like coming there, I will think afterwards to give you the unlimited level of technical insight of how it works.

And we have to read 27 billion tokens to read it. And it's a lot. It's a very large number. And this number is kind of small because it's just 140 million parameters, roughly.

And also we have nice setups. everybody has their own free minutes, right?

Managing AI Agents as Digital Workers

So if you have agents, we have agents that are running 24 -7 on our compute, so you can have as many workers as you want, but it's your job to manage them. And managing the workers is actually kind of harder than actually doing the job.

So be careful what you wish for, because I know so many people that, oh, I don't want to do the the job okay now you get 100 people under you to do the job and trust me this job is that much harder than actually doing the job that you will be like willing to actually go into the literal minds to avoid managing people and like in this case like it's like very hard to teach uh it's very hard to be a manager and it's much easier to just like do the work and it's like psychology

because now AI forces you to become managers you are no longer people who are like doing the work you have your skill is a management of a agent so we

Open Datasets for Polish Language AI

we are very open about what comes into the or data set you can replicate our results you we have created something called polish the network which is over still growing data set of like data set of open data which can be used for pre -training so you don't have to spend money on lawyers or like software

engineers to make data set for you you just take this and train on this because we spend so much money on like getting it compliant and we will be making a paper on that so and it's kind of popular you see like number of like

like 14 ,000 people downloaded it last month. And I think it was about 200 most popular data set for pre -training in the world.

Ambitions for Fabryka AI

So our ambitions are much bigger than that. I think we can become a largest, if you see organizations.

One minute, right? So let me show the ambition of us. So you see organizations.

So that's why I'm asking you to follow the Slayer Lab for Fabryka .ai on Hugging Face. Because in order to be on the first page of organizations, I just need 4 ,000 followers about.

So I think I will get this, like, I will force you to download Hugging Face, make account, and follow me. Because it gives us ability.

And I worked at Apple. And I have, like, personal vendetta against them. And I'm like, I will become better than you at AI Models. You will see.

I just need about 8 ,000 followers. Yeah?

Free Access to Models

Would you give free access to your model? MARIUSZ GASIEWSKI - Yeah, I have free access. We give free access to our model. We are using the VC Monday to buy compute and give away a lot of this for free.