Memes, AI & Benjamin Bloom - Jonathan Libov - NY

So, one thing you'll notice is that there's no AI fight in the hero section of our website. It's going to take a minute for me to get to the AI demo. It won't be obvious at first. First, I'm going to give you an introduction to antimatter.
We are a platform for unlocking mastery in today's students and learners, we have students learners right formal learners and formal education settings, getting formal education, and we give them tools activities experiences.
that better resemble the tools, activities, and experiences you all have as informal learners on the internet. And for us, that means memes, puzzles, and games.
Now, if you're asking yourself, I don't know, I don't see memes in my best group chat. Well, you know, may not be in sort of like the best group chat because I think we've all sort of experienced how much memes really synthesize.
out learning, which is maybe not true just across group chats, but across Twitter, et cetera, et cetera. That's kind of where we're going to start is around memes for learning. Not using memes to teach students, but to have students make memes for what teachers call formative assessment. Because you think about it, it's really hard to tell a good story, create a good puzzle, tell a good joke.
if you don't really understand the subject matter. Now, we offer teachers classroom activities for learning, right? You know, illustration that memes are themselves puzzles. I'm gonna give you sort of a brief sort of preview of what anti-matter looks like. This is an activity we call meme quiz.
I've invited myself in a different browser to make a meme. And so this topic here is chemical bonding. I'm being asked to make a meme without using the prompt in the meme. And I promise we're going to get to the AI stuff shortly. So I think my prompt is, sorry, covalent bonding. And does anyone here remember whether it's covalent bonding refers to metallic atoms or nonmetallic atoms?
Anybody? Well, I do this demo a lot, so I know that it's non-metallic atoms. And here's my meme about covalent bonding. I'm going to finish that, submit it, and here as the teacher, I can start the review.
And now you have a quiz about chemical bonding, right? This refers to covalent bonding. Correct. So these are our sort of classroom activities. We have a number of activities. Another one is called heat map. Get to make memes on a map. Like, you know, our sort of mantra, you know, for antimatter is like what a time to be a student or learner. We build tools and activities that we would have wanted as students when we were in school. One of the things that's really neat about antimatter
is that, this is the heat map activity actually. You can zoom in all around the map. The end of the week, we select memes of the week. So students go nuts for this because they feel like they get selected for like a national competition where probably the only ed tech platform or what happens in the classroom doesn't stay in the classroom. We publish all the memes and celebrate them across the community, which points to sort of where we're going.
OK, this is the AI. This is called Sorcerer. And Sorcerer is a personal assessment tool. So these are memes made by students in a classroom in a unit about the colonization of the Carolinas. So let's look at one meme created by a student.
Hold on, I lost the image. And maybe we have a, looks like we have a bug. Hold on, there it is. Okay. So, anybody know anything about 17th and 18th century Carolinas? Well, that's right. I'm actually going to skip to something that is made for Mindstone.
So this is a question that I spun up this source for I think last night according to the seven leadership traits of Romani Thomas, COO of Mindstone. Let me just venture a guess what those traits are. Oops, crap, hold on. I'm being cheeky because obviously I'm not doing these off the top of my head. All right, these are the seven leadership traits
One of the things you'll notice is that one through six are accurate. Seven is not quite so accurate. This is how I sort of behave as CEO, but maybe it's just not a good leadership trait. And so let's talk about what sourcerer is.
Okay, it begins with sort of a visual prompt and usually a puzzle. In this case, I just included a fun little image from the blog post. But as you saw in the colonization of the Carolinas, right, it starts with a meme. It's designed to be encouraging and to ask questions, not to give answers, right? So ask me a question here. Here is my response.
And it even gives me insights, as well as the teacher who can see this, into how I did. And I got a score out of 60 because I got six out of seven word for word. Probably would have got 100 if I had just nailed all of them. You'll see that it even tells me that the seventh trait is incorrect.
Well, you saw it. And then you can see here, right, it's designed to be encouraging. You've done well to list six to seven traits. However, the seventh is not correct. And then if I'm not mistaken, seventh, have honest conversations with your team. Let's see how my response is. If this is wrong, I have it open in the next tab.
Good try. You can see it's designed to be encouraging. It's also very different from your experience with ChatGBT, where you ask ChatGBT questions. I speak to students all the time who say, yes, I use it. It feels like cheating.
Sorcerer is designed to be an AI tool for the classroom that feels like learning, right? Like basically, almost a, not quite zero marginal cost, but a zero marginal cost teacher's assistant for that classroom, then eventually for groups of learners like AP exams or first year law student, et cetera, et cetera. This product is about six weeks old.
You can, one of the things that before I sort of open it up for questions is worth noting because this is a question that I frequently get is, well, does the AI understand the means, right? This is intended to be something of a technical talk.
Actually, I have two topics covered. This is one of them. So we did a study on this. This is my personal blog on how well open AI can interpret memes. We had sort of a seminal moment during the presentation of ChatGPT4 in which, I'm sorry, I forget his name, but during the presentation he had
He took a picture of, or he presented a picture of a squirrel holding a camera and fed it to Chachiputi and said, you know, why is, and said, why is this funny? And Chachiputi said, well, squirrels don't normally hold human instruments, right? The correct answer should have been, that's not funny, right? If you saw a picture of a squirrel holding a camera, nobody in here would laugh, in contrast to some of the memes that you see created by antimatter students, right? This is a meme about natural selection,
Made by an antimatter student. This is a meme made about the plum pudding model of the atom. This is a meme about Roman architecture.
and a meme about the proclamation of 1763. And, you know, you saw this meme earlier, by the way, this is what, you know, this is what Dali 2 generated as a meme for natural selection. Not so good.
This is a meme generated by ChachiB4. It looks more like a cartoon. Put it this way, if it was that the HSS Beagle had landed here and Darwin might have been a little bit late in discovering evolution had he landed on this island.
And so we did a study of how well OpenAI interprets memes. This was not created on anti-matter. This is created on r slash history memes. And OpenAI or chapter GBT gives sort of almost a perfect answer about why this is funny. But one of the things you'll notice is that if you took away the imagery from this meme,
it would make just as much sense as it does with the imagery. The imagery is almost like just decoration. So let's look at memes that have sort of increasingly prominent visual narratives. This is one that was made on anti-matter. And CHPG does pretty well in determining that leaving contemporary issues is a bit darker than entering the contemporary issues unit for this classroom.
Here's where it starts to get interesting. This is the meeting that we saw earlier about the colonization of the Carolinas.
And Chatjibiti completely fails to make sense of this. It does not recognize that this is intended to represent Native Americans. It doesn't recognize the state of South Carolina. And then as we progress to memes which have zero visual narrative at all, I don't know if you've ever seen this meme. This is intended to represent the double slit experiment. This is, you know, an image that is widely available on the, all these images are widely available on the internet, but there is no textual content in this meme.
ChatGPT thinks these are doors. It makes no sense of it at all. Which speaks to, you know, why we deal with learning memes in the first place, is that they're your sort of uniquely human creations. They are puzzles. I would go as far as to say is I think even pre-AI, the reason that memes have become so prominent on the internet and in the classroom, is there something of a response to the otherwise
legibility of the internet, right? Like you can feel it. You know that you type in something about the Knicks on Facebook, you're probably going to start getting ads from fanatics about Knicks, Jerseys, et cetera, et cetera, right? Memes are sort of escaping that kind of legibility. I think it's even possible that in ChatGPT 5 and 6, as they get better at understanding visual narratives,
Memes may even evolve to escape the legibility of open AI, which is kind of an uplifting message that we're not all going to be sort of conquered by machines. Some of us will. Maybe not all of us. The good memers won't. The last thing, I guess, I think something I overlooked in presenting Sorcerer is that there is, you know,
Just feeding, you know, feeding OpenAI a question about the seven leadership traits according to Mindstone COO, reminded Thomas or
This topic, which is the Spain and Britain clash over the Carolinas, it will give you a response. It will not always be terribly thoughtful. That is called zero-shot prompting. We do much, much more than that.
we ask the machine to think. So this is a TypeScript file that represents the prompt that we send to ChatGBT, forcing it to sort of think about the response, thinking about the response, and then to think about the question and to think about thinking about the question in a nutshell, which is why we're able to make it encouraging,
differentiated according to your own mastery level, right? One of the things that's notable is that if I were to go, not only will it respond differently if I give sort of responses that demonstrate mastery or lack of mastery, we can also tune it to difficulty level.
So I can set this to high school or middle school. But again, the most important thing is that it's differentiated by learner and mastery level. This is not to sound grandiose.
believe this might even make a significant dent in Bloom's two sigma problem, which is the idea that, I mean, we know as fact that students who receive both sort of direct broadcast instruction and one-on-one tutoring perform orders of magnitude better than students who don't, right? And what we are aiming for with Sorcerer is to give students that kind of one-on-one tutoring that is differentiated for their mastery level.
And that is anti-matter and sorcerer.

Finished reading?