The poison well

Stone of Gurranq! Flame, Fall Upon Them*

Belladonna: lumen inter tenebras by Evaboneva

In the art world, Ben Zhao needs no introduction. He’s the man running the team behind Glaze and Nightshade, the tools of choice for artists who prefer not to feed their creativity and talent to the maws of the generative AI data scrapers. Officially, he’s the Neubauer Professor of Computer Science and the Director of Graduate Studies in Computer Science at the University of Chicago. Unofficially, he’s a deeply passionate technologist with his heart beating fiercely in the human world.

I spoke to Ben about his work, IP and copyright, the persistent misconceptions about the lives of artists, how generative AI is damaging the way creators earn their living, and the role of friction in the system.

Enjoy! And remember to take your magic potion tonight.

---

Birgitte Rasine: Ben it’s great to talk to you again and catch up. Congrats on launching Nightshade! First thing I’m curious about, how many people are using it—and Glaze—now?

Ben Zhao: 2.2 million Glaze downloads last I checked, and Nightshade is a quarter million after five days.¹ But you know, we don’t keep a running count. I imagine it’s more than that now.

BR: Do you feel there’s anything missing in mainstream media coverage of your work?

BZ: There’s been a lot of media coverage [of our work—see here, here, and here]. I don’t know that there’s anything super obvious that’s been missed but there’s still a prevailing misconception of who artists are and what their daily lives are like. That’s probably the biggest thing that fuels a lot of the opposition to artists, this idea that these are elitist snobs sitting in an art gallery drinking wine and raking in the money. Boy these artists are whiny! Trying to keep the masses from unleashing their creativity.

BR: Yes that’s the big question that’s gotten thrown around a fair bit in the past year—various iterations on the why—why should we protect the work that artists do. To us creators it’s fairly obvious.

BZ: Yeah that is often times the view in the tech sector, but it comes from a lot of misunderstanding of real artistry is and what artists’ lives are like. Over the last 14-15 months, [my team] has interacted with literally thousands of artists. The large majority of artists, even the successful ones, don’t earn that much money. These are all middle class, working class people. Most them I would say are freelancers. They don’t have job security. They work on a contract basis. Some of them live on commissions alone, and so there’s zero sense of stability; any month can be wildly up or down.

So the sense of this entitled elitist group sitting around in art galleries is quite the opposite [of reality]. In fact, even art galleries have a tough time. And when artists deal with art galleries, it’s extremely expensive; art galleries take something like 50% of the profits from the sale. Many artists are already teetering on the edge of being able to support themselves and their families. So disruption like [generative AI] really is an existential threat. How many artists can survive this disruption? You’re not only reducing the amount of commissions people pay individual artists, but also at the corporate level, how many companies are downsizing entire creative departments because budgets are tight and, you know, “we’d like to increase our profit margins. So we’re gonna work with something that looks a little less authentic, a little more glossy and plastic and that’s okay, because profits matter right now.”

So when that happens, the layoffs, the downsizing is just massive. A new report came out recently interviewing different companies, managers and people who control hiring in all these creative fields and they expect something like 260,000 jobs will be disrupted in the next couple of years. It’s just astounding. In California alone, they’re saying 60,000 jobs in entertainment, and then gaming and then illustrations, other sectors, just massive amount of job loss.

BR: Those numbers are huge. But, what they don’t talk about—and should—is the knock-on impacts. When that many people lose their jobs, that affects restaurants, shops, services, and even rentals where they normally spend their money. So, Mr. or Ms CEO, yeah you’ll fatten up your profit margin, but sooner or later it’ll come back and bite you.

BZ: Absolutely. That’s the thing. So much of AI right now is built on hype, and hopes and dreams. There’s the assumption that AI is going to get all fixed in a year, in monumental leaps. They talk about this magical ‘democratization of creativity,’ but nobody’s getting hired. All the companies that are downsizing because they can [do various tasks] with AI, they’re all talking about this increase in productivity, but that’s just a fancy term for ‘your co-workers are gonna get fired, and you’re gonna be doing the work of five people instead of one.’ So all this talk [about democratizing creativity] is questionable and it’s wrecking the entire industry, destroying livelihoods. It’s NOT creating jobs.

I hear this from a fair number of people in these industries who are optimistic and they’re like, ‘I’m gonna be able to work 20% of the time and get paid like I worked 100% of the time.’ Of course, as we all know, this is a capitalist society. You’re not going to reap the benefit and pass it up the chain. Your friends are going to get fired and you’re going to be asked to do entire teams’ worth of work instead of your single job.

And that’s what’s happening. A lot of artists are reporting exactly that—that entire teams are being laid off, and [the people left] are being asked to use AI tools. [The managers are asking], ‘Why can’t you ramp up your productivity by 5x?’ Well, because the AI tools are crap and in many cases, artists have to deconstruct them, then use them to create their own sort of authentic art—and that actually encourages more work, not less. So yeah, there’s just so much misunderstanding and hype.

No getting past this professor! Photo courtesy of Ben Zhao

BR: It’s crazy they’re trying to sell us tools that are not only trained on artists’ work without consent, credit, or compensation, that are not only taking jobs away, but also, insult on top of injury, we’re supposed to pay for them, and they create more managerial work just to get to a result that’s not as good as what the artist would have produced on their own.

So let’s talk about protection. In the cybersecurity business, as you know, there’s the famous cycle of one guy introducing a problem so they can sell a solution for it. If I introduce malware, I’m going to create a market for antivirus software. Now it’s at this fever pitch where hackers and security people are racing each other to the bottom (or top, depending on how you look at it). Do you see anything like this happening in the art sector?

BZ: It’s hard to predict that far down the road because we have nothing in that arena right now. I mean, literally, right now, in IP protection there’s copyright, which is under attack in the public discourse with a lot of people saying, ‘Yeah, copyright’s dead, who needs copyright.’ But if you step away from that discussion, and you look at what’s reality and where the battlefield stands, if you will, there’s nothing on the other side. AI model trainers can train whatever they want. They can download whatever they want. [Now] tools generate prompts from these images and feed them into the pipeline.

What’s on the other side? There are pending legal cases that will take years to play out—and then you’ll have appeals. There are legislation efforts that are hamstrung by companies’ lobbyists and countries’ leadership, not just in the U.S. but across the world, countries that are extremely fearful of losing tax revenue by these supposed booms in AI; so they go soft on AI regulation. Who knows when real legislation will come into the U.S.? Not surprisingly, the EU is moving forward and they’re doing great things. But even that was challenging. Until this week, I think Germany was still on the fence; they could have blocked the AI legislation. Now it looks like they’re going to move forward.

So who knows how long that’s all gonna take? Basically, there is nothing protecting creators right now. There’s nothing stopping companies from ripping off their images and sound and games and characters. In fact, some of that’s happening in real time today.

BR: It’s not a battle if one side has no weapons. It’s a massacre.

BZ: Yeah. Sooner or later people are going to realize this is a one-sided game right now. And all the power is on the side of the model trainers and the AI tech companies. So before we can talk about this balancing act, there has to be something on the other side first, and the only thing that we have so far is Nightshade. Opt-out is the only other option, but it’s not a real option. It’s a great carrot for these model training companies to throw out there. You see them dragged in front of the Senate judiciary hearings on Capitol Hill, and when they get grilled about the ethics of [what they’re doing], they always fall back on opt-out. They say, ‘Oh, we have opt-out lists.’

But how does that work exactly? The artists put their names into a hat, and hope that the companies are going to opt against their own basic for-profit directive, and that out of the goodness of their hearts they’ll respect the wishes of these creatives who have absolutely no leverage. There’s no validation, there’s no enforcement. You can’t check to see if anyone has violated opt-out. Opt-outs are an empty promise, a hope and a dream that a company is going to [sacrifice] their profit margins and say, ‘Yeah, we’re going to intentionally and willingly downgrade our models, and take on the complexity and cost it takes to implement something that filters people out. We’re gonna do all of that just so we can be nice to you even though there’s absolutely no way that anyone can actually verify that we’ve respected or violated opt-out because, you know, it’s a paper list.’

Right now, the technology is not there for us to look at a model and be able to say whether it got trained on a particular image. We just can’t prove it.

BR: Opt-outs are pretty disingenuous, in my humble opinion. There’s a term bubbling up now: AI washing. Like greenwashing but related to artificial intelligence development or usage.

BZ: Yeah and even in the most positive and optimistic terms, opt-out still only involves the companies that have reputational risk—the companies that get dragged in front of the Senate. All the smaller companies, what do they care? They don’t have any risks. They have no need to offer any sort of opt-out. They just take whatever they want, and all’s well.

So that’s the playing field that we have today. Everything on the side of the companies, and the other side, the creators’ side, has empty promises and a hope and a dream. And so that’s where we find ourselves. Hopefully Nightshade can tip the balance just a teeny bit back.

BR: I’m seeing others trying to do something similar. There’s a startup called kin.art for example, that took a bit of a shot at you guys, suggesting you don’t go far enough and their approach is better. How would you characterize what they’re trying to do?

BZ: I can’t say that I fully understand what they’re doing, because it’s all private. All I’ve seen is a couple of superficial press releases. Until there’s some demonstration of real technical work, it’s very difficult to say anything. It sounds like they’re going to make it harder for companies to take the images and associate the right labels with them—[this is called] fuzzy labels.

The problem is that this is not what the companies care about—they don’t care about labels or alt tags. They just strip it all away. They use tools like Blip to automatically generate the prompts. Blip is a classifier that basically looks at images and tells you what’s inside—to the best of its ability. So if you want to give an image a fuzzy label that says, here’s a picture of a dog but it attaches the word cat to it instead. Okay, great. But the [image scraping] companies will just say, ‘Yeah, this website does this weird thing with the labels. We’ll just take all their images, strip out all the tags and run everything through Blip. Done.’

The other thing they mention is image segmentation, which is unfortunate because image segmentation is a classical term in computer vision that’s been around for decades. In computer vision, image segmentation just means taking an image and identifying the exact outline that allows you to extract a particular component of the image. I’m gonna guess that what Kin.art is doing is chopping an image into pieces and displaying an amalgam of those pieces on the screen. So visually that looks to you and me like a single image, but in terms of actual files it might be split into multiple pieces. Okay, so that might be an approach.

Except, again, the problem here is that this is a single web site. They’re not a moving target. So if I’m a company who wants to scrape their content, how hard would it be to say, ‘I know this is what they’re doing, and I’m going to run these images through a post processor that reconstructs the whole images from the partial images.’ Again, I’m inferring here. They could have interesting techniques, very complicated and very robust things that I don’t know about; this is entirely possible. I don’t want to put down anyone else’s approach or make statements about what they can and cannot do. I’m just saying these seem like very basic things. If it is that basic, then that’s not great. If they’re doing more complicated, smarter things, awesome.

BR: What struck me in their press release is the tone they took with your work, when you guys are the pioneers. So far all the coverage I’ve seen basically regurgitates the press release without digging deeper or asking tougher questions.

BZ: Which is really amusing—in their press release, they said Glaze and Nightshade work on the image after it’s been scraped. Boy, there’s a reason why we do that! I’ve been doing research in Internet security for more than 20 years. I’ve written scrapers and I’ve written scrape defenses. I know what that battlefield looks like. It’s a lost cause for a company, for a single website to say ‘We’re gonna protect your images from scrapers,’ when the scrapers are Microsoft and Google. Are you kidding me? Google has more IP addresses than small countries. They are running their own parallel version of the Internet that’s bigger than the physical Internet right now. And as a little website, you’re gonna say, ‘Yeah, we’re gonna stop all their bots.’ I mean... it’s good to be optimistic and idealistic. It’s just that I know exactly how these battles are fought and I can name 16 different strategies of how Google will easily get through any sort of anti-scraping technology.

So again, I wish them the best. I always find it counterproductive for people who are supposed to be on the same side to criticize each other. If their [approach] works, fantastic. It’s yet another thing to protect artists. I would still question their monetary business model. But having done what I’ve done for 20 plus years and knowing exactly how all these systems work, I wish them the best. I wish them good luck. It’s going to be a tough, tough fight.

BR: The other difference is that Glaze and Nightshade are free, whereas kin.art is charging a small fee for their platform.

BZ: You know, the one thing that is clear, I think… [sighs] I think this is not the time to be monetizing the sector. I think that a VC group that’s putting down $3 million to fund a company that says, ‘Let’s monetize off the sort of destruction that’s happening to the art community. Let’s monetize off the fears of these independent artists who are potentially losing their homes or not able to pay rent because they’ve lost their jobs,’ that’s just not a great business model. How do you convince people who are desperate for money to pony up money for commissions so that you can make returns on your VC investment? That part doesn’t quite work out for me and I think that’s just a misunderstanding of the art community and where people are right now.

BR: Your team and the kin.art team are all engineers, clearly. But there’s a marked difference in your approaches, and especially all of the advocacy you’ve done personally Ben, speaking on panels with Karla Ortiz and Jon Lam, and traveling and talking to artists’ organizations. Much respect.

BZ: It’s human empathy, right? Can you put yourself in someone else’s shoes—and if you can, you can actually be bothered to learn about how other people are living, and change your mind. But a lot of people don’t. It’s a lot easier not to.

And there’s so little profit margin. I see some of the numbers from the artists we talk to, and I’m like, God it’s brutal! You drive across the country to attend this convention, you sell this much stuff, you pay for the booth, the hotel, the food, and per hour, you end up maybe $10 an hour—IF they have a successful show. It’s just brutal.

So from day one, I said, we’re not monetizing this [Glaze and Nightshade]. It’s a stupid thing to do. These are people that need help—and you can’t get anybody to trust you if you’re trying to make a buck off them.

BR: I can attest to that personally, having organized—and funded—all of my daughter’s art shows and exhibits. Very tough to make a living this way unless you’re a big name. That’s why the unauthorized mass scraping of creative work has been so devastating for all these artists. So now that it’s been out in the world for a while, I mean Glaze, and now Nightshade, how is the broader world responding?

BZ: I’m active in the research community and I get to see a lot of papers under review. I will say there’s a lot more interest now in poisoning attacks. I’m seeing a lot of papers on poisoning and attacking large language models. I think lost somewhere in all this, are really good ideas and people who know much more than me in the space and who can probably do much more. So I’m hopeful that there are effective techniques out there that, once published, people will actually use and, as I said before, balance the playing field.

It’s funny, I was in Sweden in late January speaking to the Swedish Association of Artists, this big national organization, and one of my hosts had this very interesting take on things. He said, ‘I love this work because all systems need friction.’ In his view, Nightshade is adding friction to the system to try to balance things. That’s a very intellectual and healthy way to look at it. You can have a struggle and you can have a back and forth, but not unless the two sides both have something to work with. If one side has nothing, it’s really just a beatdown. So that’s what we’re trying to do, is attain that friction, that balance.

---

If you’d like to support the work Ben and his team are doing, get in touch. And be sure to stay subscribed to The Muse for more dives down the rabbit holes of the creative lives we humans lead.

* If you know where these incantations come from, you are a true wizard. Or witch.

1. As of February 1, 2024.