Understanding the multimodal model CLIP, which judges whether an image matches a text. While learning how it scores images, we look at the biases of today’s AI models towards images and text.
It began with our own question: why does this particular image come out of a generator? Looking for an answer led us to CLIP, and to tools such as CLIP Interrogator, which guesses the prompt behind an image.
CLIP puts images and texts in the same space and gives a single score for how well they match. Many image generators and image search tools use it to decide whether a picture fits a description, a step their users rarely see.
Matching
Participants give CLIP an image and a few short texts and see which one it finds most fitting: Magritte’s painting of a pipe against “this is a pipe” and “this isn’t a pipe”; a photo of a cat against “cat”, “Katze” and “gato”; an AI-generated face against four emojis with different expressions. The scores are hardly ever neutral, and the surprises are what the exercise is about.
Searching
Then CLIP goes looking. A loop keeps pulling random faces from a website that generates people who do not exist, scores each one against a word such as “happy”, and keeps the highest and the lowest so far. Swap the word for “nurse”, “programmer” or “dumplings”, and the machine’s stereotype of each word slowly appears. Food is just as strange: in one round for “happy”, a plate of soup with a sausage scored 20.48, higher than a golden schnitzel at 19.38.
Sessions
2025 · Generating for the Archive, Storage Museum Düsseldorf
6 January 2026 · KISDtalk, KISD Köln International School of Design, Cologne
7 May 2026 · mtrail Ausbildungscamp, Lindau
Workshop and lecture requests: please get in touch by email.
