What are LLMs and Generative AI Good At?
Image by ChatGPT with the prompt “generate an image of "The Thinker" statue but recursive”
Large language models and Generative AI have monopolized the conversation in tech for a few years. They dominate programmer forums and pull all the investment money.
I’m neutral on this tech. It’s fine. There are tasks it is good at and tasks it is not good at. I’ve got no product to sell you here. After mulling it over for a few years I do have thoughts to share. Here’s what I think Generative AI is good at:
Accessibility Layer
Fuzzy in, fuzzy out
Averaging Engine
Accessibility Layer
LLMs, specifically, are a terrific accessibility layer. They take natural language and transform it into something resembling understanding and intent. LLM technology has improved search engines. LLM technology can turn natural language into actions (agentic workflows). Generative AI can turn text into pictures. It’s wild. Using natural language as an interface into all of the magic that computers can do is a fun use of the technology. But all of this magic comes with a caveat…
Fuzzy in, Fuzzy out
LLMs and generative AI are not deterministic. They are probabilistic. They accept tokens in the form of natural language, context text, images, and other media and transform them into an output that is likely to be in the right direction. They are transformation tools. Sometimes you can use what it gives you to great effect. Sometimes teasing out exactly what you want is tough.
When I worked at Rec Room, I led a team applying generative AI tools to gameplay. One of the cool experiments that we put together was a game where you can make your own monster. I created a workflow where you provided 4 adjectives and generative ai would produce a 3d model of a monster that would then chase you around a maze. It felt like magic. But sometimes people didn’t love their monster and wanted to tweak it. If you wanted to, say, change the monster’s color that would generally work. If you wanted to add limbs or change the hairstyle that was less likely to work. One person asked the workflow to “make the monster more musical” and it did literally nothing. There was not much we could do about the effectiveness of editing. The generative AI systems are sometimes just a slot machine. You pay for them to do something and hope for the best. There are ways to improve the quality of the output, but when the input can be literally anything it is impossible to test it all and guarantee a great result. Ultimately you are rolling dice. Frustratingly, you pay whether the output is good or not. If fuzzy input -> fuzzy output is fine, generative AI can be a great choice. If you need a deterministic or reliable output you might want to reach for a different tool.
Averaging Engine
LLMs and Generative AI are transformation tools. You feed in tokens. These tokens pass through dozens of layers of transformation based on the training data that was applied to create the model. Out the other side you get a result. The transformations are the effect of the average of all the training data. Average might not be exactly semantically correct, but it’s correct enough.
You may have noticed that LLM-generated text has a recognizable cadence. You may have noticed that AI-generated cartoon images have “a look.” These are the results of the averaging of the model’s training data.
You may have also noticed that AI-generated images can get a little loopy in fine details. Sometimes mouths have too many teeth. Sometimes a necklace weaves into a subject’s shirt. I follow a few forums where someone posts asking “is this AI?” and all the little details are scrutinized. It’s interesting to see the noise and artifacts that come out of the averaging. You may have heard the term “AI hallucination.” I view these as the result of noise and artifacts of averaging.
Average results can be useful. They are rarely great. Generative AI makes average results a commodity.
