Know your audience: six games, six shapes, one custom evaluation
Six games, measured for the shape of their text. Not the quality. The shape.
Call of Duty, Hades, Borderlands 2, Sekiro, Disco Elysium, Syberia. How much of each is UI, how much is description, how much is dialogue. How stiff the writing is. And which language the studio actually wrote it in.
They are all different. In Call of Duty, 60% of the lines are short UI strings: names, buttons, menu items. In Disco Elysium, 96% of the lines are dialogue and narration. Sekiro looked like an outlier at first. Then it turned out the studio wrote it in Japanese and the English is itself a translation. Measured against Japanese, the dot moved to the right place on the map. The map caught a wrong assumption before any real work started.
This is also what studios and LLM providers care about right now, even if nobody says it this way. A translation memory cannot be placed well until the map is known. How many memories, where to split them, which lines may share one. Get this wrong and the styles mix. A menu string starts to sound like a subtitle, a subtitle starts to sound like a tooltip, and nobody can point to the line where it went wrong. The map is what keeps the styles apart.
For a translation pipeline built as a graph of LLM calls, the first question is not which model or which prompt. The first question is: what does this client's text look like? Where does it sit on the map? With that answered, the nodes and the memories have a place to go.
The two ends of the map need completely different pipelines
On the Call of Duty end, most lines are names, buttons and menu items. This is not translation in the usual sense. It is closer to careful transcription plus metadata: what kind of string is this, where does it appear, which tags and placeholders must survive. A long chain of LLM calls does not help here. One well-shaped call with the right metadata does. What matters most on this end is consistency. The string next to this one, in the same menu or the same item family, has already made the choice. The best thing the call can do is peek at its neighbors before it writes anything.
Every studio also has its own placeholder dialect, visible just by opening the text files.
Hades keeps its variables in curly braces like {$Keywords.Cast}. Borderlands 2 uses
angle-bracket aliases like <StringAliasMap:GFx_Cancel>. Sekiro puts controller
icons in tags like <?kgiconKc@17?>. Each one is a different way of saying do
not touch this, and each target language has its own way of living next to it. None of this is
glamorous, and a lot of the real work is here.
On the Disco Elysium or Syberia end, almost everything is dialogue and narration. Here the metadata matters much less. What matters is the relationship between lines, who is speaking to whom, and the register of the whole scene. This is where breaking a sentence down, looking up the right domain, and checking the result actually pays off. The same steps that helped there made the UI text worse.
In the middle, descriptions and tooltips need a third approach: get the sentence form right, one sentence in and one sentence out, and stop there.
So a UI call and a subtitle call are two different calls. They carry different metadata, they need different checks, and they fail in different ways. One pipeline for both gets tuned for one and quietly breaks the other.
The map is where a custom evaluation starts
Most clients cannot say, off the top of their head, how much of their text is short UI, how much is description, and how much is dialogue. Fewer still can say where their current translation sits in stiffness or register. Put the dot on the map and both questions get an answer in one picture: this is where you are, and this is the direction a style change or an accuracy push would move you. Every request after that, a new register, a tighter glossary, a different tone for one franchise, becomes a move on the map instead of a feeling.
And once that data exists, it does not stay on a slide. The same numbers can go straight into the graph: as a node that reads the client's shape before anything else runs, or as the condition that decides which state gets loaded for this line. A UI-heavy client routes to the transcription path with its metadata. A dialogue-heavy client routes to the relationship path. Nobody has to remember which client is which. This is the point where the automation actually begins, at the design stage of the LLM system, not after it.
The generic metrics do not see this
COMET and MetricX are fine for ordinary translation. But the better a studio's translation is, the more of its own voice it carries, and the further it drifts from what a generic metric expects. In two of the three games where this was measured, the human-written golden scored worse on the generic metric than a plain machine draft. Open those lines and the golden is clearly the better text. The metric is not wrong about translation in general. It is wrong about this house.
So the real job of a localization architect is to keep building an evaluation that reflects the client's own quality, not the average of the internet. That is where all the effort goes now. It is the thing everyone in this field has wanted for a long time: to say this translation is fine as a number, and have that number mean the same thing the client means. Nobody has fully done it yet, because it cannot be done in general. It can only be done per house, and only after the house has been measured.
The same map helps with a brand-new title. When there is no golden yet, the client can point at the neighbors on the map and say like that one, but warmer, and that sentence becomes a target: a register, a punctuation habit, a share of dialogue, all with numbers attached. The style guide stops being a document people argue about and becomes a benchmark the pipeline can be scored against.
And the map is not only for studios that already have a translation history. It works just as well for a first title, or for a studio that has never measured anything. Once a few hundred games are profiled this way, a request like make it feel like GTA V stops being a vibe. It becomes a set of numbers: this much dialogue, this register, this punctuation habit, this way of handling placeholders. The client can see exactly what that request means before a single line is written, and can say yes or no to it.
This is why an LLM architect in localization is really a consultant. The value is not the graph itself. The value is reading a client's data first, putting a dot on the map, deciding which nodes, which state and which memories that client actually needs, and then building the custom evaluation that says whether it worked, in the client's own terms. After a few projects that judgment becomes know-how, and it is different for every studio.