Can an LLM Write the Rules That Train a Game Designer?
A language model writes the scoring rules that teach another AI to build game levels from a one-line story. With feedback, its rules matched hand-written ones.
In one paragraph
Researchers asked whether a language model can turn a one-line game story into the reward rules that teach a reinforcement learning agent to build matching dungeon levels. Rules written once almost never worked, scoring 0.053 at best. With a loop in which the model checks its rules, sees the levels the agent built and revises, the average rose to 0.420, close to 0.422 for a human-designed reward. Humans still did better on stories with several conditions, and the evidence comes from one small simulated game.
How can a language model design rewards for a game-level generator?
Some game tools use reinforcement learning, where an agent learns by trial and error, to build levels one tile at a time. The agent is steered by a reward function: code that adds or removes points after every change. Writing that code is slow work for experts.
The authors built a loop called PCGRLLM. GPT-4o receives a short story, such as a player who must meet a bat before taking a key to the door, and writes a reward function. In a self-alignment step, a random agent plays one episode and the model adjusts the reward's scale. An agent then trains on the reward for 50,000,000 steps and builds 16 × 16 dungeon levels. The model reviews those levels, writes one point of feedback and revises the rules, for 6 iterations.
Three prompting styles were compared: Chain-of-Thought, which edits the latest draft, and Tree- and Graph-of-Thought, which keep several drafts and can return to the best one. A path-finding check scored how well 30 generated levels matched each of 4 stories, over 5 random seeds.
Did the language model's rules work as well as a human's?
Rules written once, with no loop, almost never produced matching levels: average scores were between 0.010 and 0.053 on a scale where 1 is perfect. With the full loop, averages rose to between 0.281 and 0.420, and the best, with Graph-of-Thought, came close to the 0.422 of a human-designed reward.
The average hides a split. On stories with several conditions, such as meeting two different monsters, human rules scored 1.000 in one scenario against at most 0.144 for the loop. On spatial puzzles, where a treasure sits behind a locked door, the loop reached 0.873 against 0.007 for the human rules.
The quality of feedback mattered: specific notes about the generated levels reached 0.427 by the sixth iteration, against 0.398 for generic notes and 0.287 with no feedback. Letting the model grade its own levels instead of the path-finding check made results worse, with changes of -0.211 and -0.056 where the fixed check gave gains of +0.162 and +0.090.
Why does automatic reward design matter for game development?
Hand-written rewards are a bottleneck for procedural content generation, the practice of having software build game content. A loop that turns a designer's sentence into a working first draft of the rules could save expert time.
The design keeps the language model out of building levels directly. It only writes the rules, so a fast trained agent does the building and the reward code stays readable, which makes it easier to see what the agent was rewarded for.
A fair reading is that the self-grading result is the most transferable lesson: the model drafted useful rules but judged its own output poorly, so an independent check remained necessary. The authors suggest a division of labour in which the model drafts coarse rules, especially for spatial structure, and human experts tune the trade-offs between several conditions.
What are the limits of this study on LLM-designed game rewards?
Read before you quote it
The study has clear limits:
- One small game. All tests used one simulated two-dimensional tile game with four short story scenarios.
- Noisy results. Scores vary widely between runs (for example 0.420 ± 0.328) with 5 seeds per setting, and the differences between prompting styles were not statistically significant.
- An outside judge is still needed. The model could not reliably grade its own levels, so the loop depends on a hand-written evaluation check.
- Expensive iterations. Every iteration trains a new agent for 50,000,000 steps.
The study does not show that language models can replace human designers, since humans still did better on stories with several conditions. It also does not show that the generated levels are fun or that the approach works in real, commercial or three-dimensional games; no players were tested.
- It does not show that language models can replace human designers: humans still won on stories with several conditions.
- It does not show that the generated levels are fun or work in real, commercial or three-dimensional games; no players were tested.
- All tests took place in one simulated two-dimensional tile game with four short story scenarios.
- Results vary widely between runs (for example 0.420 ± 0.328), with five seeds per setting, and the differences between prompting styles were not statistically significant.
- The language model could not reliably judge its own levels, so the loop depends on a hand-written evaluation check.
- Every iteration trains a new agent for 50,000,000 steps, which makes the loop computationally expensive.
Takeaways
- saysRules written once barely worked, and with the feedback loop and Graph-of-Thought the average reached zero point four two, against zero point four two two for human-designed rules.
- saysHumans did better on stories with several conditions, and the language model did better on spatial key-and-door puzzles.
- saysWhen the model graded its own levels, results got worse, so the loop still needs an outside check.
- explainsA reward function is the scoring code that tells a learning agent which changes earn points, so it decides what kind of level the agent learns to build.
- interpretsA fair reading is that language models are useful drafters of game rules but poor judges of their own results.
Ask the paper
In PaperSays you can ask this paper anything, by text or by saying “Hey Aiden”. Three questions listeners asked:
Can GPT-4o design reward functions for game level generation?
With a feedback loop, its rules matched a human-designed reward on average in this study, though not on every kind of story. Hear the full answer in the app →
Why did the language model's own grading make results worse?
Its scores for its own levels were not objective or consistent, so they steered the search in the wrong direction. Ask in the app →
Where did hand-written rules still beat the language model?
On stories with several conditions to satisfy at once, such as meeting two specific monsters before reaching the key. Ask in the app →
Briefing reviewed by an editor before publication. The paper belongs to its authors; this page summarises it and links to the original. Paper licence: CC BY 4.0. Corrections: support@papersays.com.