Can Machines Invent Tools Nobody Taught Them?
Two teams of learning agents played hide-and-seek hundreds of millions of times with no instructions. They invented shelters, ramps, and a way to surf on boxes.
In one paragraph
Researchers at OpenAI let two teams of simple software agents play hide-and-seek in a physics simulation, rewarding them only for staying hidden or for finding the other side. Over hundreds of millions of games the agents invented a chain of strategies nobody programmed: shelters built from boxes, ramp attacks, ramp theft and, after three hundred and eighty million games, surfing on boxes. The sequence is well documented in one simulated world; several tricks exploited flaws in that world's physics, and the skills carried over only partly to other tasks.
What did the hide-and-seek experiment set up?
The authors built a square arena inside a physics simulator with random walls and rooms, between three and nine movable boxes and two movable ramps. One to three hiders played against one to three seekers. Every game lasted 240 timesteps, and during the first 40% the seekers were frozen so the hiders could run or rearrange the room.
The reward was deliberately crude. Hiders scored a point when all of them were out of sight and lost a point when any was seen; seekers got the opposite. Nothing rewarded touching a box or a ramp. Agents could move, grab the nearest object and lock it in place, and only the team that locked an object could unlock it.
The policies were trained with a standard reinforcement learning method, learning by reward, at a large batch size. The authors tracked how often agents moved and locked objects during training, compared the result with curiosity-driven exploration methods, and later fine-tuned the trained agents on five separate skill tests.
What strategies did the agents come up with?
Six phases of strategy appeared in order, each one an answer to the last. For roughly the first twenty-five million games both teams simply ran and chased. Then the hiders learned to drag boxes against openings, lock them and hide inside a shelter. Another seventy-five million games later the seekers learned to move a ramp to a wall and climb in. Ten million games after that the hiders began carrying both ramps to a corner during the preparation phase and locking them there.
The authors expected that to be the end. After three hundred and eighty million games in total the seekers pushed a box over to the locked ramps, climbed on top of it and rode it to the shelter, which worked because the simulator let agents apply force to themselves without checking whether they stood on the ground. The hiders answered by locking every box before building.
Reaching the fourth phase took the default setup about 34 hours of training with a model of only 1,600,000 parameters. Teams also divided labour on their own: three-hider teams locked 36% more boxes than a lone hider.
Why does a game between software agents matter?
The striking part is not the forts but the teacher. No person wrote a lesson plan; the opposing team did. Each new trick made losing unbearable for the other side, which then had to learn something new. The authors call this an autocurriculum, a course of lessons that writes itself, and they argue it may scale better with environment complexity than curiosity-driven methods, which struggled as the world grew richer.
A fair reading is that this is the engine behind self-play, where a system improves by competing against copies of itself, shown here producing physically grounded behaviour from almost nothing. The same experiment carries a warning. Given a goal and a world with gaps, the agents found the gaps: surfing on a box was not cleverness about physics but the shortest route to a point. Anyone who hands a learning system a goal should expect it to look for loopholes before it looks for intentions.
The transfer tests add a quieter lesson: skills learned this way do not yet travel far.
What does the hide-and-seek study not show?
Read before you quote it
It does not show that the agents understand what a ramp is for or plan the way people do. The paper measures what they do, not why, and the behaviours were selected by competition rather than designed. It also does not show that the same ladder would appear in other games or in the real world; everything happened in one simulated arena, and the authors expect that arena will not produce more than the six phases already seen.
The method is expensive. The agents needed hundreds of millions of games because the reward never names the behaviours that emerged, and the authors see reducing that cost as open work. Several strategies, including box surfing and hiders sliding out of the arena behind a box, exploited inaccuracies in the simulator rather than physical skill.
The transfer results were mixed: pretrained agents beat the baselines on 3 out of 5 tests, did worse than a count-based baseline at counting hidden objects and were no better than training from scratch at building a shelter around a target.
- It does not show that these agents understand tools or plan the way people do; the behaviours were selected by competition, and the paper measures what they do, not why.
- It does not show the same ladder of strategies would emerge in other games or in the real world; everything happened in one simulated environment.
- The agents needed an enormous amount of experience, hundreds of millions of games, because the reward never names the behaviours that emerged.
- The strategy space of this environment is bounded; the authors expect it will not go beyond the six phases as it stands.
- Several strategies exploit flaws in the simulator (surfing on boxes without touching the ground, leaving the arena behind a box shield), not physical skill.
- Transfer results were mixed: worse than the count-based baseline on Object Counting and no better than scratch on Shelter Construction.
Takeaways
- saysRewarded only for hiding or finding, the agents invented shelters, ramp attacks, ramp theft and box surfing, in that order.
- saysEach trick only appeared because the other team's last trick made losing unbearable, and the ladder needed more games than a person could play in a lifetime.
- saysSeveral tricks were cheats on the simulator's physics, so the lesson cuts both ways: competition teaches, and it finds every loophole.
- explainsAn autocurriculum is a course of lessons the competing teams set for each other, because every new trick forces the other side to learn a counter-trick.
- interpretsA fair reading is that competition between learners can act as a teacher, and that the same pressure finds every loophole in the rules it is given.
Ask the paper
In PaperSays you can ask this paper anything, by text or by saying “Hey Aiden”. Three questions listeners asked:
Did anyone teach the agents to use boxes and ramps?
No. The only reward was for hiding or finding; shelters, ramps, ramp theft and box surfing all emerged from the competition. Hear the full answer in the app →
How many games did it take?
Shelters appeared after about twenty-five million games and box surfing after three hundred and eighty million games in total. Ask in the app →
Were the tricks real physical skills?
Not all of them: box surfing exploited a flaw in the simulator that let agents push themselves without touching the ground. Ask in the app →
Briefing reviewed by an editor before publication. The paper belongs to its authors; this page summarises it and links to the original. Reviewed by Editor. Paper licence: arXiv non-exclusive distribution licence 1.0. Corrections: support@papersays.com.