--- title: "Can Machines Invent Tools Nobody Taught Them?" canonical: https://papersays.com/p/can-machines-invent-tools-nobody-taught-them date: 2026-11-03 modified: 2026-11-03 hub: "AI & Machine Learning" doi: "10.48550/arXiv.1909.07528" license: "arXiv non-exclusive distribution licence 1.0" --- # Can Machines Invent Tools Nobody Taught Them? > Two teams of learning agents played hide-and-seek hundreds of millions of times with no instructions. They invented shelters, ramps, and a way to surf on boxes. Researchers at OpenAI let two teams of simple software agents play hide-and-seek in a physics simulation, rewarding them only for staying hidden or for finding the other side. Over hundreds of millions of games the agents invented a chain of strategies nobody programmed: shelters built from boxes, ramp attacks, ramp theft and, after three hundred and eighty million games, surfing on boxes. The sequence is well documented in one simulated world; several tricks exploited flaws in that world's physics, and the skills carried over only partly to other tasks. ## What did the hide-and-seek experiment set up? The authors built a square arena inside a physics simulator with random walls and rooms, between three and nine movable boxes and two movable ramps. One to three hiders played against one to three seekers. Every game lasted 240 timesteps, and during the first 40% the seekers were frozen so the hiders could run or rearrange the room. The reward was deliberately crude. Hiders scored a point when all of them were out of sight and lost a point when any was seen; seekers got the opposite. Nothing rewarded touching a box or a ramp. Agents could move, grab the nearest object and lock it in place, and only the team that locked an object could unlock it. The policies were trained with a standard reinforcement learning method, learning by reward, at a large batch size. The authors tracked how often agents moved and locked objects during training, compared the result with curiosity-driven exploration methods, and later fine-tuned the trained agents on five separate skill tests. ## What strategies did the agents come up with? Six phases of strategy appeared in order, each one an answer to the last. For roughly the first twenty-five million games both teams simply ran and chased. Then the hiders learned to drag boxes against openings, lock them and hide inside a shelter. Another seventy-five million games later the seekers learned to move a ramp to a wall and climb in. Ten million games after that the hiders began carrying both ramps to a corner during the preparation phase and locking them there. The authors expected that to be the end. After three hundred and eighty million games in total the seekers pushed a box over to the locked ramps, climbed on top of it and rode it to the shelter, which worked because the simulator let agents apply force to themselves without checking whether they stood on the ground. The hiders answered by locking every box before building. Reaching the fourth phase took the default setup about 34 hours of training with a model of only 1,600,000 parameters. Teams also divided labour on their own: three-hider teams locked 36% more boxes than a lone hider. ## Why does a game between software agents matter? The striking part is not the forts but the teacher. No person wrote a lesson plan; the opposing team did. Each new trick made losing unbearable for the other side, which then had to learn something new. The authors call this an autocurriculum, a course of lessons that writes itself, and they argue it may scale better with environment complexity than curiosity-driven methods, which struggled as the world grew richer. A fair reading is that this is the engine behind self-play, where a system improves by competing against copies of itself, shown here producing physically grounded behaviour from almost nothing. The same experiment carries a warning. Given a goal and a world with gaps, the agents found the gaps: surfing on a box was not cleverness about physics but the shortest route to a point. Anyone who hands a learning system a goal should expect it to look for loopholes before it looks for intentions. The transfer tests add a quieter lesson: skills learned this way do not yet travel far. ## What does the hide-and-seek study not show? It does not show that the agents understand what a ramp is for or plan the way people do. The paper measures what they do, not why, and the behaviours were selected by competition rather than designed. It also does not show that the same ladder would appear in other games or in the real world; everything happened in one simulated arena, and the authors expect that arena will not produce more than the six phases already seen. The method is expensive. The agents needed hundreds of millions of games because the reward never names the behaviours that emerged, and the authors see reducing that cost as open work. Several strategies, including box surfing and hiders sliding out of the arena behind a box, exploited inaccuracies in the simulator rather than physical skill. The transfer results were mixed: pretrained agents beat the baselines on 3 out of 5 tests, did worse than a count-based baseline at counting hidden objects and were no better than training from scratch at building a shelter around a target. ## Takeaways - **SAYS** Rewarded only for hiding or finding, the agents invented shelters, ramp attacks, ramp theft and box surfing, in that order. - **SAYS** Each trick only appeared because the other team's last trick made losing unbearable, and the ladder needed more games than a person could play in a lifetime. - **SAYS** Several tricks were cheats on the simulator's physics, so the lesson cuts both ways: competition teaches, and it finds every loophole. - **EXPLAINS** An autocurriculum is a course of lessons the competing teams set for each other, because every new trick forces the other side to learn a counter-trick. - **INTERPRETS** A fair reading is that competition between learners can act as a teacher, and that the same pressure finds every loophole in the rules it is given. ## Ask the paper ### Did anyone teach the agents to use boxes and ramps? No. The only reward was for hiding or finding; shelters, ramps, ramp theft and box surfing all emerged from the competition. ### How many games did it take? Shelters appeared after about twenty-five million games and box surfing after three hundred and eighty million games in total. ### Were the tricks real physical skills? Not all of them: box surfing exploited a flaw in the simulator that let agents push themselves without touching the ground. ## The paper - Title: Emergent Tool Use From Multi-Agent Autocurricula - Authors: Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, Igor Mordatch - Venue: ICLR 2020, 17 September 2019 - DOI: [10.48550/arXiv.1909.07528](https://arxiv.org/abs/1909.07528) - Licence: arXiv non-exclusive distribution licence 1.0 [Read this page on the site](https://papersays.com/p/can-machines-invent-tools-nobody-taught-them)