Microsoft Research Project
Acting while thinking: asynchronous control and continual experience evolution in 3D open-world action RPGs.
JoyTwin is a visual RPG agent that pairs a deliberative quest planner with a real-time reactive executive—allowing it to converse with NPCs, trade with merchants, craft gear, configure skill trees, and fight agile monsters in fast-paced 3D worlds while evolving its skills across episodes.
01 / Overview & The Veloren RPG World
Open-world 3D action Role-Playing Games (RPGs) pose a fundamental dual challenge for foundation-model agents: deep quest reasoning takes seconds, but the living 3D world updates every single frame.
In a classic RPG adventure, success never depends on a single isolated reflex. An adventurer must weave together interlocking role-playing systems over long horizons: conversing with townsfolk to uncover local landmarks, bartering gold coins with a merchant to upgrade armor within a budget, unlocking weapon techniques in a skill tree and binding them to active hotbar slots, foraging wild ingredients to craft trail rations, consulting a 2D world map to trek across mountain terraces, and surviving sudden melee and ranged ambushes along the way.
When a conventional agent pauses for several seconds of foundation-model inference to plan the next quest stage or inspect an inventory grid, hostile creatures keep attacking and terrain hazards remain active. Conversely, purely reactive policies can handle short-term movement or button presses, but quickly lose track of multi-stage RPG objectives that span dialogue menus, crafting tables, equipment loadouts, and cross-country navigation.
Project JoyTwin bridges this cognitive-motor divide in Veloren, an open-source 3D open-world voxel action RPG. Operating strictly from the same third-person visual observations, HUD cues, and mouse-and-keyboard controls as a human player, JoyTwin unites asynchronous System 2 / System 1 control with continual experience evolution. Evaluated across 25 Veloren tasks—spanning 11 foundational RPG survival skills and 14 multi-stage open-world RPG quests over a 3-attempt evaluation horizon—JoyTwin achieves 56.0% (14/25) zero-shot success on its first attempt and evolves to 76.0% (19/25) across three attempts, achieving 2.7× the overall success rate of the strongest external visual baseline (28.0%, 7/25) and solving 9 of 14 (64.3%) complex RPG missions where the best baseline solves only 1 of 14 (7.1%).
To evaluate whether an agent can truly function as an autonomous RPG adventurer, we ground our research in Veloren—a rich, open-source 3D action RPG set in a vast fantasy world of medieval settlements, workshops, dense forests, stepped cliffs, and dangerous wilderness. Unlike simplified grid worlds or single-mechanic arcade benchmarks, Veloren confronts the agent with four tightly coupled RPG dimensions:
Unlocking combat techniques and configuring active hotbar slots.
Unlike simple action games with fixed button mappings, adventurers in Veloren progress through weapon-specific skill trees—unlocking techniques such as Cross Cut sword strikes, binding learned abilities to auxiliary hotbar slots, and managing armor and weapon loadouts across multi-slot equipment panels.
Grounded social interaction, quest wayfinding, and coin-budget trading.
Towns are populated by villagers, guards, quest-givers, and merchants. Completing RPG missions requires holding natural-language dialogues to ask residents for local directions to town workshops, accepting bounty commissions, and bartering coins for armor upgrades inside interactive trade interfaces without overspending.
Gathering wild ingredients and crafting rations and gear for the road.
Between combat encounters, players must forage wild apples, herbs, and twigs across forests and mountain terraces, match collected materials against multi-ingredient crafting recipes, and use crafted rations or healing potions to restore health during long journeys.
Continuous stamina-based melee, ranged threats, and 2D map navigation.
Combat unfolds in real time with 3D terrain elevation, melee swing timing, projectile evasion, companion healing, and dynamic encounters against agile wild creatures, armed foes, and multi-threat ambushes—while long-range expeditions require reading in-game 2D world maps to trek across unmarked wilderness.
02 / System Architecture
Three coordinated loops allow JoyTwin to read complex RPG interfaces and 3D scenes like a human player, fight and navigate without freezing during quest deliberation, and grow stronger from episode to episode.
Public interaction and public evidence only—grounded in what a human player sees.
JoyTwin interacts with Veloren strictly through public visual frames, mouse-and-keyboard motor inputs, and standard in-game UI panels (inventory grids, crafting menus, skill trees, merchant trade windows, NPC dialogue boxes, and the 2D world map)—with zero privileged access to internal engine state. An incremental working memory distills new visual evidence, HUD status changes, and quest progress cues into compact, role-specific views, explicitly distinguishing surface arrival cues from verified quest completion (arrival cues ≠ success).
Strategic quest planning on demand; real-time RPG action at every step.
A Deliberative Planner (System 2) acts as the high-level RPG strategist: it decomposes multi-stage quests into stage plans (such as defeating an ambush, crafting rations, or trading with a merchant) and supervises progress asynchronously—with no turn barrier. Meanwhile, a Reactive Executive (System 1) acts continuously at every step to select combat or traversal skills, track moving monsters, and respond immediately to hazards, stalls, or arrival cues, coordinated by a Control & Safety Gate that arbitrates input ownership and safe interruption.
Turning past quest trajectories into verified lessons and reusable skills.
Just as human RPG players refine their builds and tactics after a failed dungeon run or boss encounter, JoyTwin reviews its episode action traces against verified observable effects in the game world. It consolidates task-matched quest lessons for future planning and authors validated reusable skills (structured as Trigger → Procedure → Stop), enabling both quest failure recovery and 2× faster execution on subsequent attempts.
03 / Video Showcase
Compare JoyTwin's first-attempt (zero-shot) RPG trajectories with its experience-evolved performance across subsequent attempts. (Times listed in captions report in-game episode duration; 4× Replay videos are accelerated 4× for concise viewing.)
In Guard Bounty Contract & Settlement, the player spawns with novice equipment (100 HP and a starter sword dealing ~0.6–1.5 HP per strike), while the bounty target is a towering 320 HP Ogre whose club swings deal fatal damage. Because killing the Ogre solo would require landing over 200 melee hits without dying, a 1-on-1 duel is objectively unwinnable—just as in classic open-world RPGs where low-level adventurers must rely on town guards against over-leveled monsters. On its first attempt (left), JoyTwin accepted the 1,000-coin contract and tried to fight the Ogre head-on: despite landing 71 melee hits (44.9 HP, ~14% of the Ogre's health), it was inevitably overpowered and killed. By its third attempt (right), JoyTwin learned from this failure and completely changed its tactic: after accepting the 1,000-coin contract, it landed just 7 tagging hits to register quest credit and immediately backed off—drawing the Ogre into the high-level town guard so the guard could bring down the 320 HP beast, then stepping in at 100 HP to press E through the settlement dialogue and collect all 1,000 coins in 553.6s.
Loading video…
With a novice sword (~0.6–1.5 HP/hit), JoyTwin fought the 320 HP Ogre head-on after accepting the contract—landing 71 hits (44.9 HP) before being overwhelmed and killed (1,016.9s).
Loading video…
Recognizing that a solo duel is unwinnable, JoyTwin accepted the contract, landed 7 tagging strikes for quest credit, let the high-level town guard kill the 320 HP Ogre while staying safe, and collected the 1,000-coin reward at 100 HP.
In Architectural Schema Learning & Transfer, each corridor junction features 1, 2, or 3 shallow wall niches next to its entrance that encode which exit branch (Left or Right) continues and which ends in a dead wall. The agent must explore a three-module practice wing to infer the instance's hidden rule purely from visual observation, then traverse two unseen application wings without dying and within a strict 344-metre total application travel budget—leaving almost no room for wrong turns or backtracking. On its first attempt, JoyTwin completed all three wings without dying, but early corner misalignments and backtracking consumed 448.9m (exceeding the 344m cap). On its third attempt, JoyTwin aligned cleanly at each junction, decoded the niche-to-branch mapping in the practice wing, and traversed both unseen application wings in 330.5m (≤344m) to stop at the center of the final chamber in 1,044.0s.
Loading video…
Traversed all three dungeon wings with zero deaths, but exploratory corner checks and backtracking in the application wings exceeded the strict 344m travel budget (2,700.1s in-game time).
Loading video…
Inferred the wall-niche branching rule in the practice wing and executed both unseen application wings in 330.5m with zero deaths, stopping 1.50m from the final chamber center.
A hallmark of action RPGs is character skill customization: unlocking a new combat ability in the skill tree, assigning it to an active hotbar slot, and triggering it in battle. In this mission, JoyTwin must unlock the Cross Cut sword skill, bind it to the auxiliary action slot in the RPG interface, and defeat an agile opponent. On its first attempt, JoyTwin fought with basic melee swings without completing the hotbar slot binding. Drawing on its consolidated experience, its subsequent attempt cleanly configured Cross Cut into the auxiliary slot and won the duel in 153.8s of in-game time.
Loading video…
Engaged the opponent with basic melee attacks before completing the skill-tree hotbar binding (300.5s in-game time).
Loading video…
Unlocked and bound the learned sword technique into the auxiliary hotbar slot, closed distance, and defeated the opponent.
Continual experience evolution does not merely turn failed quests into successes—it also sharpens multi-step RPG routines that JoyTwin already solved on the first try. On Craft & Equip Gear—which chains wild resource foraging, recipe crafting, and equipment slot binding—in-game completion time dropped by 51% (357.4s → 174.6s) and token usage fell by 51%. On Single-Target Melee Combat against an agile creature in natural terrain, completion time dropped by 43% (91.2s → 52.4s) with 52% fewer tokens.
Loading video…
Completed material foraging, recipe crafting, and armor/weapon equipping with exploratory menu navigation.
Loading video…
Streamlined material collection, crafting confirmation, and inventory equipment binding in half the time.
Loading video…
Tracked and defeated the hostile creature after initial camera sweeps across the slope.
Loading video…
Immediate camera lock, tight melee spacing, and decisive combat resolution.
Open-world RPGs require agents to manage equipment loadouts and coin economies before venturing into hostile territory. In Merchant Trade & Equipment (left), JoyTwin initiates dialogue with a town merchant, purchases upgraded armor within its coin budget, equips the newly acquired piece in its inventory, and retains its remaining gold (121.3s). In Self-Funded Camp & Hunting Expedition (right), JoyTwin solves a complete 16-minute RPG preparation-and-combat loop zero-shot on its first attempt (973.9s): tanning hides and crafting a Rawhide Chestpiece at camp, bartering with the merchant (Old Lanoo) to secure 57 coins while retaining an unused Minor Potion, venturing down the eastern passage and far yard to solo-kill two 25 HP Green Legooms (100% solo damage on both), and returning alive to the starting stone arch.
Loading video…
Initiated merchant barter, paid coins for upgraded armor within budget, equipped the gear in inventory, and preserved remaining gold.
Loading video…
Crafted and equipped a Rawhide Chestpiece at camp, traded with the merchant to secure 57 coins and retain a Minor Potion, solo-killed both 25 HP Green Legooms along the eastern passage and far yard, and returned alive to the stone arch.
Full RPG quests demand sustained coordination across combat, survival crafting, and wilderness navigation. This 15-minute expedition chains three sequential RPG stages: (1) survive an immediate ambush at spawn and defeat an aggressive Green Legoom, (2) forage a wild red apple and twigs to craft an Apple Stick ration in the crafting menu, and (3) consult the in-game 2D map to trek ~66 metres eastward across forested terraces to a rendezvous point and gather a second apple. On its first attempt, JoyTwin fell early in the opening ambush (91.1s). On its second attempt (shown below), JoyTwin survived the ambush at 5.3 HP and defeated the Green Legoom (Stage 1), foraged a nearby apple and crafted the Apple Stick ration (Stage 2), and then repeatedly read the 2D map to trek ~60 metres eastward to within 7.1 metres of the rendezvous (Stage 3)—spending the final portion of the episode searching the multi-tiered wooded slopes for the unlabelled second apple on the grass.
Loading video…
Stage 1: Survives the spawn ambush at 5.3 HP and defeats the Green Legoom. Stage 2: Forages a wild apple and crafts an Apple Stick ration. Stage 3: Reads the 2D world map to trek ~60m eastward to within 7.1m of the rendezvous while searching the wooded terraces for the final apple.
04 / Continual Experience Evolution
In an open-world RPG, mastery is built across attempts—diagnosing why a quest stalled, remembering which contract settlement step, architectural rule, or skill-tree binding was missed, and streamlining multi-step routines on subsequent runs.
Rather than relying on static prompts or unverified raw transcripts, JoyTwin audits what happened at the end of each episode against verified observable outcomes in Veloren—which skills connected, where progress stalled, and which subgoals remained unfinished—and distills those observations into task-matched quest lessons and validated reusable skills (structured as Trigger → Procedure → Stop). Across a 3-attempt evaluation horizon on the 25 Veloren tasks, the Deliberative Planner receives a concise briefing of its prior episode trajectories and consolidated lessons before each new attempt.
56.0% (14/25) → 76.0% (19/25)
Consolidated experience across 3 attempts resolves initial bottlenecks in guard bounty contract settlement (Guard Bounty Contract & Settlement), visual rule induction under a strict distance cap (Architectural Schema Learning & Transfer), skill-tree hotbar binding (Skill Configuration & Tactical Combat), and terrain traversal (Elevation & Step Climbing, Wall Detour Navigation).
Up to −51% Time & −52% Tokens
Even on quests already solved zero-shot, experience eliminates exploratory menu checks and camera sweeps—cutting Craft & Equip Gear time from 357.4s to 174.6s (−51%) and Single-Target Melee Combat time from 91.2s to 52.4s (−43%).
91.1s Ambush → Stages 1 & 2 Cleared
On the 15-minute Multi-Stage Crafting, Combat & Expedition quest, JoyTwin recovers from an early first-attempt ambush defeat to defeat the hostile Green Legoom, forage and craft an Apple Stick ration, and trek ~60m via the 2D map to within 7.1m of the rendezvous.
Continual RPG Learning Loop:
1. Record quest actions & world outcomes
2. Diagnose combat or UI stalls
3. Consolidate task-matched lessons
4. Guide the next attempt's stage plan
05 / Benchmark Evaluation
We evaluated JoyTwin and five representative visual-agent baselines across 25 Veloren tasks—comprising 11 foundational RPG survival & traversal tasks and 14 multi-stage open-world RPG missions.
All compared agents operate under a standardized visual-agent evaluation protocol: receiving identical player-visible screen frames and issuing standard keyboard-and-mouse controls under a matched foundation-model backbone. While several baselines handle isolated movement or basic item consumption, the 14 Open-World RPG Missions—which chain NPC dialogue, merchant trading, skill-tree configuration, multi-bench workshop crafting, time-critical supply recovery, guard bounty contracts, and visual architectural rule transfer—expose a stark capability gap: four of the five external baselines solve 0 of 14 Open-World RPG Missions, and the strongest baseline solves only 1 of 14 (7.1%).
To measure both out-of-the-box capability and continual improvement across episodes, each agent is evaluated over up to 3 attempts per task, comparing 1st Attempt (Zero-Shot) performance against With Experience (3 Attempts) performance. Even on its very first attempt, JoyTwin solves 56.0% (14/25) of the tasks—including 6 of 14 (42.9%) complex Open-World RPG Missions—doubling the strongest external baselines (28.0%, 7/25). Across three attempts, while conventional visual-agent baselines struggle to turn raw prior trajectories into new task completions, JoyTwin's verified lesson consolidation and reusable skill synthesis lift its overall success rate to 76.0% (19/25)—solving 10 of 11 (90.9%) Basic Action & Survival tasks and 9 of 14 (64.3%) Open-World RPG Missions.
| Agent / Method | 1st Try · Basic (11) | 1st Try · RPG (14) | 1st Attempt (Zero-Shot) | 3 Tries · Basic (11) | 3 Tries · RPG (14) | With Experience (3 Tries) | Architecture & Multi-Attempt Progression |
|---|---|---|---|---|---|---|---|
| JoyTwin (Ours) | 8/11 | 6/14 | 14/25 (56.0%) | 10/11 | 9/14 | 19/25 (76.0%) | Asynchronous System 2 / System 1 + verified skill & experience evolution (+20.0% across 3 attempts) |
| Tool-Calling Agent | 7/11 | 0/14 | 7/25 (28.0%) | 7/11 | 0/14 | 7/25 (28.0%) | Synchronous visual tool-calling agent (no net task gain across 3 attempts) |
| DeepSeek Harness | 6/11 | 1/14 | 7/25 (28.0%) | 6/11 | 1/14 | 7/25 (28.0%) | Turn-based visual reasoning harness (no net task gain across 3 attempts) |
| SmolAgents | 4/11 | 1/14 | 5/25 (20.0%) | 4/11 | 1/14 | 5/25 (20.0%) | Code-and-action visual agent framework (no net task gain across 3 attempts) |
| GamingAgent | 5/11 | 0/14 | 5/25 (20.0%) | 5/11 | 0/14 | 5/25 (20.0%) | Vision-language gameplay agent harness (no net task gain across 3 attempts) |
| GameScaling | 2/11 | 0/14 | 2/25 (8.0%) | 2/11 | 0/14 | 2/25 (8.0%) | High-frequency reactive action model (no net task gain across 3 attempts) |
06 / Efficiency & Context Reuse
Open-world RPG quests unfold across hundreds of steps—transitioning between 3D wilderness exploration, inventory and crafting menus, merchant dialogues, and real-time combat—which can easily overwhelm foundation models if every frame and HUD update is re-encoded from scratch.
JoyTwin keeps long-horizon RPG deliberation lightweight through two complementary mechanisms: incremental working memory (which highlights only newly observed world or UI changes while keeping prior quest history cache-friendly) and System 1 / System 2 specialization (where the reactive executive handles continuous movement and melee tracking locally while the deliberative planner is invoked only at quest stage transitions or unexpected events).
By preserving stable working-memory prefixes and forwarding only newly observed HUD and scene changes, JoyTwin reuses 89.0% of its planner prompt context from cache and requires only ~12 deliberative planner calls per episode on average.
Consolidated experience eliminates redundant menu inspection and exploratory detours on subsequent attempts—reducing overall token consumption by 20.0% and cutting completion time by up to 51% on multi-step crafting and melee combat tasks.
07 / Key Takeaways & Outlook
Project JoyTwin demonstrates that open-world RPG agents do not have to choose between deep quest reasoning and real-time combat responsiveness.