
Ethz.ch
Robotics has always drawn a clean line between hands and legs. A robotic arm grips; a legged robot moves. Engineers building a system that needs to do both bolt the two together, pay for both, and debug both when something breaks. A research paper published this week from ETH Zurich’s Soft Robotics Lab erases that line entirely — by teaching a dexterous robotic hand to use its own fingers as legs, crawl across 14 different surface types, recover from falls, and press keyboard keys while supporting its own weight.
The paper, “Fingers as Legs: Learning Self-Supported Locomotion and Manipulation with an Anthropomorphic Hand,” was submitted to arXiv on September 15 by PhD researcher Amirhossein Kazemipour, co-author Hehui Zheng, and Professor Robert Katzschmann, the lab’s director. The complete system — including onboard battery and a small onboard computer — weighs 818 grams (1.80 lbs). It has no wheels, no separate locomotion hardware, and no tether. It is, functionally, a hand that has learned to get around.
Fingers Doing Double Duty: The Core Problem
The hardware the ETH Zurich team used is a WUJI Hand, a commercially available five-finger robotic hand manufactured in China, with 20 active degrees of freedom and direct-drive brushless actuators. It was designed for grasping and manipulation, not locomotion. Its five fingers are not symmetric — the thumb differs from the index finger, the index from the pinky, each with its own joint geometry, range of motion, and load-bearing capacity.
That asymmetry is the central engineering problem. When any one finger lifts off the ground to take a step, the remaining fingers must continue supporting the entire system’s weight while still being available for manipulation tasks. There is no division of labor. Every degree of freedom does double or triple duty simultaneously.
The specific constraint this creates is one standard robotic locomotion approaches cannot handle: reward functions designed for quadruped robots assume symmetric legs. An anisotropic hand — one where each “leg” is mechanically different — needs different guidance to learn how to move efficiently.
The Reinforcement Learning Architecture That Made It Work
The ETH Zurich team trained the hand’s locomotion policies using reinforcement learning entirely in simulation before deploying them on real hardware, a now-standard approach in the field called sim-to-real transfer. What distinguishes their approach is how they closed the gap between simulation and reality, and how they formulated the reward signal that teaches the hand what a “good step” looks like.
Simulator calibration from hardware measurements. Rather than relying solely on domain randomization — the common practice of randomly varying physics parameters during training to make policies robust to real-world uncertainty — the team calibrated their simulator directly against measurements from the physical WUJI Hand. The hand’s actual actuator dynamics, friction, and contact behavior informed the simulation environment. This real2sim approach reduces the gap the policy must cross when moving from virtual training to physical hardware.
Asymmetric actor-critic PPO. The learning algorithm is Proximal Policy Optimization with an asymmetric actor-critic structure. During training, a critic network has access to privileged information available only in simulation: joint torques and fingertip contact forces. The actor — the policy that actually runs on the physical hand — sees only what real hardware sensors can provide: eight consecutive snapshots of joint angles, tilt, rotation rate, and previous control outputs, plus a movement command. The actor’s input is 392 values total (8 time steps × 49 sensor readings per step). A feedforward neural network with three hidden layers (512, 256, and 128 units) converts those inputs into 20 joint-target adjustments. The built-in position controller on the WUJI Hand then executes those adjustments. Training uses 4,096 parallel simulated environments simultaneously.
A reward formulation built for unequal fingers. This is the team’s specific technical contribution. Instead of transplanting reward functions designed for symmetric quadruped robots, they built a per-finger “virtual spring” cost: each fingertip has a reference position relative to the hand’s body in a calibrated reference frame, and straying from that reference incurs a cost. Crucially, the cost is anisotropic — moving forward costs 25% as much as moving laterally or vertically (cost per finger = 0.25 × forward² + sideways² + vertical²). This asymmetry in cost structure gives the policy “soft guidance” about the direction of travel without fixing the exact sequence of steps; the policy learns which finger lifts when, and how far it moves, through trial and error.
In simulation, the result is quantifiably better. Across 12 independent training seeds and 256 randomized episodes per seed, the custom formulation produced a mean crawling speed of 1.67 centimeters per second (0.66 in/s). Standard reward functions tuned for quadruped robots achieved 1.02 centimeters per second (0.40 in/s) on the same hardware. The 95% confidence interval for the 0.65 cm/s speed improvement is [+0.26, +1.02], ruling out chance at conventional significance thresholds.
What the Hardware Can Actually Do
On real hardware, the trained policies enabled the following demonstrated capabilities:
Untethered crawling across 14 surface types. The hand walked on rubber mat, carpet, tile, hardwood, metal grating (diamond plate), asphalt, dry concrete, cut stone, artificial turf, weathered stone, grass, gravel, hard court, and a hard outdoor surface beside the Limmat River in Zurich. Steering responds to directional commands using only the hand’s onboard sensors — no camera, no external tracking.
Fall recovery. A dedicated recovery policy, separate from the crawling policy, returns the hand to an operational upright position after being toppled. In hardware testing, the recovery succeeded in 21 of 25 trials (84%) across two fall directions.
Self-supported keyboard commands. In the manipulation demonstration, the hand executed Sokoban puzzle moves by pressing 12 keyboard keys in sequence — while supporting its own weight on the remaining fingers. No camera input was used for key targeting; the system relied entirely on proprioceptive sensing (joint position and body orientation). Accuracy: 29 of 32 commands executed correctly (91%), with a median correct-press latency of 0.25 seconds.
Object pushing with visual feedback. Using an overhead camera mounted above the workspace, the hand approached a small cube and pushed it to a target location. Mean final target error across 15 demonstrated trials: 17 millimeters (0.67 in).
Why This Changes the Math on Mobile Manipulation
The conventional approach to building a robot that needs to both move through the world and interact with it is straightforward but expensive: put an arm on a mobile base, or bolt a manipulation end-effector to a legged robot. Each approach requires two complete hardware systems — two actuator trains, two power systems, two control stacks — with every added component representing both a cost and a potential failure point.
The ETH Zurich result suggests a different path. A single commercial robotic hand, augmented with a small battery and onboard computer, can perform both functions using the same 20 joints. The design does not require custom hardware; the WUJI Hand is an off-the-shelf research platform priced around $16,000 USD. The entire walking-and-manipulating system weighs under one kilogram.
The specific applications the paper points to are spaces where full mobile manipulation platforms simply cannot fit: server rack interiors, collapsed structural environments, and densely packed industrial shelving. A hand that can walk into a tight space, position itself, and then press a button or push an object to a target does not need the arm and mobile base that would normally be required to reach that target.
This work connects to a broader current in robotics research called “loco-manipulation,” which studies the boundary between moving through the world and acting on it. Prior work from ETH Zurich explored the inverse premise: the “Pedipulate” system presented at ICRA 2024 by ETH Zurich used a quadruped robot’s leg as a manipulation appendage — legs doing a hand’s job. This paper inverts that: a hand doing a leg’s job.
Limitations the Paper Acknowledges
The walking speed — 1.67 cm/s (0.66 in/s) in simulation, somewhat slower on real hardware — is substantially below what dedicated legged robots achieve. Boston Dynamics’ Spot, for example, walks at roughly 1.6 meters per second (63 in/s), approximately 100 times faster. Finger-based locomotion trades speed for compactness.
The joint wear concern is real and not yet quantified: the WUJI Hand’s actuators were designed for grasping, where load-bearing cycles are brief. Continuous locomotion subjects the same joints to sustained weight-bearing forces across hundreds or thousands of steps. The WUJI Hand is rated for over 300,000 grasp cycles, but the fatigue profile for locomotion loading is a different mechanical regime than grasping.
The manipulation demonstrations, while technically impressive, operate in structured settings. The keyboard task requires manual initial alignment; the cube-pushing task uses an overhead camera in a controlled workspace. Extending these capabilities to unstructured environments — where keys may not be precisely positioned and visual feedback may be occluded — is the next research challenge.
The sim-to-real gap, while reduced by the calibration approach, remains an active area of work. Katzschmann’s lab has invested in high-fidelity simulator calibration and domain randomization techniques to improve policy transfer, but the gap between calibrated simulation and the full complexity of uncontrolled real environments is not eliminated.
Code for this project has not been released publicly as of the paper’s publication date.
How Does a Robotic Hand Learn to Walk Using Reinforcement Learning?
The short answer: through millions of simulated attempts, with a reward signal that tells the policy what “better” looks like. In this case, the policy runs in 4,096 parallel simulated environments simultaneously, testing different joint-movement sequences and accumulating experience about which sequences result in forward motion without falling over. The Proximal Policy Optimization algorithm updates the policy’s neural network weights based on that accumulated experience, gradually converging on a strategy that balances crawling speed against stability.
The distinguishing feature of the ETH Zurich approach is the reward formulation. Because the hand’s fingers are not symmetric (a thumb is not a pinky), a generic reward signal designed for symmetric quadruped legs trains a less effective crawling policy. The team’s anisotropic virtual-spring reward, which makes forward deviation from reference cheaper than lateral or vertical deviation, guides the policy toward efficient crawling motion specific to this hand’s geometry.
About ETH Zurich’s Soft Robotics Lab
The Soft Robotics Lab is led by Professor Robert Katzschmann, a German roboticist who completed his PhD at MIT‘s Computer Science and Artificial Intelligence Laboratory under Daniela Rus and has been a TED Fellow since 2022. The lab is part of ETH Zurich’s Institute of Robotics and Intelligent Systems (IRIS) and is a member of the ETH AI Center and the Max Planck ETH Center for Learning Systems.
Prior work from the lab includes SoFi, an autonomous soft robotic fish capable of swimming alongside real fish, described in Science Robotics in 2018, and covered at the time by The New York Times. The lab has also developed soft robotic hands for pick-and-place tasks and multi-segment arms for dynamic object manipulation. “Fingers as Legs” extends the lab’s ongoing interest in systems that blur the boundary between locomotion and manipulation — here, using commercially available dexterous hardware rather than custom soft materials.
The paper is available at arXiv preprint 2609.17172. A research film and technical project page, including interactive explanations of the reward formulation, are live at the Soft Robotics Lab’s website.
Frequently Asked QuestionsWhat is the key technical advance in the ETH Zurich “Fingers as Legs” paper?
The main advance is a reinforcement learning reward formulation specifically designed for a hand’s asymmetric finger geometry. Standard quadruped locomotion rewards assume symmetric limbs — which a human-style hand does not have. By building a per-finger “virtual spring” cost with anisotropic weights (forward movement costs less than sideways or upward), the ETH Zurich team produced a policy that crawls 63% faster in simulation than the same hardware trained with generic quadruped rewards. On real hardware, this translated to a system that walks untethered across 14 surface types, recovers from falls 84% of the time, and presses keyboard keys with 91% accuracy while supporting its own weight.
How does sim-to-real transfer work in robot locomotion research?
Robots trained entirely through physical trial and error take too long and risk hardware damage. Researchers instead train in simulation — a physics model of the robot and its environment — then transfer the trained policy to real hardware. The challenge is that simulation is never a perfect match for the real world: actuator friction, sensor noise, and contact dynamics differ. Standard approaches include domain randomization (randomly varying physics parameters during training) and asymmetric actor-critic methods (giving the training algorithm access to information that real sensors cannot provide). The ETH Zurich team also calibrated their simulator directly from measurements of the physical WUJI Hand, reducing the gap between training environment and real-world deployment.
Does this research eliminate the need for separate mobile base hardware in manipulation robots?
Not yet — and probably not for high-speed or long-distance tasks. The demonstrated system reaches about 1.67 cm/s (0.66 in/s) in simulation and somewhat slower on hardware. Dedicated legged robots move roughly 100 times faster. What this approach offers instead is compactness: an 818-gram (1.80-lb) system that fits in spaces where a wheeled base or a legged robot with an arm cannot, while still being capable of manipulation tasks once positioned. The economics of single-system design — one actuator train serving both locomotion and manipulation — reduce hardware cost and complexity in specific use cases rather than competing with dedicated locomotion platforms on performance.
What are the limitations of using manipulation hardware for walking?
Three concrete constraints: speed (manipulation-optimized actuators are not designed for sustained locomotion pace), joint wear (the load-bearing profile during walking differs from the brief, high-force cycles the hand was rated for, and fatigue effects are not yet characterized), and environment scope (the manipulation tasks demonstrated — keyboard pressing, cube pushing — require structured settings with controlled initial conditions). The team also notes that steering remains somewhat asymmetric, reflecting the hand’s inherent left-right asymmetry. These are the next research challenges, not fundamental barriers — as the arXiv preprint 2609.17172 acknowledges directly.