Human Hands,
Teaching Robot Hands
A person wearing the OpenMuscle band performs an everyday task. The band reads the forearm, and a second device records exactly what the hand did. Every session becomes labeled demonstration data, the kind that trains dexterous and humanoid robot hands.
Why forearm sensing makes good robot data
Dexterous manipulation models are starved for one thing: large, diverse, labeled recordings of real human hands doing real tasks. Camera rigs and instrumented gloves can capture this, but they are expensive, get in the way of the task, and rarely leave the lab.
OpenMuscle takes a different route. A flexible band of pressure sensors reads the muscle topology of the forearm while the hand moves freely and unobstructed. Pair that signal with a ground-truth record of what the fingers actually did, and you have a clean training pair: muscle input in, hand pose out. The same mapping that drives a prosthetic hand is the mapping a humanoid robot needs to imitate human dexterity.
Because the sensor never blocks the hand, capture can happen during the activity itself: typing, gaming, playing an instrument, manipulating objects. The task provides the labels for free.
Three ways to capture ground truth
The sensor band is one half of every training pair. The other half is a record of what the hand truly did. OpenMuscle supports three label sources, each richer than the last.
LASK5 actuator wand
A handheld labeling wand drives four linear actuators that the wearer follows. Each actuator position is a precise, repeatable label for one finger. This is the ground truth behind the finger-prediction results on the Technology page.
4 degrees of freedom, deterministic.
VR hand tracking
A WebXR client uses a Meta Quest 3's onboard hand tracking to record the full articulated hand pose while the wearer moves naturally. Far richer than four actuators, and the model's live prediction is drawn back as a ghost hand overlaid on the real one.
Full hand pose, no extra hardware on the hand.
HID input devices
Standard Human Interface Devices become label tracks. Type on a keyboard, play a game controller, or perform on a MIDI instrument while wearing the band, and every keystroke, button, or note is a timestamped record of finger intent, captured for free during a natural activity.
Keyboard, game controller, MIDI (MIDI first).
VR integration
In developmentVirtual reality headsets ship with high-quality hand tracking built in. OpenMuscle uses it as a ground-truth sensor. The wearer puts on the band and a Quest 3, and a browser-based WebXR client streams the headset's per-joint hand pose to the same pipeline that records the band's pressure data. The two are paired by timestamp into training examples.
During inference, the loop runs the other way: the model's prediction renders as a translucent ghost hand floating over your real hand, so you can see where the sensors agree with reality and where they do not. It is a fast, hardware-light way to gather rich, high degree-of-freedom labels and to debug a model in real time.
What VR capture adds
- → Full articulated hand pose instead of four actuator values
- → Nothing strapped to the hand, so motion stays natural
- → Live ghost-hand overlay for instant model feedback
- → Runs in the headset browser, no app install
Built on the WebXR client in OpenMuscle-AR.
HID capabilities
On the roadmapInput as label
A Human Interface Device already knows, precisely and with timestamps, what your fingers did. OpenMuscle records that stream next to the sensor data so the device does the labeling for you. The person just plays, types, or performs.
- • Keyboard: each key is a known finger and position
- • Game controller: buttons, triggers, and sticks map to finger pressure
- • MIDI instrument: notes and velocity give graded, expressive finger motion
Output as control
The same pipeline runs in reverse. Once a model can read finger intent from the band, its predictions can act as an input device, driving a cursor, a game, a robot hand, or a humanoid end effector without a physical controller.
Capture and control share one format, so a model trained from HID labels can later emit HID-style commands. The human teaches by doing; the robot performs by predicting.
From demonstration to dexterity
Demonstrate
A person wears the band and performs a task while a label source (LASK5, VR, or HID) records what the hand does.
Pair
Sensor packets and labels are paired by timestamp into clean training examples: muscle pressure in, hand pose out.
Publish
Sessions are released as an open dataset under a permissive license, so any lab can train against the same data.
Transfer
Models learned from human demonstration drive prosthetic hands today, and dexterous and humanoid robot hands as the dataset grows.
An open dataset, free for robotics research
The capture sessions behind this work are published openly under CC BY 4.0. Use them to train, benchmark, or bootstrap your own dexterous-manipulation models. The corpus grows with every new recording.
Building humanoid hardware or manipulation models? We would like to capture data with you.