IEEE TIE · 2026 Robot learning / Augmented reality

Scalable Dexterous
Robot Learning with
AR-based Remote Human–Robot Interactions

From remote human demonstrations to dexterous robot skills: behavior cloning meets contrastive reinforcement learning.

Yicheng Yang, Ruijiao Li, Lifeng Wang, Shuai Zheng, Shunzheng Ma,
Keyu Zhang, Tuoyu Sun, Chenyun Dai, Jie Ding, and Zhuo ZouFudan University · IEEE Transactions on Industrial Electronics

Read paper ↗arXiv ↗Core code ↗Watch experiments ↓
Dexterous manipulation across simulated and physical environments.
Dexterous manipulation across simulated and physical environments.
15expert demonstrations in the main setup
95.70%bottle success in simulation
94%bottle success on the real robot
01 / Approach

Demonstrate once.
Learn beyond imitation.

An AR interface captures remote human demonstrations. The actor first learns from expert state–action pairs, then improves through interaction with a contrastive objective that retains expert action structure.

01 / COLLECT

AR demonstrations

Remote arm and hand interaction provides aligned expert states and actions for robot learning.

02 / INITIALIZE

Behavior cloning

Supervised actor pretraining provides a demonstration-based starting point for reinforcement learning.

03 / REFINE

Contrastive SAC

SAC combines task rewards with expert-paired projected actions; the projection head is used during training.

System overview: AR interaction, demonstration collection, two-stage learning, and robot execution.
System overview: AR interaction, demonstration collection, two-stage learning, and robot execution.
The projection head aligns policy and expert action representations during reinforcement learning.
The projection head aligns policy and expert action representations during reinforcement learning.
Learned manipulation policies execute grasping sequences across the evaluated objects.
Learned manipulation policies execute grasping sequences across the evaluated objects.
02 / Evaluation

Learning from fewer
demonstrations.

Compare the proposed method with behavior cloning and reinforcement learning baselines across bottle, ball, and drill tasks. Values below are reported in the published paper.

Simulation success rate

Mean ± standard deviation · three random seeds

Source: published paper, Table IV. Seeds: 42, 97, and 215. Success rates are percentages.

Training curves for the three simulation tasks, as reported in the paper.
Training curves for the three simulation tasks, as reported in the paper.
03 / In action

From simulation
to physical grasping.

Recorded bottle and ball experiments show policy execution in simulation and on the physical robotic hand. Use the player controls to inspect each rollout.

Bottle manipulationSimulation
Ball manipulationSimulation
Bottle graspingReal robot
Ball graspingReal robot
Real-robot success rates for bottle, ball, and drill grasping; 50 trials per object and method.
Real-robot success rates for bottle, ball, and drill grasping; 50 trials per object and method.
04 / Implementation

Inside the learning loop.

Selected core modules expose the two-stage learning method, expert-paired contrastive objective, and task rewards. The policy maps a 20-dimensional observation to an 8-dimensional action.

↗Behavior cloning

Pretrain the SAC actor using aligned expert observations and actions.

↗Contrastive SAC

SAC actor–critic updates with a 256–256–128 action projection head.

↗Task reward

Distance, progress, event, and hand-pose alignment components.

↗Two-stage training

Connect demonstration pretraining and reinforcement learning through an environment factory.

Explore the repository ↗Visuo-neuromotor project ↗
Citation
Y. Yang et al., “Scalable Dexterous Robot Learning with AR-based Remote Human-Robot Interactions,” IEEE Transactions on Industrial Electronics, 2026.
DOI: 10.1109/TIE.2026.3713549