Under Review Human–robot interaction / sEMG

Visuo-neuromotor Integration
for Flexible Human–Robot
Interactions

Combining visual guidance and muscle signals to translate human motion and force intention into flexible robotic grasping.

Explore core code ↗Discover the method ↓View grasping experiments ↓
Robotic grasping of rigid, deformable, and fragile objects.
Robotic grasping of rigid, deformable, and fragile objects.
8 channelssEMG acquisition at 1,000 Hz
6 outputsthree joint angles + three forces
20 mssliding-window update interval
01 / Method

Two sources of intention.
One interaction loop.

AR captures macro-scale arm motion while sEMG captures hand motion and force intention. A learned decoder and sparse visual calibration connect these complementary signals to compliant hand control.

01 / SENSE

Visual + neuromotor

Eight-channel muscle signals are segmented into 200 ms windows. AR provides arm guidance and sparse hand-pose anchors.

02 / DECODE

CT-Net + adaptive KF

A CNN–Transformer encodes sEMG into 64 features. A 12-state Kalman model estimates angles, forces, and their rates.

03 / INTERACT

Variable admittance

Decoded motion and force references modulate the hand command, balancing intended posture and external contact.

Visuo-neuromotor system overview linking AR guidance, sEMG intent estimation, and robot interaction.
Visuo-neuromotor system overview linking AR guidance, sEMG intent estimation, and robot interaction.
Two-stage estimation: supervised CT-Net training followed by Kalman identification with frozen learned features.
Two-stage estimation: supervised CT-Net training followed by Kalman identification with frozen learned features.
Wearable sEMG and force-sensing hardware used for intention acquisition.
Wearable sEMG and force-sensing hardware used for intention acquisition.
Training → Online adaptation

Learn a representation.
Calibrate with visual anchors.

Stage I trains the CNN–Transformer to predict three joint angles and three forces. Stage II freezes the encoder and fits state dynamics and feature observations by least squares.

During online decoding, sparse visual poses update an exponential moving-average observation bias, while force-related states retain their predicted values.

02 / Reported results

Motion and force,
decoded together.

The manuscript evaluates decoding across 15 recording sessions over 30 days. The following values are manuscript-reported results from one participant, with a 90% / 10% training and test split.

0.875Mean joint-angle R²Proposed method
0.907Mean force R²Proposed method
0.120Mean joint-angle RMSENormalized regression targets
Decoding performance · manuscript Table II · means across three outputs per target
MethodAngle R² ↑Angle RMSE ↓Force R² ↑Force RMSE ↓
CNN0.5940.2380.8650.150
CNN + KF0.6350.2250.8840.142
Proposed method0.8750.1200.9070.109
Representative joint-angle and force trajectories comparing estimates with reference measurements.
Representative joint-angle and force trajectories comparing estimates with reference measurements.
03 / Interaction

Adapt to contact.
Handle diverse objects.

The study evaluates six grasping objects, spanning rigid, deformable, and fragile targets, with 30 trials for each object: a resistance handgripper, water-filled thermos, foam ball, grape, potato chip, and raw egg.

Dual-reference variable admittance combines decoded joint intention, force intention, and external force feedback.
Dual-reference variable admittance combines decoded joint intention, force intention, and external force feedback.
Reported success rates across the six grasping objects and evaluated methods.
Reported success rates across the six grasping objects and evaluated methods.
Reference and estimated joint-angle and force signals during robotic grasping.
Reference and estimated joint-angle and force signals during robotic grasping.
Ablation trajectories examining the CNN, Transformer, Kalman filter, and visual bias adaptation components.
Ablation trajectories examining the CNN, Transformer, Kalman filter, and visual bias adaptation components.
04 / Core implementation

From signal windows
to hand commands.

The repository provides selected modules for sEMG preprocessing, CT-Net training, Kalman identification and online adaptation, and dual-reference admittance. Project status: Under Review.

↗CNN–Transformer encoder

Input: 8 × 200 sEMG window. Outputs: six regression targets or a 64-dimensional feature vector.

↗Adaptive Kalman decoder

Block-structured 12-state identification, six intent outputs, and sparse visual-anchor bias correction.

↗Dual-reference admittance

A 20 ms recursive controller with force-dependent damping and stiffness.

↗Training stages

Supervised encoder learning followed by frozen-feature Kalman fitting.

Browse the repository ↗Scalable project ↗

Visuo-neuromotor Integration for Flexible Human-Robot Interactions · Under Review