arxiv_ml 95% Match Research Paper Robotics researchers,ML engineers working on robotics,AI researchers,Robot developers 2 weeks ago

DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation

robotics › manipulation

📄 Abstract

Abstract: We present DexCanvas, a large-scale hybrid real-synthetic human manipulation dataset containing 7,000 hours of dexterous hand-object interactions seeded from 70 hours of real human demonstrations, organized across 21 fundamental manipulation types based on the Cutkosky taxonomy. Each entry combines synchronized multi-view RGB-D, high-precision mocap with MANO hand parameters, and per-frame contact points with physically consistent force profiles. Our real-to-sim pipeline uses reinforcement learning to train policies that control an actuated MANO hand in physics simulation, reproducing human demonstrations while discovering the underlying contact forces that generate the observed object motion. DexCanvas is the first manipulation dataset to combine large-scale real demonstrations, systematic skill coverage based on established taxonomies, and physics-validated contact annotations. The dataset can facilitate research in robotic manipulation learning, contact-rich control, and skill transfer across different hand morphologies.

Authors (12)

Xinyue Xu
Daisy

Jieqiang Sun
Daisy

Jing
Daisy

Dai

Siyuan Chen

Lanjie Ma

+6 more

Institutions

🏛️ Daisy

Submitted

October 17, 2025

arXiv Category

cs.RO

arXiv PDF

Key Contributions

DexCanvas is a large-scale hybrid real-synthetic dataset for dexterous manipulation, containing 7,000 hours of interactions derived from 70 hours of human demonstrations across 21 manipulation types. It uniquely combines multi-view RGB-D, high-precision mocap with MANO hand parameters, and physically consistent contact force profiles. The paper also presents a real-to-sim pipeline using RL to train policies for an actuated MANO hand in simulation, reproducing human demonstrations and discovering contact forces.

Business Value

Accelerates the development of robots capable of complex manipulation tasks, opening doors for automation in manufacturing, logistics, and domestic assistance.

Paper Metadata

Innovation Type

Dataset and Methodology

Deployment Feasibility

Moderate. The dataset and pipeline are valuable research tools. Deploying learned policies on real robots requires significant engineering effort for sim-to-real transfer and hardware integration.

Limitations Addressed

Lack of large-scale, diverse datasets for dexterous manipulation,Challenges in sim-to-real transfer for contact-rich tasks,Difficulty in learning complex manipulation skills from scratch

Performance Gains

Facilitates research in robotic manipulation learning by providing a comprehensive dataset and a pipeline for training policies that reproduce human demonstrations.

Technical Tags

Robotic ManipulationDexterous HandsHuman DemonstrationsReinforcement LearningSim-to-Real TransferDataset CreationContact-rich ControlHumanoid RobotsSynthetic Data Generation

Research Topics

Robotic Dexterous ManipulationLearning from DemonstrationSim-to-Real TransferRobotic Hand ControlDataset for Robotics

Methods & Architectures

Hybrid real-synthetic datasetReinforcement learningReal-to-sim pipelinePhysics simulationMANO hand model Actuated MANO hand modelReinforcement learning policies

Applications & Tasks

Robotics Humanoid Robotics Industrial Automation Human-Robot Interaction Enabling dexterous manipulation for robotsBridging the gap between human demonstrations and robot learningImproving sim-to-real transfer for manipulation tasksCreating large-scale, diverse manipulation datasets Dexterous hand-object interactionLearning manipulation skillsRobotic grasping and manipulation

Datasets & Benchmarks

Datasets

DexCanvas, Cutkosky taxonomy

Manipulation skill reproductionContact force accuracyPolicy performance in simulation

Related Fields

RoboticsMachine LearningComputer VisionHuman-Computer InteractionSimulation

Keywords

roboticsmanipulationdexterous handshuman demonstrationsreinforcement learningsim-to-realdatasetcontactphysics simulationMANOhumanoidrobot learning

Academic Context

#Robotic Dexterous Manipulation#Learning from Demonstration#Sim-to-Real Transfer#Robotic Hand Control#Dataset for Robotics

Commercial Potential

Potential Products

Robotic manipulation platformsAI software for robot skill learningSimulation environments for robotics training

Target Industries

ManufacturingLogisticsWarehousingConsumer ElectronicsHealthcare (e.g., assistive robots)

Use Case Examples

Training robots for assembly line tasks requiring fine motor skills.Developing robots for handling delicate objects in warehouses.Creating humanoid robots capable of performing household chores.

Competitive Edge

Provides a unique, large-scale, and richly annotated dataset specifically for dexterous manipulation, coupled with a practical real-to-sim training pipeline, addressing key gaps in current research.

Market Opportunity

Large and growing market for advanced robotics, particularly in automation and logistics.

Revenue Models

Licensing of the dataset and training methodologiesdevelopment of specialized robotic systems.

Resource Requirements

Compute Needs

High for training RL policies and generating synthetic data; moderate for dataset usage.

Data Requirements

The DexCanvas dataset itself is a primary requirement for using the presented methods.

Deployment Constraints

Significant challenges in transferring learned policies from simulation to real-world robots, especially for precise contact-rich manipulation.

Scalability

The dataset is large-scale (7,000 hours). The RL training process can be computationally intensive but is designed for scalability.

Regulatory Considerations

Safety standards for human-robot interaction and industrial robotics.

Production Readiness

Maturity Level

Research

Time to Market

3-5 years for robust commercial deployment of manipulation capabilities.

Patent Potential

Moderate (related to dataset creation methodology and RL pipeline)

View Full Paper Back to Papers