arxiv_ai 95% Match Research Paper Robotics Researchers,RL Engineers,Simulation Developers,Computer Graphics Researchers 2 weeks ago

GaussGym: An open-source real-to-sim framework for learning locomotion from pixels

robotics › sim-to-real

📄 Abstract

Abstract: We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in renderer within vectorized physics simulators such as IsaacGym. This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPUs -- while maintaining high visual fidelity, which we showcase across diverse tasks. We additionally demonstrate its applicability in a sim-to-real robotics setting. Beyond depth-based sensing, our results highlight how rich visual semantics improve navigation and decision-making, such as avoiding undesirable regions. We further showcase the ease of incorporating thousands of environments from iPhone scans, large-scale scene datasets (e.g., GrandTour, ARKit), and outputs from generative video models like Veo, enabling rapid creation of realistic training worlds. This work bridges high-throughput simulation and high-fidelity perception, advancing scalable and generalizable robot learning. All code and data will be open-sourced for the community to build upon. Videos, code, and data available at https://escontrela.me/gauss_gym/.

Authors (7)

Alejandro Escontrela

Justin Kerr

Arthur Allshire

Jonas Frey

Rocky Duan

Carmelo Sferrazza

+1 more

Submitted

October 17, 2025

arXiv Category

cs.RO

arXiv PDF Code

Key Contributions

GaussGym integrates 3D Gaussian Splatting with vectorized physics simulators (like Isaac Gym) to achieve unprecedented simulation speed (>100k steps/sec) and high visual fidelity. It enables learning locomotion from pixels, improves sim-to-real transfer, and facilitates rapid creation of diverse training environments from various sources.

Business Value

Dramatically accelerates robot training by enabling faster, more realistic simulations. This reduces development time and cost, and improves the reliability of robots deployed in the real world.

Paper Metadata

Innovation Type

Novel Integration/Framework

Deployment Feasibility

High. Designed as a drop-in renderer and integrates with existing simulators, making it accessible for robotics research and development.

Limitations Addressed

Trade-off between simulation speed and visual fidelity,Difficulty in creating diverse and realistic simulation environments,Sim-to-real gap in robot learning

Performance Gains

Exceeds 100,000 steps/sec on consumer GPUs,High visual fidelity,Improved navigation and decision-making due to rich visual semantics

View Code on GitHub

Technical Tags

3D Gaussian SplattingRobot SimulationLocomotion LearningIsaac GymVectorized PhysicsHigh-Fidelity RenderingSim-to-RealPerceptionScene Generation

Research Topics

RoboticsSimulationComputer GraphicsReinforcement LearningSim-to-Real TransferPerception

Methods & Architectures

3D Gaussian Splatting rendererIntegration with Isaac GymVectorized physics simulationLearning from pixelsScene generation from scans/generative models Gaussian Splatting RendererIsaac Gym

Applications & Tasks

Robotics Simulation Computer Graphics Virtual Reality High-throughput simulationHigh-fidelity visual renderingSim-to-real gapRealistic scene generation Learning robot locomotion from pixelsEnabling fast and photorealistic robot simulationImproving sim-to-real transfer for robots

Datasets & Benchmarks

Datasets

GrandTour, ARKit

Benchmarks

Speed: exceeding 100,000 steps per second on consumer GPUs

Simulation speedVisual fidelityLocomotion performanceSim-to-real accuracy

Related Fields

RoboticsComputer GraphicsReinforcement LearningSimulationComputer Vision

Keywords

robot simulationgaussian splattinglocomotionIsaac Gymsim-to-realreinforcement learninghigh-fidelityreal-time simulationperceptionscene generationGaussGymrobotics

Academic Context

#Robotics#Simulation#Computer Graphics#Reinforcement Learning#Sim-to-Real Transfer#Perception

Companies & Organizations

Companies Mentioned

Google

Technology Stack

Frameworks & Libraries

Isaac GymGaussian Splatting

Commercial Potential

Potential Products

High-speed robot simulatorsTools for generating realistic virtual environmentsPlatforms for sim-to-real robot learning

Target Industries

RoboticsAutomotiveManufacturingLogisticsGaming (for realistic environments)

Use Case Examples

Training legged robots for locomotion in complex terrainsDeveloping autonomous driving systems in highly realistic virtual worldsCreating diverse training scenarios for drone navigation

Competitive Edge

Achieves a unique combination of extreme speed and high visual fidelity in robot simulation, surpassing previous approaches.

Market Opportunity

Large and growing market for robot simulation and AI training platforms.

Revenue Models

Licensing of the frameworkintegration servicescloud-based simulation platforms.

Resource Requirements

Compute Needs

High, requires consumer GPUs capable of running Gaussian Splatting and vectorized physics simulations.

Data Requirements

Requires 3D scene data (e.g., from iPhone scans, large-scale datasets, generative models).

Deployment Constraints

Requires integration with specific physics simulators like Isaac Gym.

Scalability

Designed for high throughput simulation, enabling rapid iteration and large-scale training.

Production Readiness

Maturity Level

Research/Development

Time to Market

Medium, for integrating into production robotics pipelines.

Licensing

Open Source (likely)

View Full Paper Back to Papers