Back to projects

Research | NTU

Beyond Video Streaming: Digital Twin Teleoperation at Kbps

3DGS-based robot teleoperation with 360° free-viewpoint rendering at minimal bandwidth

Sep 2023 – Present

Overview

A digital twin teleoperation paradigm: pre-build the remote scene as a high-fidelity 3D twin, transmit only robot pose and dynamic obstacles at ~500 Kbps, enabling photorealistic 360° free viewpoints and day/night operation.

Tele-Twin (Gen 1)

Mesh-based digital twin teleoperation with photogrammetry reconstruction and 3-DOF motion feedback.

Teleoperation concept

Communication Architecture

Communication

3-DOF Motion Platform

Motion platform
Gallery image 1
Gallery image 2
Gallery image 3
Gallery image 4

Tele-GS (Gen 2)

3D Gaussian Splatting replaces mesh — photorealistic rendering with 59% less GPU memory and 84% less storage.

System Overview

System overview

Scene Reconstruction Pipeline

Reconstruction pipeline

Communication Architecture

Communication architecture
3D Gaussian SplattingUnreal Engine 5ZenohZeroMQLiDAR CenterPointTensorRTNDT+EKF LocalizationROS2