🌱 PhysGaia

A Physics-Aware Dataset of Multi-Body Interactions
for Dynamic Novel View Synthesis

arXiv arXiv GitHub GitHub Hugging Face Dataset 📝 Cite
Example 1
Example 2
Example 3
Example 4
Example 5
Example 6
Example 7
Example 8

TL;DR: PhysGaia is the first physics-aware dataset for DyNVS
focused on physical realism, not just photorealism.

Abstract

We introduce PhysGaia, a novel physics-aware dataset specifically designed for Dynamic Novel View Synthesis (DyNVS), encompassing both structured objects and unstructured physical phenomena. Unlike existing datasets that primarily focus on photorealistic reconstruction, PhysGaia is created to actively support physics-aware dynamic scene modeling. Our dataset provides complex dynamic scenarios with rich interactions among multiple bodies, where they realistically collide with each other and exchange forces. Furthermore, it contains a diverse range of physical materials, such as liquid, gas, textile, and viscoelastic substances, which moves beyond the rigid bodies prevalent in existing datasets. All scenes in PhysGaia are faithfully generated to strictly adhere to physical laws, leveraging carefully selected material-specific physics solvers. To enable quantitative evaluation of physical modeling, our dataset provides essential ground-truth physics information, including 3D particle trajectories and physics parameters, e.g., viscosity. To facilitate research adoption, we also provide essential integration pipelines for using state-of-the-art 4D Gaussian Splatting models with our dataset and report theirs results. By addressing the critical lack of datasets for physics-aware modeling, PhysGaia will significantly advance research in dynamic view synthesis, physics-based scene understanding, and deep learning models integrated with physical simulation ultimately enabling more faithful reconstruction and interpretation of complex dynamic scenes.

🔥 Ideal for "Next" Research

  • 🧠 Physical reasoning in dynamic scenes
    • Offering ground-truth physics parameters for precise evaluation of inverse physics estimation
    • Offering ground-truth 3D trajectories for assessing actual motion beyond photorealism
  • 🤝 Multi-body physical interaction modeling
  • 🧪 Material-specific physics solver integration
  • 🧬 Compatibility with existing DyNVS models
    • Check implementation code of recent methods on PhysGaia: D-3DGS, 4DGS, STG, SOM
Ground Truth

Ground Truth

D-3DGS

D-3DGS (CVPR24)

4DGS

4DGS (CVPR24)

STG

STG (CVPR24)

SOM

SOM (arXiv 24.07)

Material Types

PhysGaia features four distinct material types, each with unique physical properties and behaviors:

📊 Dataset Overview

4
Material Types
17
Scenes
4-6
Camera Views
240
Frames

📁 Data Structure

{material_type}_{scene_name}.zip
│
├── render/                              # Rendered images
│   ├── train/                           # Images for training
│   └── test/                            # Images for evaluation
│
├── point_cloud.ply                      # COLMAP initialization (PatchMatch & downsampling)
├── camera_info_test.json                # Monocular camera info for test
├── camera_info_train_mono.json          # Monocular camera info for training
├── camera_info_train_multi.json         # Multi-view camera info for training
│
├── {scene_name}.hipnc                   # Houdini source file (simulation or scene setup)
├── particles/                           # Ground-truth trajectories

📹 Camera Setup

Camera Configuration

  • 4 to 6 viewpoints per scene
  • Spiral camera trajectory around physics interactions
  • Multiple Settings for both monocular and multi-view
  • 240 frames capturing complete dynamics

Technical Specifications

  • Resolution: 720×960 pixels
  • Format: PNG images
  • Frame Rate: 24 FPS
  • Tool: Houdini 20.5

Citation

BibTeX

TBD