FluxThemis
A standalone VLA evaluation framework for multiple models and simulators
Contributor Code
TL;DR
FluxThemis is a standalone VLA evaluation framework that supports multiple models and simulators. It separates model inference from simulation evaluation and isolates simulator dependencies, making evaluation pipelines easier to reproduce and extend.
My Role
I led the development of the LIBERO-Plus evaluation pipeline. ROS decouples model inference from simulation evaluation, while separate Conda environments isolate LIBERO, LIBERO-Plus, and robosuite dependencies.
Highlights
- Large-scale robustness evaluation. Covers 10,030 LIBERO-Plus robustness tasks.
- Flexible analysis. Supports metadata filtering and grouped success-rate statistics.
- Parallel execution. Provides multi-process rollouts for faster evaluation.
Browse the source code on GitHub.