a.AffordSim

AffordSim: A Scalable Data Generator and Benchmark
for Affordance-Aware Robotic Manipulation

NeurIPS 2026 · Evaluations & Datasets Track

Mingyang Li1,*, Haofan Xu1,*, Haowen Sun1,*, Xinzhe Chen1,
Sihua Ren1, Liqi Huang1, Xinyang Sui1, Chenyang Miao1, Jiawei Ye1,
Qiongjie Cui1, Xingyu Chen1, Zeyang Liu1,†, Xuguang Lan1,†

1National Key Laboratory of Human-Machine Hybrid Augmented Intelligence
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

* Equal contribution   † Corresponding authors

arXiv Code
Overview of the AffordSim data generation framework, benchmark and real-world deployment.

Abstract

Many everyday robot manipulation skills are affordance-dependent, with success determined by whether the robot contacts the functional object region required by the subsequent action. Current simulation data generators obtain contacts from generic grasp estimators or per-object manual contact annotations, but generic estimators rank stable grasps without task semantics and often select contacts that are misaligned with the downstream action, while manual contact annotations must be rewritten for each new object and task. To solve these challenges, we introduce AffordSim, a scalable data generator and benchmark that integrates open-vocabulary 3D affordance prediction into simulation-based trajectory generation. Given a natural-language task description, AffordSim synthesizes a task-relevant scene, emits affordance queries, grounds them on object surfaces, samples region-conditioned grasps, and selects executable candidates with motion planning. It further randomizes object pose, texture, lighting, image noise, and cross-viewpoint backgrounds for sim-to-real transfer. We instantiate AffordSim as a 50-task benchmark across diverse manipulation skills, five robot embodiments, and 500+ rigid and articulated objects. AffordSim achieves 93% of the trajectory collection success rate of manual contact annotations on affordance-critical tasks and 89% on hard composite tasks. Vision-language-action policies trained on AffordSim data transfer zero-shot to a real Franka FR3, reaching 24% average success. The data generator, benchmark, and code are available at our project repository.

01 / THE IDEA

Where a robot interacts matters.

The right grasp is more than a stable grasp. It must also support what the robot needs to do next.

AffordSim brings open-vocabulary 3D affordance prediction into simulation-based data generation. From a natural-language task, it builds a scene, identifies task-relevant object regions, and plans executable grasps and trajectories. The resulting data supports policy learning on a benchmark designed around affordance-dependent manipulation.

Two tasks compare AnyGrasp with task-specific grasp and placement or hanging points from AffordSim.

02 / THE FRAMEWORK

The Framework

AffordSim connects scene generation, 3D affordance grounding, grasp selection, and motion planning in one data-generation pipeline.

01

Understand the task

A language description is used to construct a simulated scene and extract task-relevant affordance queries.

02

Ground the interaction

Open-vocabulary 3D affordance prediction identifies object regions relevant to the intended action.

03

Generate trajectories

Region-conditioned grasps and motion planning produce executable demonstrations for policy learning.

AffordSim pipeline from task and scene generation to affordance prediction and trajectory collection.

03 / THE BENCHMARK

50 AffordSim Affordance-Aware Tasks

50 tasks are grouped by affordance reasoning and task horizon:

  • Short-horizon tasks where any stable grasp is sufficient.
  • Short-horizon tasks requiring contact with a specific functional region.
  • Functional-region constraints with long sequences or especially tight precision and stability requirements.

All task examples shown.

04 / MULTI-EMBODIMENT

Multi-Embodiments Support

AffordSim supports five robot embodiments, allowing the same affordance-aware benchmark to be studied across different arms.

FrankaFranka robot configuration.
KinovaKinova robot configuration.
SawyerSawyer robot configuration.
UR5eUR5e robot configuration.
xArm7xArm7 robot configuration.

05 / DOMAIN RANDOMIZATION

Domain randomization.

To support transfer, AffordSim varies object pose, texture, lighting, image noise, and cross-viewpoint backgrounds reconstructed from real scenes.

Examples of varied appearances, lighting, object poses, and backgrounds used in domain randomization.

REAL-WORLD BACKGROUNDS

Real-world background replacement.

Use real-world scenes as simulation backgrounds and switch between camera viewpoints.

Coffee shop

Library

Workshop

Coffee shop

Pick banana and place on plate

Camera viewpoint
Coffee shop, view 1, orbit angle −90 degrees.

Library

Pick chocolate box and place on sugar box

Camera viewpoint
Library, view 1, orbit angle −90 degrees.

Workshop

Pick bowl and place on plate

Camera viewpoint
Workshop, view 1, orbit angle −90 degrees.

06 / SIM-TO-REAL

From simulation to the real world.

As an initial transfer evaluation, policies trained on AffordSim demonstrations are deployed zero-shot on a real Franka FR3 across four manipulation tasks.

Real robot placing a banana on a plate.
Real robot placing a cup on a shelf.
Real robot performing a pouring task.
Real robot hanging a mug on a rack.

24% average zero-shot success across the four real-world tasks for vision-language-action policies trained on AffordSim data. See the paper for the experimental setup and per-task results.

CITATION

Explore the work.

If you find AffordSim useful, please consider citing our work.

Read the paper
BIBTEX
@inproceedings{li2026affordsim,
  title     = {AffordSim: A Scalable Data Generator and Benchmark
               for Affordance-Aware Robotic Manipulation},
  author    = {Li, Mingyang and Xu, Haofan and Sun, Haowen and
               Chen, Xinzhe and Ren, Sihua and Huang, Liqi and
               Sui, Xinyang and Miao, Chenyang and Ye, Jiawei and
               Cui, Qiongjie and Chen, Xingyu and Liu, Zeyang and
               Lan, Xuguang},
  booktitle = {The Fortieth Annual Conference on Neural Information
               Processing Systems, Track on Evaluations and Datasets},
  year      = {2026}
}