Visual Affordance Prediction: Survey and Reproducibility

Abstract

Affordances are the potential actions an agent can perform on an object, as observed by a camera. Visual affordance prediction is formulated differently for tasks such as grasping detection, affordance classification, affordance segmentation, and hand pose estimation. This diversity in formulations leads to inconsistent definitions that prevent fair comparisons between methods. In this paper, we propose a unified formulation of visual affordance prediction by accounting for the complete information on the objects of interest and the interaction of the agent with the objects to accomplish a task. This unified formulation allows us to comprehensively and systematically review disparate visual affordance works, highlighting strengths and limitations of both methods and datasets. We also discuss reproducibility issues, such as the unavailability of methods implementation and experimental setups details, making benchmarks for visual affordance prediction unfair and unreliable. To favour transparency, we introduce the Affordance Sheet, a document that details the solution, datasets, and validation of a method, supporting future reproducibility and fairness in the community.

Highlights

Unified problem formulation of visual affordance
Systematic review of methods and datasets
Reproducibility issues in visual affordance datasets and benchmarks
Affordance Sheet to favour reproducibility
Open challenges and future directions of visual affordance for robotics

Motivation

Given a high-level task the agent has to perform, we consider as a visual affordance the combination of the following three aspects:

what: the potential action on the most suitable objects in the image to accomplish the task;
where: the region where the agent will interact with the object through its hand; and
how: the most physically plausible end-effector (hand) pose to interact with the object.

Our definition enables the grouping and comparison of previous works, showing that an incomplete formulation of the affordance (maximum two aspects) was considered. None of the previous surveys discussed the limitations in affordance formulations or provided a unifying view that enables an agent to interact with objects.

Unified affordance problem formulation

Our formulation integrates the redefinitions related to affordance prediction given the task to accomplish and the RGB image. We decompose visual affordance prediction in the following subtasks and related components:

Localise the object of interest (object localisation).
Predict the actions for each localised object (functional classification).
Predict the object regions that enable to perform the action (functional segmentation).
Estimate the end-effector pose on the object, given the hand model and previous extracted information (end-effector pose estimation).

Each component of our formulation instantiates one or more subtasks. For example, functional segmentation groups affordance segmentation and affordance grounding, as these subtasks have a similar problem formulation; grasping detection can be considered as a special case of end-effector pose estimation.

Previous methods

Redefinition	Considered aspects	Challenges	Limitations
Functional classification	What: Potential actions or affordance classes an agent can perform on an object from an input image given a specific task.	Without a defined task, a single object has multiple plausible affordances (e.g., a cup can suggest picking or filling). Objects with highly similar visual appearances can afford entirely different actions (e.g., a trowel vs. a turner). Graphical complexity and dimensionality reduction techniques used in some methods challenge scalability to high-resolution images.	Physical interaction between the agent and the object is not considered because the predicted action is not associated with a specific interaction region in the image. Multiple options on how and where to perform the interaction.
Functional segmentation	Where: Predicts and identifies where the agent needs to perform the interaction on the object by separating its functional regions or grounding the specific region needed for a defined action.	Occlusions, blurred or poorly defined edges, and transparent objects Processing feature maps at low resolutions causes important details like boundaries to degrade for background objects. When attention weights rely heavily on object class supervision, errors during occlusions can cause a complete mismatch between predicted object classes and segmented affordance regions.	For detector-dependent models, inaccurate or incorrect object detection branch predictions can result in segmenting affordance regions entirely outside of the actual physical objects. Semantic models that bypass an object detector entirely can mistakenly classify background pixels as affordance regions when boundaries are unclear or occluded.
End-effector (hand) pose estimation	How & Where: Estimates how and where the agent's hand interacts with the object by predicting the physical pose of the end-effector or hand model directly on the interaction region.	The hand itself is not observed in the input image, forcing the model to rely solely on the visual features of the object. For two-finger grasping, each object presents multiple grasping points, but only a small subset leads to a successful grasp, and estimating these from a single view is difficult due to self-occlusion. For multi-finger models, performance drops significantly if multiple objects are present simultaneously.	Grasping detection only considers the specific interaction of picking. Most grasping detection models assume a top-down camera view on a tabletop or floor, failing to generalize to varied viewpoints. Multi-finger pose estimation methods show how and where the hand interacts but fail to identify what specific action or task is being performed.

Previous datasets

Contrary to previous surveys, we compare image-based datasets for visual affordance prediction, highlighting their similarities and limitations. We consider elements such as: the environment type (e.g. indoor), the camera viewpoint (e.g. third person); object categories (e.g. containers), type of images (e.g. mixed-reality), occlusions (e.g. hand manipulating the object), and annotation (e.g. accuracy and procedure).

Each dataset is specific to an affordance redefinition rather than the unified formulation, preventing datasets reuse. Furthermore, annotating affordances in images and videos is non-trivial due to ambiguous regional boundaries, overlapping action zones, and the complexity of labelling 3D end-effector poses. These challenges limit the cross-datasets evaluation of methods and the scalability of datasets, as manual annotations are time-consuming and ambiguous, and require expensive resources The majority of the analysed datasets has fewer than 20 object categories and 10 affordances, focusing on tools and containers.

Legend: Affordance classification, Affordance detection and segmentation, Affordance grounding, Hand-object pose estimation.

Reproducibility in visual affordance

Reproducibility challenges (RCs) in different redefinitions of visual affordance prediction include: data availability for benchmarking (RC1); availability of a method's implementation (RC2); availability of the trained model (RC3); details of the experimental setups (RC4); and details of the performance measures used for the evaluation (RC5).

Affordance sheet

To promote reproducibility in affordance prediction and overcome reproducibility challenges, we propose the Affordance Sheet, an organised collection of good practices that can facilitate fair comparisons and the development of new solutions (see the Table below). Our Affordance Sheet includes Model Cards and adds sections that complement the released information.