Learning Egocentric In-Hand Object Segmentation through Weak Supervision from Human Narrations

Pixel-level recognition of objects manipulated by the user from egocentric images enables key applications spanning assistive technologies, industrial safety, and activity monitoring. However, progress in this area is currently hindered by the scarcity of annotated datasets, as existing approaches rely on costly manual labels.

In this paper, we propose to learn human-object interaction detection leveraging narrations – natural language descriptions of the actions performed by the camera wearer which contain clues about manipulated objects (e.g., "I am pouring vegetables from the chopping board to the pan"). Narrations provide a form of weak supervision that is cheap to acquire and readily available in state-of-the-art egocentric datasets.

We introduce Narration-Supervised in-Hand Object Segmentation (NS-iHOS), a novel task where models have to learn to segment in-hand objects by learning from natural-language narrations. We showcase the potential of the task by proposing Weakly-Supervised In-hand Object Segmentation from Human Narrations (WISH), an end-to-end model distilling knowledge from narrations to learn plausible hand-object associations and enable in-hand object segmentation without using narrations at test time.

We benchmark WISH against different baselines based on open-vocabulary object detectors and vision-language models, showing the superiority of its design. Experiments on EPIC-Kitchens and Ego4D show that WISH surpasses all baselines, recovering more than 50% of the performance of fully supervised methods, without employing fine-grained pixel-wise annotations.

The Architecture of WISH: Our model operates in two stages sharing a common backbone. (a) An object segmenter and a CLIP-based backbone extract visual embeddings for all object and hand proposals. (b) In Stage 1, we learn a shared embedding space to align hand-specific noun phrases from narrations with their corresponding visual object embeddings. (c) In Stage 2, we generate pseudo-labels from this alignment to train two specialized heads: a Contactness head (C) and a Matching head (M). At test time, only the backbone and Stage 2 are used for narration-free in-hand object segmentation.

If you use this work in your research, please cite:

Coming soon!