TouchAnything

Overview

Abstract

Vision has become the dominant sensing modality for hand-object interaction, yet it remains insufficient for dexterous manipulation as it cannot directly capture contact forces, pressure distributions, or subtle contact state changes. While tactile sensing provides these critical cues, high-quality tactile hardware is difficult to deploy at scale, motivating growing interest in vision-to-touch prediction, which aims to infer tactile feedback from visual observations. However, progress in this area is fundamentally constrained by the lack of large-scale, diverse, and high-fidelity datasets. Existing datasets are often limited in scale, restricted to single-view capture or narrow interaction scenarios, or lack dense tactile ground truth. In particular, egocentric single-view observations suffer from severe occlusion of hand-object contact regions, making accurate tactile inference highly challenging. To address these limitations, we introduce EgoTouch, a large-scale multi-view tactile dataset for egocentric hand-object interaction. EgoTouch comprises 302 diverse manipulation tasks spanning 4,530 episodes across both indoor and outdoor environments, with synchronized multi-view observations (egocentric and dual wrist-mounted cameras), bimanual 3D hand pose annotations, and dense continuous pressure maps captured by wearable tactile sensors. This comprehensive setup enables learning cross-view consistent representations of contact dynamics under realistic occlusion and interaction variability. Building upon EgoTouch, we further propose TouchAnything, a unified architecture for multi-view tactile prediction. The model integrates a shared DINOv2 encoder, learnable view embeddings, cross-view attention, and a view dropout strategy, enabling flexible inference under varying view availability. Effectively leveraging multi-view observations is non-trivial due to view inconsistency and partial observability, which our design explicitly addresses. Extensive experiments demonstrate that incorporating complementary wrist views is crucial for resolving occluded contact regions and enabling reliable tactile inference, particularly under severe occlusions. These results highlight the importance of multi-view perception for tactile reasoning in real-world interaction scenarios. We will publicly release the dataset, code, and benchmark to facilitate future research bridging visual perception and physical interaction in embodied systems.

Key Features

Dataset Statistics

Dataset Comparison

EgoTouch is the first dataset to jointly provide multi-view video, bimanual hand pose, and real dense pressure data across diverse scenes.

Main Contributions

• First large-scale multi-view tactile dataset for egocentric hand-object interaction, comprising 302 tasks, 4,530 episodes, bimanual pose, and dense continuous pressure maps across diverse indoor and outdoor scenes
• First multi-view tactile prediction benchmark, including evaluation protocols that quantify the role of complementary wrist views and show clear gains under severe occlusion
• New multi-view tactile prediction architecture with shared vision encoding, cross-view attention, and view dropout, enabling flexible inference with any combination of available views

Data Collection Setup

Multi-sensor data collection system with head-mounted egocentric camera, dual wrist cameras, and pressure-sensing gloves.

Our data collection system integrates:

• Head-mounted Wide-Angle Camera: Captures global manipulation context from a wide-field first-person perspective
• Dual Wrist Cameras: Observe hand-object contact regions to overcome occlusion
• Pressure-Sensing Gloves: Record dense 16×16 pressure maps on each palm
• Motion Capture: Tracks 42-joint bimanual 3D hand pose at 30Hz
• Temporal Synchronization: All modalities aligned with millisecond precision

Example Multi-Modal Data

Example showing synchronized multi-view video, hand pose, and pressure maps revealing contact information invisible to vision alone.

Dataset Analysis

Comprehensive statistics showing diverse pressure patterns, balanced task coverage, and rich contact interactions.

Method Architecture

Multi-view tactile prediction architecture with shared vision encoding, cross-view attention, and pose-aware fusion.

We propose a multi-view tactile prediction model based on:

• Shared DINOv2 Vision Encoder: Extracts features from all camera views
• Learnable View Embeddings: Distinguishes between egocentric and wrist viewpoints
• Cross-View Transformer Attention: Fuses complementary information across views
• View Dropout Training: Enables robust inference with missing views

This design allows the model to work with any subset of available views at inference time, from ego-only to full multi-view, making it practical for real-world deployment.

Model Inference Results

Real-world tactile prediction results on diverse manipulation tasks. Our model accurately predicts contact regions and pressure distributions from multi-view video input.

These results demonstrate our model's ability to predict realistic tactile feedback across diverse manipulation scenarios, from static grasping to dynamic interactions.

Data Scaling Analysis

Performance improves consistently with more training data, demonstrating the model's ability to leverage large-scale datasets.

Qualitative Results

Tactile prediction results showing accurate contact region and pressure intensity prediction across diverse manipulation tasks.

Applications

Core Applications

Broader Impact

Author Contributions

This work represents a collaborative effort across multiple domains, from hardware setup to model development and data collection.