Vision-language-action models have handed robots a surprising amount of general competence. Trained on oceans of images, video and text, then fine-tuned on teleoperated demonstrations, they can fold laundry, tidy a living room and work kitchen gadgets from nothing but a camera feed and a plain-English instruction. Ask one to plug in a USB cable or turn a key in a lock, though, and it falls apart. The missing ingredient is touch — and as IEEE Spectrum reported on September 10, 2026, academic labs and startups are now racing to build the tactile datasets that could supply it.
“Most dexterous manipulation can be done by humans with their eyes closed,” says Trevor Darrell, professor of computer science at the University of California, Berkeley. “Understanding force, slip, and precise grasping is not something that can be done well with traditional vision sensors.”
Darrell’s group tackled that with T-Rex, posted to arXiv on June 15, 2026. The recipe: pretrain on existing datasets, then bolt on a specialist submodel trained on 100 hours of purpose-collected tactile data covering wiping, grasping, twisting and pouring across more than 200 household objects. The hard part was speed, because a grip correction has to land faster than a vision-language model normally thinks. So T-Rex splits the work between two “experts”: an action expert that produces motion plans, and a tactile expert running four times faster that nudges those plans in real time based on what the robot is feeling. Fine-tuned on about 100 teleoperated demonstrations of jobs like screwing in a light bulb, applying toothpaste to a toothbrush or transferring an egg between trays, it averaged a 65 percent success rate across 12 tasks.
Its limitation is that all that data came from one robot. Hands range from five-finger articulated designs to simple pincers, and tactile sensors use wildly different physics — resistance changes in one, images of a deforming gel pad in another. That makes most tactile AI sensor-specific, says Chengbo Yuan, a master’s student at Tsinghua University in Beijing. His answer, FTP-1 (arXiv, June 11, 2026), pools more than 3,000 hours of tactile robotic data from publicly available datasets covering 21 sensor types and a variety of embodiments, translating every sensor’s output into a shared format mapped onto labeled positions on a template of a human hand. It beat a baseline even on hardware it had never met. Yuan now leads an 80-institution collaboration aiming at a much bigger standardized set.
Others are already chasing raw scale:
- Fudan University and its spin-out NeoteAI published N0-Foundation on July 25, 2026, built on NeoData — more than 30,000 hours of synchronized visual and tactile demonstrations, 1.4M episodes, six embodiments and 450+ tasks, gathered with a proprietary sensor on assorted robot arms and a human-operated handheld gripper. The resulting model doesn’t merely react to touch; it predicts what it should be feeling and uses the mismatch to guide and grade its own actions.
- The University of Southern California released a model that infers tactile information from vision alone, trained on more than 2,700 demonstrations recorded by a handheld gripper carrying both a camera and tactile sensors — a rudimentary sense of touch for robots that have no touch sensors at all.
- Long Cheng of the Chinese Academy of Sciences targets a subtler failure: vision is a continuous firehose of pixels while tactile signals are sparse and intermittent, so models learn to ignore them. His system predicts the expected touch from vision, compares it with the real signal, and amplifies the surprising readings while damping predictable ones. Across five contact-rich tasks it averaged 62.8% success versus 28.2% for the same model without touch. The work is due at IROS 2026, running September 27 to October 1, 2026 in Pittsburgh.
Nobody knows where the threshold sits. Yuan reckons touch has so far mainly made robots more efficient learners at tasks they could already manage, and suspects genuinely impossible manipulations will need new algorithms. Shunlin Lu, a postdoc at Fudan and CTO of NeoteAI, bets on volume instead — closer to 100,000 hours, collected in messy real-world settings rather than labs. “I think tactile intelligence is actually the next step for physical AI,” he says.
