AI, robotics and crypto, in context Deutsch

Robotics

Robots Have a Data Problem: Where Training Examples Come From

Language models learned from text people had already written. Robots have no such archive. How teams gather training data from demonstrations, simulation and working fleets, and why it stays scarce.

Robots Have a Data Problem: Where Training Examples Come From

Deep learning has a simple appetite: it needs examples, and lots of them. Image recognition improved dramatically once large collections of labeled photos became available. Language models grew out of the vast amount of text people had already put online. In both cases, most of the training data existed before anyone set out to build the model.

Robots have no such archive. There is no internet of robot movements, no huge store of recorded grasps with forces and outcomes attached. Every useful example has to be produced on purpose, by a robot, a person or a simulator. That makes data, more than algorithms or computing power, the main bottleneck for learned robot skills.

This article looks at where robot training data comes from today, what each source is good for and why even well-funded teams struggle to get enough of it.

What robot training data actually contains

A training example for a robot is more than a picture. For a learned grasping or assembly skill, it typically includes camera images, sometimes depth data, the position of every joint, the commands the robot received and often the forces it felt at the wrist or fingers. It also needs a label or signal that says whether the attempt worked.

All of these streams have to be recorded together and kept in sync. A camera frame that is a fraction of a second out of step with the joint data can teach the model the wrong link between seeing and acting. Data from different robots also arrives in different formats, with different sensors, mounting positions and control rates.

That is why robot data is expensive even before the first example is collected. Teams need a recording pipeline, storage, calibration routines and tools to inspect and clean what they gathered.

Demonstrations: good data, slowly

The most direct source of data is a person showing the robot what to do. In teleoperation, an operator controls the robot remotely with a joystick, a handheld controller, a virtual reality headset or a second, smaller arm that the main robot mirrors. Every movement is recorded along with the camera views.

Demonstration data is valuable because it shows a task done properly, by the actual robot, in a real setting. But it is slow. An operator can record only so many attempts per hour, and fatigue lowers quality. Complex tasks, such as folding clothes or plugging in cables, may need many demonstrations across different objects, positions and lighting before a model generalizes.

There are subtler problems too. Different operators solve the same task in different ways, which can confuse a model. Teleoperation often lacks a sense of touch, so demonstrations of delicate, force-sensitive work tend to be clumsy. And people rarely demonstrate how to recover from mistakes, although recovery is exactly what robots need most.

Simulation: data at scale, with a gap

Simulation promises to solve the volume problem. In a virtual world, thousands of robot copies can practice in parallel, day and night, without wear or risk. Trial and error, which would be slow and dangerous on real hardware, becomes cheap.

The catch is known as the reality gap. Simulators handle rigid objects and simple motion well. They struggle with friction, contact between many surfaces, cables, cloth, liquids and soft packaging. A skill that works perfectly in simulation can fail on a real robot because the physical world behaves slightly differently.

Engineers narrow the gap in several ways. They randomize colors, lighting, object shapes and physical parameters during training, so the model learns to cope with variety instead of memorizing one setting. They also calibrate simulations against real measurements and mix simulated data with a smaller amount of real data. These methods help, but for tasks with a lot of contact, real-world data usually remains necessary.

Text and images were lying around waiting to be used. Robot data has to be made, one attempt at a time.

Other sources: human video, working fleets, shared datasets

Three further sources are attracting attention, each with its own trade-offs.

  • Video of people. Recordings of humans cooking, assembling or tidying are plentiful. They show what tasks look like and in what order steps happen. But a human hand is not a robot gripper, and video contains no force data. Models can learn goals and sequences from video, but not the exact movements a robot needs.
  • Data from robots in operation. Robots already working in warehouses or factories generate data every day. This data reflects real conditions and real variety. However, it is dominated by routine successes, while the rare, difficult cases are the ones a model most needs.
  • Shared datasets. Research groups have started pooling demonstrations from many different robots into common collections. Models trained on such mixtures can sometimes transfer skills between machines, although differences in body shape, sensors and control remain a hurdle.

Many teams combine these sources. Video and simulation provide broad background knowledge, demonstrations add precise examples, and fleet data tunes the model to real conditions.

Why the shortage persists

Several structural reasons keep robot data scarce. Every robot body is different, so data from one arm does not simply apply to another. Physical attempts take real time, unlike generating text. Safety rules limit how freely a robot may experiment near people or expensive equipment. And in homes, hospitals or offices, recording cameras raises privacy questions that limit what can be collected and kept.

Labeling adds cost as well. Deciding whether a grasp succeeded may be easy. Judging whether a cable was routed correctly or a garment folded neatly often needs a person to review the recording.

What this means for companies using learned robots

For businesses deploying robots with learned skills, data strategy matters as much as the choice of model. The useful habits are fairly simple. Record what the robots see and do in a consistent format. Keep the failures, not just the successes. Log every time an operator has to step in, because those interventions show precisely where the model falls short and can become new training examples.

Data also has a shelf life. A new product, a packaging change or a moved camera can make old data less relevant. Teams that plan for regular collection and retraining tend to keep their robots useful longer than those that train once and hope the world stays still.

The data gap will likely narrow as fleets grow, simulators improve and shared datasets expand. For now, the teams that make progress fastest usually treat data collection as engineering work in its own right, with budgets, tools and people dedicated to it.

Questions and answers

Why can't robots learn from internet data the way language models do?

Internet text and images show what the world looks like, but not how a specific robot should move or how much force to apply. Robot learning needs examples that link perception to action and outcome. Such data rarely exists online and has to be recorded deliberately.

What is teleoperation in robot training?

In teleoperation, a person controls the robot remotely while the system records camera images, joint positions and commands. The recordings then serve as demonstrations for the model to imitate. It produces high-quality data but takes a lot of human time.

What is the reality gap?

The reality gap is the difference between how things behave in a simulator and in the physical world. Friction, contact and soft materials are especially hard to simulate accurately. Skills trained only in simulation may therefore fail on real hardware.

Can data from one robot be used to train another?

Sometimes, but rarely directly. Robots differ in arms, grippers, sensors and control systems. Shared datasets show that some knowledge transfers between machines, but adapting a skill to a new robot usually still needs data recorded on that robot.

What data should a company keep from its robots?

It helps to keep consistent records of sensor data, commands and outcomes, including failures. Moments when an operator had to intervene are especially valuable, because they show where the model struggles. Privacy and data protection rules should be checked before recording in shared spaces.