Robot AI predicts future scenes to guide its next moves

edited by Gaby Clark, reviewed by Robert Egan

This article has been reviewed according to Science X's editorial process and policies. Editors have highlighted the following attributes while ensuring the content's credibility:

Add to Preferred Sources Block stacking was evaluated separately under a setting with no required stacking order and a setting requiring blocks to be stacked in a specified color sequence. Credit: Seoul National University College of Engineering A robot artificial intelligence (AI) system has been developed that can understand human instructions, predict what a scene will look like after an action and then carry out the task. Rather than simply translating instructions into physical movements, the technology also uses the anticipated outcomes of those actions to control the robot.

A research team led by professor Jaeyoung Do of the AIDAS Lab in the Department of Electrical and Computer Engineering at Seoul National University College of Engineering has developed "Dynin-Robotics," a robotic foundation model that handles language, images, video and robot actions together within a single model. A foundation model is a general-purpose AI model trained on diverse types of data so that it can be applied to a wide range of tasks. The paper is published on the arXiv preprint server.

Dynin-Robotics generates robot movements based on human instructions and camera images. It can also predict, in image form, what the next scene will look like after a particular action or what the target scene should look like once a task is completed, and use these predictions to generate and select actions. In evaluations across four real-world robot manipulation settings, Dynin-Robotics achieved an average success rate of 78.4%, outperforming the comparison models from NVIDIA and U.S.-based robot AI company Physical Intelligence used in the experiments.

The paper describing the research has been accepted to NeurIPS 2026, one of the world's leading international conferences in artificial intelligence and machine learning, and is scheduled to be presented in December.

Robot hardware is advancing rapidly, but understanding new instructions and carrying out tasks in unfamiliar environments remains a major challenge. Unlike repeating movements learned under predetermined conditions, real-world tasks involve objects whose positions and shapes change from one situation to another, along with varying requests from people. For example, to carry out the instruction "Put the cup on the desk into the drawer and close it," a robot must identify the locations of the cup and drawer and determine a sequence of actions that includes moving the cup and then closing the drawer.

Recent research has explored using the language-understanding capabilities of large language models such as ChatGPT for robot control, as well as using video-generation models to predict future scenes. To use both the ability to understand human instructions and the ability to predict the outcomes of actions in generating robot movements, the research team integrated these capabilities into a single model.

The new technology extends "Dynin-Omni," an omnimodal AI model previously developed by the research team, to robot control. Omnimodal AI refers to artificial intelligence that can understand and generate multiple forms of information—including language, images, video and audio—within a single model. By adding robot-action data to this framework, the researchers designed a physical AI model capable of carrying out real-world tasks such as moving and stacking objects.

A key feature of Dynin-Robotics is that it jointly learns to understand instructions and scenes, predict the outcomes of actions and generate physical movements. The model can also describe in sentences what a robot has done based on video of the task.

The researchers demonstrated the benefits of this unified learning approach in simulated "fruit picking" and "flower arrangement" tasks. A model trained jointly on understanding, prediction and action generation was better at carrying out instructions expressed in unfamiliar ways than a model trained only on action generation. It responded not only to instructions that directly specified an object, such as "Put the rose in," but also to instructions that conveyed the intended object indirectly through meaning, such as "Put in the flower that symbolizes love."

Future-scene prediction is used to help the robot decide what action to take. Just as a person might envision where an object will be placed and what the result will look like before moving it, the robot can generate an image of the expected scene after completing a task based on the current scene and the instruction. The research team implemented methods that use the predicted target scene either as a reference for generating actions or to evaluate candidate actions and select the more appropriate one.

The researchers also reduced the computation time required for action generation. They applied an approach that groups information needed for a sequence of actions and predicts it in parallel. Information repeatedly used during computation is also reused to improve the efficiency of generating the robot's next action. The model's internal action-generation speed improved by up to approximately 29.2 times compared with the unoptimized version and was up to approximately 36.6 times faster than Physical Intelligence's "π0.5."

In real-world robot experiments, the team evaluated four manipulation settings: moving fruit, sorting blocks by color, stacking blocks without a required order and stacking blocks in a specified color sequence. Averaged across the four settings, NVIDIA's "GR00T N1.6" achieved a success rate of 73.0%, NVIDIA's "Cosmos Policy" achieved 73.6%, and Physical Intelligence's "π0.5" achieved 76.9%. Dynin-Robotics recorded 78.4%, the highest average success rate among the comparison models used in the evaluation.

The technology could be applied to tasks that require robots to consider both human instructions and their surroundings, such as sorting and transporting objects in logistics operations or handling components in manufacturing processes. However, the current results were validated in simulations and predetermined real-world robot experimental environments. Before the technology can be deployed in industrial settings, its performance will need to be evaluated across a wider variety of objects, tasks and operating environments.

Building on the Dynin foundation model series, the research team plans to evaluate its applicability to component handling, assembly, object sorting and other tasks in manufacturing and logistics environments. The team also plans to further develop domain-specific models through additional training tailored to the data and requirements of the manufacturing and medical fields.

To this end, the researchers are developing technologies for precise manipulation using robotic hands, recognition of contact states using tactile and force sensors, and rapid adaptation to new tasks and environments. They are also pursuing joint research with researchers at Seoul National University College of Medicine on applying AI to surgical-assistance robots and gastrointestinal endoscopic robots.

The team is also conducting research on AI agents that divide a human-defined goal into multiple tasks, assign those tasks to robots, evaluate the results and adjust subsequent actions accordingly. For example, when given a request such as "Clear the dining table," an AI agent could break the request down into the necessary tasks and pass them to the robot, evaluate the results, and have the robot retry failed tasks using a different approach.

By drawing on the accumulated experience of both successful and failed attempts, the researchers plan to develop what they describe as a "robot that gets smarter the more it is used," continuously improving its task-planning and execution capabilities.

Do, who supervised the research, said, "This study is significant in that a single model jointly handles what a robot should do, how its surroundings will change after it moves, and what actions it should actually take." He added, "Building on this achievement, which extends Dynin-Omni's unified AI technology to robot control, we will continue developing robot intelligence that moves beyond experimental settings to understand human instructions and perform challenging tasks in real-world environments."

Hoeun Lee et al., Dynin-Robotics: Omnimodal Unified Diffusion Vision-Language-Action Model, arXiv (2026). DOI: 10.48550/arxiv.2609.13053 Journal information: arXiv

MA in English, copy editor since 2021 with experience in higher education and health content. Dedicated to trustworthy science news. Full profile →

Bachelor's in mathematical biology, Master's in creative writing. Well-traveled with unique perspectives on science and language. Full profile →