A Good Model Is Not a Good System: What Changes When AI Vision Leaves the Lab

A vision model that performs well in a demo has solved one part of the problem. Lê Thái Sơn, Project Manager at Solomon Technology Corp., calls the rest "industrial AI engineering," and he says it is the major difference between that field and AI research. He has worked for ten years on AI vision and robotics for factory automation.

His talk in the episode AI-Powered Robotics: How Physical AI & Embodied AI Will Transform How We Live and Work contains no code, framework names or benchmarks. What it offers is a systems-level view of why models that look good in demos struggle on factory floors. This article summarizes that view for engineers. Anything that goes beyond the episode is marked as my own framing.

Sơn's example is a robot picking from a bin of mixed products:

He describes this as three key components: perception, cognition and action, or simply seeing, understanding and acting. He calls that the foundation of embodied AI. Throughout the talk, the unit he evaluates is this whole chain, not any single stage.

Sơn's illustration is a lab system with accuracy of around 80–90%, where "the demo looks great." In the factory, he says, the situation is completely different:

From this he draws a conclusion: "a good AI model does not necessarily make a good automation system." The Q&A repeats the point from another angle. Claiming "My model achieves 99%," he says, is one of the most common mistakes, because that figure holds under ideal conditions.

In ML terms, and this is my framing, most items on the list are variations in the input data. They include shifted lighting, noise from vibration and dust, out-of-distribution products, and timing that changes with line speed. The model's inputs are no longer controlled, so the accuracy measured on a clean dataset is not the number that matters in production.

Sơn gives a set of system-level questions that go beyond accuracy:

Grouped by concern (again my framing), these cover sensing stability, timing, actuation reliability, and behavior under uncertainty and failure. Only the first group is about the model, and it is the one with the least say in the last two.

Sơn lists computing power as one of the reasons Physical AI has become practical. Many AI tasks once had to run on large server systems, he says. Today they can run on small devices, including edge devices, with edge AI operating directly on the factory floor at very low latency.

That connects to his cycle-time question. My reading is that where inference runs is therefore a design decision tied to how fast the line moves. The episode does not go further than this, and it gives no latency figures.

The same hypothetical from the talk applies here: a one-point accuracy gain, from 95% to 96%, might require a very large GPU system and more complex servicing and maintenance. Sơn contrasts that with improvements to the traditional setup, such as redesigning the lighting, changing the camera position or improving the mechanical setup, which may be much better and far more stable. His practical rule is to ask what the customer's actual problem is before deciding whether to improve the AI or the engineering around it.

For engineers, the useful reading is that sensing and mechanics belong to the input pipeline. They are not background conditions to work around.

Sơn argues for balance, not replacement. Traditional automation, he says, must keep providing stability and reliability, while the AI contributes its adaptive capabilities. A system that only does what it was programmed to do struggles when products sit in random positions, vary in shape, or arrive in jumbled arrangements. AI vision helps there. Conventional PLC and industrial-robot control remains what makes the system dependable.

Digital twins and simulation appear in both parts of the episode. In the talk, Sơn says robot simulation platforms let teams build virtual factories, train robots and test AI algorithms before deployment, which helps with stability and safety. In the Q&A he describes running the system in a simulation that reflects real-world conditions: once a problem is solved there, deploying in the real world carries far less risk.

He also recommends assessing the real environment in advance, covering temperature, humidity, dust, smoke, vibration and surrounding magnetic fields. In his words, the more that can be anticipated, the more the cost and risk of deployment can be reduced.

Sơn's advice to young engineers is practical: "Do not learn AI only through a computer screen." He suggests connecting AI to peripheral devices such as cameras, sensors and motors, and programming and controlling a robotic system or a sensor system running on a device. Only then, he says, will you see "the huge gap between 'the model works' and 'the system works.'"

In the Q&A he adds that engineers should broaden out from a single specialty. They should understand hardware, software, cloud, servers, databases and front-end and back-end development, and he notes that they can learn all of this with AI right now.

The episode's engineering argument comes down to a change of unit. The thing to evaluate is the whole chain, from sensor to actuator, including what happens when something goes wrong. Model accuracy is one input to that evaluation. Watch the full episode with Lê Thái Sơn