A New Data Paradigm in the Age of Physical AI

1. Physical AI is the next major frontier of artificial intelligence.

Over the past few years, AI capability has shown rapid advancements, with frontier models experiencing a 30% improvement on their performance on Humanity’s Last Exam. Progress has been unprecedented across a variety of digital tasks, including mathematical reasoning and software engineering, as well as language understanding, visual reasoning, and science. At the same time, investment and interest in physical AI and embodied intelligence has been growing, foreshadowing a new frontier for this technology.

Earlier this week, Jensen Huang predicted that “hardware-in-the-loop”–whereby AI systems connect with physical hardware to enable real-world interaction and feedback–would be the “second big breakthrough” for the AI industry. OpenAI on the same day announced expansion beyond language and software tasks into physical engineering, demonstrating the capabilities of AI to design, simulate, and help manufacture physical products. Extending AI capabilities beyond the digital realm would open up exciting new opportunities outside of software, enabling models that can perceive, understand, and interact with the real world. The potential applications for physical AI are remarkably broad, spanning everything from robotics and industrial automation to telecommunications, drug discovery, and more.

2. Physical AI entails new capabilities

In order to unlock the new frontier of Physical AI, models may need to master a fundamentally new type of understanding than what it has thus far. Success in pattern recognition and symbolic domains in the digital space does not transfer cleanly to the physical world, which requires the ability to reason about geometry, spatial relationships, materials, forces, and so on. It’s one thing to create a 3D simulation on a piece of software, but another to recreate something in the physical world that adheres to the laws of physics, for example.

3. New types of training data are required

Understanding the physical world requires models to learn not only what objects look like, but how they are structured, how they behave, and how they respond to changes in their environment. Much of this knowledge is captured in archives of engineering and design records, simulations, and real-world observations–things that are typically proprietary and fragmented across firms, making them difficult to access at scale.

While significant in their role in advancing 3D AI research, existing public datasets such as ShapeNet, Objaverse, and DeepCAD nonetheless remain limited by inconsistent quality, insufficient diversity, and a lack of complex engineering information, highlighting the need for a greater diversity of higher-quality datasets with richer engineering context that better represent real-world use cases.

Without sufficient context around design histories, high-quality geometric representations, and the relationships between engineering features, AI models risk learning superficial patterns instead of the underlying principles of how objects are designed and function. Researchers have argued that these shortcomings are fundamental limitations of existing data representations, creating a ceiling on what models can learn that cannot be overcome simply by increasing dataset size or model sophistication.

Synthetic data and simulations can help address these gaps, but they ultimately rely on assumptions, models, and parameters derived from real-world data, thus underscoring the importance of accessing real-world data to ground AI systems.

4. Beyond Physical AI: structured 3D data in foundation model training

The potential value of structured 3D data may extend beyond specialized engineering applications. Research such as CAD-Llama has already demonstrated how continued pretraining on parametric CAD sequences can improve models' ability to generate and manipulate 3D designs. Whether similar approaches could improve (or have improved) general-purpose foundation models remains an open question.

Unlike images and text, CAD and BIM files can explicitly encode geometric relationships, engineering constraints, and component dependencies. Incorporating these representations into multimodal training could provide models with complementary information about the physical world. This possibility also raises broader questions about how different modalities contribute to increasingly generalizable world representations, as explored by the Platonic Representation Hypothesis.

Arloh as the data infrastructure provider for Physical AI

To advance the next generation of AI models and physical AI, models need access to high-quality data that captures the structure and complexity of the physical world. Arloh bridges this gap by connecting proprietary engineering, design, and operational datasets with the AI companies building the next generation of physical intelligence.

The digital AI revolution was enabled in part by unprecedented access to internet-scale information. The next frontier may depend on unlocking a different kind of knowledge. Much of this knowledge already exists, but remains fragmented across proprietary archives and inaccessible to AI developers. Arloh works with data owners, researchers, and AI developers to help advance physical intelligence. If you're working in this space, we'd love to hear from you.