Building Blocks of AI: From Digital Bricks to Real-World Creations

Picture this: AI agents that can read blueprints and construct real-world objects using reusable building blocks. This isn't just a sci-fi dream anymore. Researchers are diving into whether multimodal large language models (MLLMs) have what it takes to handle such complex tasks, starting with brick assembly.

Understanding the Challenge

Brick assembly is more than just stacking one piece on top of another. It's a sequential decision-making problem. Each step involves picking the right brick from a bunch of options and figuring out exactly how to place it. It's like a 3D puzzle where precision is key.

To test this, researchers introduced BC-Bench, a benchmark specifically for evaluating MLLMs on assembly tasks using a variety of bricks. The results? Current state-of-the-art MLLMs are still struggling. They can't reliably select the right bricks or estimate where they should go. It's clear we've got a long way to go before these models become true builders.

Enter Brick-Composer

So, how do we make these models better? Enter Brick-Composer. This learning framework is designed to teach MLLMs the art of assembly using three main tools: Human Design Sparks, World Feedback, and Synthetic Experience.

Human Design Sparks offer rich demonstrations of how to construct objects, providing a starting point for AI learning. World Feedback grounds the AI's actions in their visual and physical outcomes, allowing them to learn from mistakes. Lastly, Synthetic Experience pushes learning beyond existing object designs, opening up new possibilities.

And it's working. Brick-Composer improved brick selection accuracy by more than threefold and reduced pose estimation errors. The strict step-level assembly success rate jumped from less than 1% to around 15%, a massive leap forward.

What's Next for AI Assembly?

After training, a Qwen-3-8B model could accurately compose up to 42% of the steps needed to complete an object. This suggests that with the right training, MLLMs can indeed improve their assembly capabilities.

But here's the big question: Will AI ever fully master physical assembly? The tech is moving fast, but there are still hurdles to overcome. The builders never left, and they're still tinkering away, but the journey from digital to tangible is no small feat.

MLLMs might not be the perfect builders yet, but they're laying the foundation for a future where AI can create and construct in the real world. And that's a future worth building towards.