Generative AI can create visually stunning 3D models from a simple text prompt, but there is a persistent gap between how an object looks on a screen and how it functions in the real world [S1, S5]. An AI-generated mug might look realistic but feature a sealed top or walls too thin to hold liquid [S1, S5].
This happens because current AI models understand visual plausibility but lack an understanding of physical functionality and manufacturability [S1, S4]. For users without professional CAD (Computer-Aided Design) skills, fixing these structural errors is often impossible [S1, S5].
To bridge this gap, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), Google, and Northeastern University developed InstructMesh [S1, S4]. This tool allows both novices and experts to repair AI-generated models using natural language before they are sent to a 3D printer [S1, S2].
How InstructMesh Combines Reasoning and Geometry
InstructMesh does not rely on a single AI system. Instead, it pairs Microsoft’s TRELLIS system, which generates 3D models from text and images, with the large language model GPT-4 [S1, S2].
While TRELLIS handles the visual creation, GPT-4 provides the reasoning skills necessary to interpret user requests [1]. This combination allows the software to understand a user’s description of a problem and translate it into a specific geometric change [S2, S4].
Unlike traditional tools that might require regenerating an entire object to fix one error, InstructMesh applies local modifications [4]. Users can highlight a specific problematic region and describe the fix in everyday language [S2, S4]. For those needing more precision, the interface includes sliders to adjust specific tweaks, such as extruding a part or changing its size [S1, S6].
Turning Visual Concepts into Functional Objects
The researchers tested the system by creating objects that required both creativity and physical utility [S1, S4]. These included a mug enveloped by a dragon where the tail serves as the handle, a shell-shaped blue whistle, and butterfly-wing glasses [S1, S2].
Beyond household items, the tool demonstrated utility in specialized fields [S1, S6]. MIT scientists fabricated a knee brace with a denim-like appearance to match a patient’s clothing [S1, S4]. They also created a “bristle bot” resembling a colorful shrimp, using the software to design a clever enclosure that houses a motor and wireless components [S1, S2].
Democratizing 3D Design for Non-Experts
To test if the tool actually helps beginners, the team had TRELLIS recreate popular models from Thingiverse [S1, S2]. They discovered that nearly 80% of the generated models contained structural flaws [S1, S4].
When novice users were asked to identify and fix these issues using InstructMesh, they succeeded approximately 90% of the time, as verified by experts [S1, S2]. This suggests that while beginners lack formal CAD training, they possess enough intuition to spot functional errors when given an accessible interface [S1, S2].
By moving manipulation into the latent space of the generative model, InstructMesh removes the need for deep domain expertise in mesh-editing software [S1, S4].
The Future of Contextual Fabrication
While the current version focuses on visible static geometric defects, future iterations may integrate physics simulations [S1, S6]. This would allow the software to predict how a design reacts to use, such as determining if a bowl would break when dropped or suggesting the best materials for the job [S1, S6].
There are also plans to incorporate the technology into augmented reality (AR) platforms [1]. This would enable users to prompt the system based on their physical surroundings, allowing them to design and print objects that match their environment in real time [1].
Sources
- New tool lets users repair AI-generated 3D models, then fabricate them …
- MIT Develops InstructMesh to Repair AI-Generated 3D Models | 3DHaber
- New software tool lets users repair AI-generated 3D models, then …
- MIT Researchers Let Users Repair AI-Generated 3D Models Before Printing
- InstructMesh: MIT’s AI repairs 3D models for production
- New AI Tool Helps People Fix 3D Designs Before Printing Them
- Artificial intelligence | MIT News | Massachusetts Institute of Technology
- New software tool lets users repair AI-generated 3D models, then …
