OpenAI makes it easier to create 3D objects from text prompts

The next breakthrough to take the Artificial Intelligence (AI) world by storm might be AI 3D model generators. This week, OpenAI the company behind the world famous ChatGPT and GPT-4 AI models quietly open sourced Point-E, a machine learning system that creates any 3D object using nothing more than a text prompt.

According to a paper published alongside the code base, Point-E can produce 3D models in one to two minutes on a single Nvidia V100 GPU.

Point-E doesn’t create 3D objects in the traditional sense. Rather, it generates point clouds, or discrete sets of data points in space that represent a 3D shape, hence the cheeky abbreviation.

The Future of Synthetic Content, by keynote Matthew Griffin

The “E” in Point-E is short for “Efficiency,” because it’s ostensibly faster than previous 3D object generation approaches.

Point clouds are easier to synthesize from a computational standpoint, but they don’t capture an object’s fine-grained shape or texture, a key limitation of Point-E currently.

To get around this limitation, the Point-E team trained an additional AI system to convert Point-E’s point clouds to meshes, the collections of vertices, edges and faces that define an object, which are commonly used in 3D modelling and design. But they note in the paper that the model can sometimes miss certain parts of objects, resulting in blocky or distorted shapes.

Outside of the mesh-generating model, which stands alone, Point-E consists of two models: a text-to-image model and an image-to-3D model.

The text-to-image model, similar to generative art systems like OpenAI’s own DALL-E 2 and Stable Diffusion, was trained on labelled images to understand the associations between words and visual concepts. The Image-to-3D model, on the other hand, was fed a set of images paired with 3D objects so that it learned to effectively translate between the two.

When given a text prompt, for example, “a 3D printable gear, a single gear 3 inches in diameter and half inch thick”, Point-E’s text-to-image model generates a synthetic rendered object that’s fed to the image-to-3D model, which then generates a point cloud.

After training the models on a dataset of “several million” 3D objects and associated metadata, Point-E could produce colored point clouds that frequently matched text prompts, the OpenAI researchers say. It’s not perfect, Point-E’s image-to-3D model sometimes fails to understand the image from the text-to-image model, resulting in a shape that doesn’t match the text prompt. Still, it’s orders of magnitude faster than the previous state-of-the-art, at least according to the OpenAI team.

“While our method performs worse on this evaluation than state-of-the-art techniques, it produces samples in a small fraction of the time,” they wrote in the paper. “This could make it more practical for certain applications, or could allow for the discovery of higher-quality 3D object.”

What are the applications, exactly?

Well, the OpenAI researchers point out that Point-E’s point clouds could be used to fabricate real-world objects, for example through 3D printing. And, with the additional mesh-converting model the system could also, once it’s a little more polished, find its way into game and animation development workflows to help create games, virtual assets and virtual worlds, and much more.

OpenAI might be the latest company to jump into the 3D object generator fray, but, as alluded to earlier, it certainly isn’t the first. Earlier this year, Google released DreamFusion, an expanded version of Dream Fields, a generative 3D system that the company unveiled back in 2021. Unlike Dream Fields, DreamFusion requires no prior training, meaning that it can generate 3D representations of objects without 3D data.

While all eyes are on 2D art generators at the present, model-synthesizing AI could be the next big industry disruptor. 3D models are widely used in film and TV, interior design, architecture and various science fields. Architectural firms use them to demo proposed buildings and landscapes, for example, while engineers leverage models as designs of new devices, vehicles and structures.

3D models usually take a while to craft, though, anywhere between several hours to several days. AI like Point-E could change that if the kinks are someday worked out, and make OpenAI a respectable profit doing so.

The question is what sort of intellectual property disputes might arise in time. There’s a large market for 3D models, with several online marketplaces including CGStudio and CreativeMarket allowing artists to sell content they’ve created. If Point-E catches on and its models make their way onto the marketplaces, model artists might protest, pointing to evidence that modern generative AI borrows heavily from its training data, existing 3D models, in Point-E’s case. Like DALL-E 2, Point-E doesn’t credit or cite any of the artists that might’ve influenced its generations.

But OpenAI’s leaving that issue for another day. Neither the Point-E paper nor GitHub page make any mention of copyright.

To their credit, the researchers do mention that they expect Point-E to suffer from other problems, like biases inherited from the training data and a lack of safeguards around models that might be used to create “dangerous objects.” That’s perhaps why they’re careful to characterize Point-E as a “starting point” that they hope will inspire “further work” in the field of Text-to-3D model synthesis.