Authors: Jun Gao, NVIDIA, University of Toronto, Vector Institute (jung@nvidia.com) Tianchang Shen, NVIDIA, University of Toronto, Vector Institute (frshen@nvidia.com) Zian Wang, NVIDIA, University of Toronto, Vector Institute (zianw@nvidia.com) Wenzheng Chen, NVIDIA, University of Toronto, Vector Institute (wenzchen@nvidia.com) Kangxue Yin, NVIDIA (kangxuey@nvidia.com) Daiqing Li, NVIDIA (daiqingl@nvidia.com) Or Litany, NVIDIA (olitany@nvidia.com) Zan Gojcic, NVIDIA (zgojcic@nvidia.com) Sanja Fidler, NVIDIA, University of Toronto, Vector Institute (sfidler@nvidia.com) Authors: Jun Gao, NVIDIA, University of Toronto, Vector Institute (jung@nvidia.com) Tianchang Shen, NVIDIA, University of Toronto, Vector Institute (frshen@nvidia.com) Zian Wang, NVIDIA, University of Toronto, Vector Institute (zianw@nvidia.com) Wenzheng Chen, NVIDIA, University of Toronto, Vector Institute (wenzchen@nvidia.com) Kangxue Yin, NVIDIA (kangxuey@nvidia.com) Daiqing Li, NVIDIA (daiqingl@nvidia.com) Or Litany, NVIDIA (olitany@nvidia.com) Zan Gojcic, NVIDIA (zgojcic@nvidia.com) Sanja Fidler, NVIDIA, University of Toronto, Vector Institute (sfidler@nvidia.com) Jun Gao, NVIDIA, University of Toronto, Vector Institute (jung@nvidia.com) Tianchang Shen, NVIDIA, University of Toronto, Vector Institute (frshen@nvidia.com) Zian Wang, NVIDIA, University of Toronto, Vector Institute (zianw@nvidia.com) Wenzheng Chen, NVIDIA, University of Toronto, Vector Institute (wenzchen@nvidia.com) Kangxue Yin, NVIDIA (kangxuey@nvidia.com) Daiqing Li, NVIDIA (daiqingl@nvidia.com) Or Litany, NVIDIA (olitany@nvidia.com) Zan Gojcic, NVIDIA (zgojcic@nvidia.com) Sanja Fidler, NVIDIA, University of Toronto, Vector Institute (sfidler@nvidia.com) Abstract As several industries are moving towards modeling massive 3D virtual worlds, the need for content creation tools that can scale in terms of the quantity, quality, and diversity of 3D content is becoming evident. In our work, we aim to train performant 3D generative models that synthesize textured meshes that can be directly consumed by 3D rendering engines, thus immediately usable in down-stream applications. Prior works on 3D generative modeling either lack geometric details, are limited in the mesh topology they can produce, typically do not support textures, or utilize neural renderers in the synthesis process, which makes their use in common 3D software non-trivial. In this work, we introduce GET3D, a Generative model that directly generates Explicit Textured 3D meshes with complex topology, rich geometric details, and high fidelity textures. We bridge recent success in the differentiable surface modeling, differentiable rendering as well as 2D Generative Adversarial Networks to train our model from 2D image collections. GET3D is able to generate high-quality 3D textured meshes, rang-ing from cars, chairs, animals, motorbikes and human characters to buildings, achieving significant improvements over previous methods. Our project page: https://nv-tlabs.github.io/GET3D G E T 3D https://nv-tlabs.github.io/GET3D 1 Introduction Diverse, high-quality 3D content is becoming increasingly important for several industries, including gaming, robotics, architecture, and social platforms. However, manual creation of 3D assets is very time-consuming and requires specific technical knowledge as well as artistic modeling skills. One of the main challenges is thus scale – while one can find 3D models on 3D marketplaces such as Turbosquid [4] or Sketchfab [3], creating many 3D models to, say, populate a game or a movie with a crowd of characters that all look different still takes a significant amount of artist time. 4 3 To facilitate the content creation process and make it accessible to a variety of (novice) users, generative 3D networks that can produce high-quality and diverse 3D assets have recently become an active area of research [5, 14, 43, 46, 53, 68, 75, 60, 59, 69, 23]. However, to be practically useful for current real-world applications, 3D generative models should ideally fulfill the following requirements: (a) They should have the capacity to generate shapes with detailed geometry and arbitrary topology, (b) The output should be a textured mesh, which is a primary representation used by standard graphics software packages such as Blender [15] and Maya [1], and (c) We should be able to leverage 2D images for supervision, as they are more widely available than explicit 3D shapes. 5 14 43 46 53 68 75 60 59 69 23 (a) (b) 15 1 (c) Prior work on 3D generative modeling has focused on subsets of the above requirements, but no method to date fulfills all of them (Tab. 1). For example, methods that generate 3D point clouds [5, 68, 75] typically do not produce textures and have to be converted to a mesh in post-processing. 1 5 Methods generating voxels often lack geometric details and do not produce texture [66, 20, 27, 40]. Generative models based on neural fields [43, 14] focus on extracting geometry but disregard texture. Most of these also require explicit 3D supervision. Finally, methods that directly output textured 3D meshes [54, 53] typically require pre-defined shape templates and cannot generate shapes with complex topology and variable genus. 66 20 27 40 43 14 54 53 Recently, rapid progress in neural volume rendering [45] and 2D Generative Adversarial Networks (GANs) [34, 35, 33, 29, 52] has led to the rise of 3D-aware image synthesis [7, 57, 8, 49, 51, 25]. However, this line of work aims to synthesize multi-view consistent images using neural rendering in the synthesis process and does not guarantee that meaningful 3D shapes can be generated. While a mesh can potentially be obtained from the underlying neural field representation using the marching cube algorithm [39], extracting the corresponding texture is non-trivial. 45 34 35 33 29 52 7 57 8 49 51 25 39 In this work, we introduce a novel approach that aims to tackle all the requirements of a practically useful 3D generative model. Specifically, we propose GET3D, a Generative model for 3D shapes that directly outputs Explicit Textured 3D meshes with high geometric and texture detail and arbitrary mesh topology. In the heart of our approach is a generative process that utilizes a differentiable explicit surface extraction method [60] and a differentiable rendering technique [47, 37]. The former enables us to directly optimize and output textured 3D meshes with arbitrary topology, while the latter allows us to train our model with 2D images, thus leveraging powerful and mature discriminators developed for 2D image synthesis. Since our model directly generates meshes and uses a highly efficient (differentiable) graphics renderer, we can easily scale up our model to train with image G E T 3D explicit 60 47 37 resolution as high as 1024 × 1024, allowing us to learn high-quality geometric and texture details. We demonstrate state-of-the-art performance for unconditional 3D shape generation on multiple categories with complex geometry from ShapeNet [9], Turbosquid [4] and Renderpeople [2], such as chairs, motorbikes, cars, human characters, and buildings. With explicit mesh as output representation, GET3D is also very flexible and can easily be adapted to other tasks, including: (a) learning to generate decomposed material and view-dependent lighting effects using advanced differentiable rendering [12], without supervision, (b) text-guided 3D shape generation using CLIP [56] embedding. 9 4 2 (a) 12 (b) 56 2 Related Work We review recent advances in 3D generative models for geometry and appearance, as well as 3D-aware generative image synthesis. 3D Generative Models In recent years, 2D generative models have achieved photorealistic quality in high-resolution image synthesis [34, 35, 33, 52, 29, 19, 16]. This progress has also inspired research in 3D content generation. Early approaches aimed to directly extend the 2D CNN generators to 3D voxel grids [66, 20, 27, 40, 62], but the high memory footprint and computational complexity of 3D convolutions hinder the generation process at high resolution. As an alternative, other works have explored point cloud [5, 68, 75, 46], implicit [43, 14], or octree [30] representations. However, these works focus mainly on generating geometry and disregard appearance. Their output representations also need to be post-processed to make them compatible with standard graphics engines. 3D Generative Models 34 35 33 52 29 19 16 66 20 27 40 62 5 68 75 46 43 14 30 More similar to our work, Textured3DGAN [54, 53] and DIBR [11] generate textured 3D meshes, but they formulate the generation as a deformation of a template mesh, which prevents them from generating complex topology or shapes with varying genus, which our method can do. PolyGen [48] and SurfGen [41] can produce meshes with arbitrary topology, but do not synthesize textures. 54 53 11 48 41 3D-Aware Generative Image Synthesis Inspired by the success of neural volume rendering [45] and implicit representations [43, 14], recent work started tackling the problem of 3D-aware image synthesis [7, 57, 49, 26, 25, 76, 8, 51, 58, 67]. However, neural volume rendering networks are typically slow to query, leading to long training times [7, 57], and generate images of limited resolution. GIRAFFE [49] and StyleNerf [25] improve the training and rendering efficiency by performing neural rendering at a lower resolution and then upsampling the results with a 2D CNN. However, the performance gain comes at the cost of a reduced multi-view consistency. By utilizing a dual discriminator, EG3D [8] can partially mitigate this problem. Nevertheless, extracting a textured surface from methods that are based on neural rendering is a non-trivial endeavor. In contrast, GET3D directly outputs textured 3D meshes that can be readily used in standard graphics engines. 3D-Aware Generative Image Synthesis 45 43 14 7 57 49 26 25 76 8 51 58 67 7 57 49 25 8 3 Method We now present our GET3D framework for synthesizing textured 3D shapes. Our generation process is split into two parts: a geometry branch, which differentiably outputs a surface mesh of arbitrary topology, and a texture branch that produces a texture field that can be queried at the surface points to produce colors. The latter can be extended to other surface properties such as for example materials (Sec. 4.3.1). During training, an efficient differentiable rasterizer is utilized to render the resulting textured mesh into 2D high-resolution images. The entire process is differentiable, allowing for adversarial training from images (with masks indicating an object of interest) by propagating the gradients from the 2D discriminator to both generator branches. Our model is illustrated in Fig. 2. In the following, we first introduce our 3D generator in Sec 3.1, before proceeding to the differentiable rendering and loss functions in Sec 3.2. 4.3.1 2 3.1 3.2 3.1 Generative Model of 3D Textured Meshes We aim to learn a 3D generator M, E = G(z) to map a sample from a Gaussian distribution M, E G z z ∈ N (0*,* I) to a mesh M with texture E. z I M E Since the same geometry can have different textures, and the same texture can be applied to different geometries, we sample two random input vectors z1 ∈ R512 and z2 ∈ R512. Following StyleGAN [34, 35, 33], we then use non-linear mapping networks fgeo and ftex to map z1 and z2 to intermediate latent vectors w1 = fgeo(z1) and w2 = ftex(z2) which are further used to produce styles that control the generation of 3D shapes and texture, respectively. We formally introduce the generator for geometry in Sec. 3.1.1 and the texture generator in Sec. 3.1.2. z z 34 35 33 f f z z w f z w f z styles 3.1.1 3.1.2 3.1.1 Geometry Generator We design our geometry generator to incorporate DMTet [60], a recently proposed differentiable surface representation. DMTet represents geometry as a signed distance field (SDF) defined on a deformable tetrahedral grid [22, 24], from which the surface can be differentiably recovered through marching tetrahedra [17]. Deforming the grid by moving its vertices results in a better utilization of its resolution. By adopting DMTet for surface extraction, we can produce explicit meshes with arbitrary topology and genus. We next provide a brief summary of DMTet and refer the reader to the original paper for further details. 60 22 24 17 Let (VT , T ) denote the full 3D space that the object lies in, where VT are the vertices in the tetrahedral grid T . Each tetrahedron Tk ∈ T is defined using four vertices {vak , vbk , vck , vdk }, with k ∈ {1*, . . . , K*}, where K is the total number of tetrahedra, and vik ∈ VT , vik ∈ R3. In addition to its i 3D coordinates, each vertex vi contains the SDF value si ∈ R and the deformation ∆vi ∈ R3 of the vertex from its initial canonical coordinate. This representation allows recovering the explicit mesh through differentiable marching tetrahedra [60], where SDF values in continuous space are computed by a barycentric interpolation of their value si on the deformed vertices v′ = vi + ∆vi. VT , T VT T Tk T v ak , v bk , v ck , v dk k K v ik VT , v ik i v i si v i 60 si v v i v i Network Architecture We map w1 ∈ R512 to SDF values and deformations at each vertex vi through a series of conditional 3D convolutional and fully connected layers. Specifically, we first use 3D convolutional layers to generate a feature volume conditioned on w1. We then query the feature at each vertex vi ∈ VT using trilinear interpolation and feed it into MLPs that outputs the SDF value si and the deformation ∆vi. In cases where modeling at a high-resolution is required (e.g. motorbike with thin structures in the wheels), we further use volume subdivision following [60]. Network Architecture w v i w v i VT si v i 60 Differentiable Mesh Extraction After obtaining si and ∆vi for all the vertices, we use the differentiable marching tetrahedra algorithm to extract the explicit mesh. Marching tetrahedra determines the surface topology within each tetrahedron based on the signs of si. In particular, a mesh face is extracted when sign(si) /= sign(sj), where i, j denotes the indices of vertices in the edge of tetrahedron, and the vertices mi,j of that face are determined by a linear interpolation as mi,j = v 0 i sj−v 0 j si sj−si . Note that the above equation is only evaluated when si 6= sj , thus it is differentiable, and the gradient from mi,j can be back-propagated into the SDF values si and deformations ∆vi . With this representation, the shapes with arbitrary topology can easily be generated by predicting different signs of si . Differentiable Mesh Extraction si v i si si sj i, j m i,j 3.1.2 Texture Generator Directly generating a texture map consistent with the output mesh is not trivial, as the generated shape can have an arbitrary genus and topology. We thus parameterize the texture as a texture field [50]. 50 Specifically, we model the texture field with a function ft that maps the 3D location of a surface point p ∈ R3, conditioned on the w2, to the RGB color c ∈ R3 at that location. Since the texture field depends on geometry, we additionally condition this mapping on the geometry latent code w1, such that c = ft(p*,* w1 ⊕ w2), where ⊕ denotes concatenation. ft p w c w c ft p w w Network Architecture We represent our texture field using a tri-plane representation, which is efficient and expressive in reconstructing 3D objects [55] and generating 3D-aware images [8] . Specifically, we follow [8, 35] and use a conditional 2D convolutional neural network to map the latent code w1 ⊕ w2 to three axis-aligned orthogonal feature planes of size N × N × (C × 3), where N = 256 denotes the spatial resolution and C = 32 the number of channels. Network Architecture 55 8 8 35 w w N N C N C Given the feature planes, the feature vector f t ∈ R 32 of a surface point p can be recovered as f t = P e ρ(πe(p)), where πe(p) is the projection of the point p to the feature plane e and ρ(·) denotes bilinear interpolation of the features. An additional fully connected layer is then used to map the aggregated feature vector f t to the RGB color c. Note that, different from other works on 3D-aware image synthesis [8, 25, 7, 57] that also use a neural field representation, we only need to sample the texture field at the locations of the surface points (as opposed to dense samples along a ray). This greatly reduces the computational complexity for rendering high-resolution images and guarantees to generate multi-view consistent images by construction. 3.2 Differentiable Rendering and Training In order to supervise our model during training, we draw inspiration from Nvdiffrec [47] that performs multi-view 3D object reconstruction by utilizing a differentiable renderer. Specifically, we render the extracted 3D mesh and the texture field into 2D images using a differentiable renderer [37], and supervise our network with a 2D discriminator, which tries to distinguish the image from a real object or rendered from the generated object. 47 37 Differentiable Rendering We assume that the camera distribution C that was used to acquire the images in the dataset is known. To render the generated shapes, we randomly sample a camera c from C, and utilize a highly-optimized differentiable rasterizer Nvdiffrast [37] to render the 3D mesh into a 2D silhouette as well as an image where each pixel contains the coordinates of the corresponding 3D Differentiable Rendering c 37 point on the mesh surface. These coordinates are further used to query the texture field to obtain the RGB values. Since we operate directly on the extracted mesh, we can render high-resolution images with high efficiency, allowing our model to be trained with image resolution as high as 1024×1024. Discriminator & Objective We train our model using an adversarial objective. We adopt the discriminator architecture from StyleGAN [34], and use the same non-saturating GAN objective with R1 regularization [42]. We empirically find that using two separate discriminators, one for RGB images and another one for silhouettes, yields better results than a single discriminator operating on both. Let Dx denote the discriminator, where x can either be an RGB image or a silhouette. The adversarial objective is then be defined as follows: Discriminator & Objective 34 42 Dx x where g(u) is defined as g(u) = − log(1 +exp(−u)), px is the distribution of real images, R denotes rendering, and λ is a hyperparameter. Since R is differentiable, the gradients can be backpropagated from 2D images to our 3D generators. g u g u u px R λ R Regularization To remove internal floating faces that are not visible in any of the views, we further regularize the geometry generator with a cross-entropy loss defined between the SDF values of the neighboring vertices [47]: Regularization 47 where H denotes binary cross-entropy loss and σ denotes the sigmoid function. The sum in Eq. 2 is defined over the set of unique edges Se in the tetrahedral grid, for which sign(si) /= sign(sj). H σ 2 e si sj The overall loss function is then defined as: where µ is a hyperparameter that controls the level of regularization. µ 4 Experiments We conduct extensive experiments to evaluate our model. We first compare the quality of the 3D textured meshes generated by GET3D to the existing methods using the ShapeNet [9] and Turbosquid [4] datasets. Next, we ablate our design choices in Sec. 4.2. Finally, we demonstrate the flexibility of GET3D by adapting it to downstream applications in Sec. 4.3. Additional experimental results and implementation details are provided in Appendix. 9 4 4.2 4.3 4.1 Experiments on Synthetic Datasets Datasets For evaluation on ShapeNet [9], we use three categories with complex geometry – Car, Chair, and Motorbike, which contain 7497, 6778, and 337 shapes, respectively. We randomly split each category into training (70%), validation (10 %), and test (20 %), and further remove from the test set shapes that have duplicates in the training set. To render the training data, we randomly sample camera poses from the upper hemisphere of each shape. For the Car and Chair categories, we use 24 random views, while for Motorbike we use 100 views due to less number of shapes. As models in ShapeNet only have simple textures, we also evaluate GET3D on an Animal dataset (442 shapes) collected from TurboSquid [4], where textures are more detailed and we split it into training, validation and test as defined above. Finally, to demonstrate the versatility of GET3D, we also provide qualitative results on the House dataset collected from Turbosquid (563 shapes), and Human Body dataset from Renderpeople [2] (500 shapes). We train a separate model on each category. Datasets 9 Car Chair Motorbike Car Chair Motorbike Animal 4 House Human Body 2 Baselines We compare GET3D to two groups of works: 1) 3D generative models that rely on 3D supervision: PointFlow [68] and OccNet [43]. Note that these methods only generate geometry without texture. 2) 3D-aware image generation methods: GRAF [57], PiGAN [7], and EG3D [8]. Baselines 1) 68 43 2) 57 7 8 Metrics To evaluate the quality of our synthesis, we consider both the geometry and texture of the generated shapes. For geometry, we adopt metrics from [5] and use both Chamfer Distance (CD) and Light Field Distance [10] (LFD) to compute the Coverage score and Minimum Matching Distance. For OccNet [43], GRAF [57], PiGAN [7] and EG3D [8], we use marching cubes to extract the underlying geometry. For PointFlow [68], we use Poisson surface reconstruction to convert a point cloud into a mesh when evaluating LFD. To evaluate texture quality, we adopt the FID [28] metric commonly used to evaluate image synthesis. In particular, for each category, we render the test shapes into 2D images, and also render the generated 3D shapes from each model into 50k images using the same camera distribution. We then compute FID on the two image sets. As the baselines from 3D-aware image synthesis [57, 7, 8] do not directly output textured meshes, we compute FID score in two ways: (i) we use their neural volume rendering to obtain 2D images, which we refer to as FID-Ori, and (ii) we extract the mesh from their neural field representation using marching cubes, render it, and then use the 3D location of each pixel to query the network to obtain the RGB values. We refer to this score, that is more aware of the actual 3D shape, as FID-3D. Further details on the evaluation metrics are available in the Appendix B.3. Metrics 5 10 43 57 7 8 68 28 57 7 8 i ii B.3 Experimental Results We provide quantitative results in Table. 2 and qualitative examples in Fig. 3 and Fig. 4. Additional results are available in the supplementary video. Compared to OccNet [43] that uses 3D supervision during training, GET3D achieves better performance in terms of both diversity (COV) and quality (MMD), and our generated shapes have more geometric details. Experimental Results 2 3 4 43 PointFlow [68] outperforms GET3D in terms of MMD on CD, while GET3D is better in MMD on LFG. We hypothesize that this is because PointFlow directly optimizes on point locations, which favours CD. GET3D also performs favourably when compared to 3D-aware image synthesis methods, we achieve significant improvements over PiGAN [7] and GRAF [57] in terms of all metrics on all datasets. Our generated shapes also contain more detailed geometry and texture. Compared with recent work EG3D [8]. We achieve comparable performance on generating 2D images (FID-ori), while we significantly improve on 3D shape synthesis in terms of FID-3D, which demonstrates the effectiveness of our model on learning actual 3D geometry and texture. 68 7 57 8 Since we synthesize textured meshes, we can export our shapes into Blender1. We show rendering results in Fig. 1 and 5. GET3D is able to generate shapes with diverse and high quality geometry and topology, very thin structures (motorbikes), as well as complex textures on cars, animals, and houses. 1 1 5 Shape Interpolation GET3D also enables shape interpolation, which can be useful for editing purposes. We explore the latent space of GET3D in Fig. 6, where we interpolate the latent codes to generate each shape from left to right. GET3D is able to faithfully generate a smooth and meaningful transition from one shape to another. We further explore the local latent space by slightly perturbing the latent codes to a random direction. GET3D produces novel and diverse shapes when applying local editing in the latent space (Fig. 7). Shape Interpolation 6 7 4.2 Ablations We ablate our model in two ways: 1) w/ and w/o volume subdivision, 2) training using different image resolutions. Further ablations are provided in the Appendix C.3. 1) 2) C.3 Ablation of Volume Subdivision As shown in Tbl. 2, volume subdivision significantly improves the performance on classes with thin structures (e.g., motorbikes), while not getting gains on other classes. We hypothesize that the initial tetrahedral resolution is already sufficient to capture the de-tailed geometry on Chairs and Cars, and hence the subdivision cannot provide further improvements. Ablation of Volume Subdivision 2 Ablating Different Image Resolutions We ablate the effect of the training image resolution in Tbl. 3. As expected, increased image resolution improves the performance in terms of FID and shape quality, as the network can see more details, which are often not available in the low-resolution images. This corroborates the importance of training with higher image resolution, which are often hard to make use of for implicit-based methods. 3 4.3 Applications 4.3.1 Material Generation for View-dependent Lighting Effects 4.3.1 Material Generation for View-dependent Lighting Effects GET3D can easily be extended to also generate surface materials that are directly usable in modern graphics engines. In particular, we follow the widely used Disney BRDF [6, 32] and describe the materials in terms of the base color (R3), metallic (R), and roughness (R) properties. As a result, we repurpouse our texture generator to now output a 5-channel reflectance field (instead of only RGB). To accommodate differentiable rendering of materials, we adopt an efficient spherical Gaussian (SG) based deferred rendering pipeline [12]. Specifically, we rasterize the reflectance field into a G-buffer, and randomly sample an HDR image from a set of real-world outdoor HDR panoramas Slight = {LSG}K, where LSG ∈ R32×7 is obtained by fitting 32 SG lobes to each panorama. The SG renderer [12] then uses the camera c to render an RGB image with view-dependent lighting effects, which we feed into the discriminator during training. Note that GET3D does not require material supervision during training and learns to generate decomposed materials in an unsupervised manner. 6 32 12 LSG K LSG 12 c We provide qualitative results of generated surface materials in Fig. 8. Despite unsupervised, GET3D discovers interesting material decomposition, e.g., the windows are correctly predicted with a smaller roughness value to be more glossy than the car’s body, and the car’s body is discovered as more dielectric while the window is more metallic. Generated materials enable us to produce realistic relighting results, which can account for complex specular effects under different lighting conditions. 8 4.3.2 Text-Guided 3D Synthesis Similar to image GANs, GET3D also supports text-guided 3D content synthesis by fine-tuning a pre-trained model under the guidance of CLIP [56]. Note that our final synthesis result is a textured 3D mesh. To this end, we follow the dual-generator design from styleGAN-NADA [21], where a trainable copy Gt and a frozen copy Gf of the pre-trained generator are adopted. During optimization Gt and Gf both render images from 16 random camera views. Given a text query, we sample 500 pairs of noise vectors z1 and z2. For each sample, we optimize the parameters of Gt to minimize the directional CLIP loss [21] (the source text labels are “car”, “animal” and “house” for the corresponding categories), and select the samples with minimal loss. To accelerate this process, we first run a small number of optimization steps for the 500 samples, then choose the top 50 samples with the lowest losses, and run the optimization for 300 steps. The results and comparison against a SOTA text-driven mesh stylization method, Text2Mesh [44], are provided in Fig. 9. Note that, [44] requires a mesh of the shape as an input to the method. We provide our generated meshes from the frozen generator as input meshes to it. Since it needs mesh vertices to be dense to synthesize surface details with vertex displacements, we further subdivide the input meshes with mid-point subdivision to make sure each mesh has 50k-150k vertices on average. 56 21 Gt Gf Gt Gf z z Gt 21 44 9 44 5 Conclusion We introduced GET3D, a novel 3D generative model that is able to synthesize high-quality 3D textured meshes with arbitrary topology. GET3D is trained using only 2D images as supervision. We experimentally demonstrated significant improvements on generating 3D shapes over previous state-of-the-art methods on multiple categories. We hope that this work brings us one step closer to democratizing 3D content creation using A.I.. Limitations While GET3D makes a significant step towards a practically useful 3D generative model of 3D textured shapes, it still has some limitations. In particular, we still rely on 2D silhouettes as well as the knowledge of camera distribution during training. As a consequence, GET3D was currently only evaluated on synthetic data. A promising extension could use the advances in instance segmentation and camera pose estimation to mitigate this issue and extend GET3D to real-world data. GET3D is also trained per-category; extending it to multiple categories in the future, could help us better represent the inter-category diversity. Limitations Broader Impact We proposed a novel 3D generative model that generates 3D textured meshes, which can be readily imported into current graphics engines. Our model is able to generate shapes with arbitrary topology, high quality textures and rich geometric details, paving the path for democratizing Broader Impact A.I. tool for 3D content creation. As all machine learning models, GET3D is also prone to biases introduced in the training data. Therefore, an abundance of caution should be applied when dealing with sensitive applications, such as generating 3D human bodies, as GET3D is not tailored for these applications. We do not recommend using GET3D if privacy or erroneous recognition could lead to potential misuse or any other harmful applications. Instead, we do encourage practitioners to carefully inspect and de-bias the datasets before training our model to depict a fair and wide distribution of possible skin tones, races or gender identities. 6 Disclosure of Funding This work was funded by NVIDIA. Jun Gao, Tianchang Shen, Zian Wang and Wenzheng Chen acknowledge additional revenue in the form of student scholarships from University of Toronto and the Vector Institute, which are not in direct support of this work. References [1] Autodesk Maya, https://www.autodesk.com/products/maya/overview. Accessed: 2022-05-19. https://www.autodesk.com/products/maya/overview [2] Renderpeople, http://https://renderpeople.com/. Accessed: 2022-05-19. http://https://renderpeople.com/ [3] Sketchfab, https://sketchfab.com/. Accessed: 2022-05-19. https://sketchfab.com/ [4] Turbosquid by Shutterstock, https://www.turbosquid.com/. Accessed: 2022-05-19. https://www.turbosquid.com/ [5] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pages 40–49. PMLR, 2018. International conference on machine learning [6] Brent Burley and Walt Disney Animation Studios. Physically-based shading at disney. In ACM SIGGRAPH, volume 2012, pages 1–7. vol. 2012, 2012. ACM SIGGRAPH [7] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proc. CVPR, 2021. Proc. CVPR [8] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123–16133, 2022. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition [9] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015. arXiv preprint arXiv:1512.03012 [10] Ding-Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung. On visual similarity based 3d model retrieval. In Computer graphics forum, volume 22, pages 223–232. Wiley Online Library, 2003. Computer graphics forum [11] Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation-based differentiable renderer. In Advances In Neural Information Processing Systems, 2019. Advances In Neural Information Processing Systems [12] Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khalis, Or Litany, and Sanja Fidler. DIB-R++: Learning to predict lighting and material with a hybrid differentiable renderer. In Advances in Neural Information Processing Systems (NeurIPS), 2021. Advances in Neural Information Processing Systems (NeurIPS) [13] Yanqin Chen, Xin Jin, and Qionghai Dai. Distance measurement based on light field geometry and ray tracing. Optics Express, 25(1):59–76, 2017. Optics Express [14] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) [15] Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. Blender - a 3D modelling and rendering package [16] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021. Advances in Neural Information Processing Systems [17] Akio Doi and Akio Koide. An efficient method of triangulating equi-valued surfaces by using tetrahedral cells. IEICE TRANSACTIONS on Information and Systems, 74(1):214–224, 1991. IEICE TRANSACTIONS on Information and Systems [18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. arXiv preprint arXiv:2010.11929 [19] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873–12883, 2021. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition [20] Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shape induction from 2d views of multiple objects. In 2017 International Conference on 3D Vision (3DV), pages 402–411. IEEE, 2017. 2017 International Conference on 3D Vision (3DV) [21] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 41(4):1–13, 2022. ACM Transactions on Graphics (TOG) [22] Jun Gao, Wenzheng Chen, Tommy Xiang, Clement Fuji Tsang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Learning deformable tetrahedral meshes for 3d reconstruction. In Advances In Neural Information Processing Systems, 2020. Advances In Neural Information Processing Systems [23] Jun Gao, Chengcheng Tang, Vignesh Ganapathi-Subramanian, Jiahui Huang, Hao Su, and Leonidas J Guibas. Deepspline: Data-driven reconstruction of parametric curves and surfaces. arXiv preprint arXiv:1901.03781, 2019. arXiv preprint arXiv:1901.03781 [24] Jun Gao, Zian Wang, Jinchen Xuan, and Sanja Fidler. Beyond fixed grid: Learning geometric image representation with a deformable grid. In European Conference on Computer Vision, pages 108–125. Springer, 2020. European Conference on Computer Vision [25] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: A style-based 3d aware generator for high-resolution image synthesis. In International Conference on Learning Representations, 2022. International Conference on Learning Representations [26] Zekun Hao, Arun Mallya, Serge Belongie, and Ming-Yu Liu. GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds. In ICCV, 2021. ICCV [27] Philipp Henzler, Niloy J. Mitra, and Tobias Ritschel. Escaping plato’s cave: 3d shape from adversarial rendering. In The IEEE International Conference on Computer Vision (ICCV), October 2019. The IEEE International Conference on Computer Vision (ICCV) [28] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. Advances in neural information processing systems [29] Xun Huang, Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu. Multimodal conditional image synthesis with product-of-experts GANs. In ECCV, 2022. ECCV [30] Moritz Ibing, Gregor Kobsik, and Leif Kobbelt. Octree transformer: Autoregressive 3d shape generation on hierarchically structured sequences. arXiv preprint arXiv:2111.12480, 2021. arXiv preprint arXiv:2111.12480 [31] James T. Kajiya. The rendering equation. SIGGRAPH ’86, page 143–150, 1986. [32] Brian Karis and Epic Games. Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice, 4(3), 2013. Proc. Physically Based Shading Theory Practice [33] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Proc. NeurIPS, 2021. Proc. NeurIPS [34] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition [35] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In Proc. CVPR, 2020. Proc. CVPR [36] Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposium on Geometry processing, volume 7, 2006. Proceedings of the fourth Eurographics symposium on Geometry processing [37] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics, 39(6), 2020. ACM Transactions on Graphics [38] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Conference on Computer Vision and Pattern Recognition (CVPR) [39] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163–169, 1987. ACM siggraph computer graphics [40] Sebastian Lunz, Yingzhen Li, Andrew Fitzgibbon, and Nate Kushman. Inverse graphics gan: Learning to generate 3d shapes from unstructured 2d data. arXiv preprint arXiv:2002.12674, 2020. arXiv preprint arXiv:2002.12674 [41] Andrew Luo, Tianqin Li, Wen-Hao Zhang, and Tai Sing Lee. Surfgen: Adversarial 3d shape synthesis with explicit surface discriminators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16238–16248, 2021. Proceedings of the IEEE/CVF International Conference on Computer Vision [42] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Which training methods for gans do actually converge? In International Conference on Machine Learning (ICML), 2018. International Conference on Machine Learning (ICML) [43] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition [44] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13492–13502, 2022. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition [45] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. ECCV [46] Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. Structurenet: Hierarchical graph networks for 3d shape generation. ACM Transactions on Graphics (TOG), Siggraph Asia 2019, 38(6):Article 242, 2019. ACM Transactions on Graphics (TOG), Siggraph Asia 2019 [47] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8280–8290, 2022. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition [48] Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter W. Battaglia. Polygen: An autoregressive generative model of 3d meshes. ICML, 2020. ICML [49] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) [50] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4531–4540, 2019. Proceedings of the IEEE/CVF International Conference on Computer Vision [51] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher-Shlizerman. Stylesdf: High-resolution 3d-consistent image and geometry generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13503–13513, 2022. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition [52] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni-tion, 2019. Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni-tion [53] Dario Pavllo, Jonas Kohler, Thomas Hofmann, and Aurelien Lucchi. Learning generative models of textured 3d meshes from real-world images. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021. IEEE/CVF International Conference on Computer Vision (ICCV) [54] Dario Pavllo, Graham Spinks, Thomas Hofmann, Marie-Francine Moens, and Aurelien Lucchi. Convolu-tional generation of textured 3d meshes. In Advances in Neural Information Processing Systems (NeurIPS), 2020. Advances in Neural Information Processing Systems (NeurIPS) [55] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision (ECCV), 2020. European Conference on Computer Vision (ECCV) [56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021. International Conference on Machine Learning [57] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2020. Advances in Neural Information Processing Systems (NeurIPS) [58] Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger. Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids. ARXIV, 2022. ARXIV [59] Tianchang Shen, Jun Gao, Amlan Kar, and Sanja Fidler. Interactive annotation of 3d object geometry using 2d scribbles. In European Conference on Computer Vision, pages 751–767. Springer, 2020. European Conference on Computer Vision [60] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021. Advances in Neural Information Processing Systems (NeurIPS) [61] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020. Proc. NeurIPS [62] Edward J Smith and David Meger. Improved adversarial systems for 3d object generation and reconstruction. In Conference on Robot Learning, pages 87–96. PMLR, 2017. Conference on Robot Learning [63] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016. Proceedings of the IEEE conference on computer vision and pattern recognition [64] Jiaping Wang, Peiran Ren, Minmin Gong, John Snyder, and Baining Guo. All-frequency rendering of dynamic, spatially-varying reflectance. In ACM SIGGRAPH Asia 2009 papers, pages 1–10. 2009. ACM SIGGRAPH Asia 2009 papers [65] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010. [66] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. Advances in neural information processing systems, 29, 2016. Advances in neural information processing systems [67] Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, and Bolei Zhou. 3d-aware image synthesis via learning structural and textural representations. In CVPR, 2022. CVPR [68] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Point-flow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4541–4550, 2019. Proceedings of the IEEE/CVF International Conference on Computer Vision [69] Kangxue Yin, Jun Gao, Maria Shugrina, Sameh Khamis, and Sanja Fidler. 3dstylenet: Creating 3d shapes with geometric and texture style variations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12456–12465, 2021. Proceedings of the IEEE/CVF International Conference on Computer Vision [70] Kangxue Yin, Jun Gao, Maria Shugrina, Sameh Khamis, and Sanja Fidler. 3dstylenet: Creating 3d shapes with geometric and texture style variations. In Proceedings of International Conference on Computer Vision (ICCV), 2021. Proceedings of International Conference on Computer Vision (ICCV) [71] Jonathan Young. xatlas, 2021. https://github.com/jpcy/xatlas. https://github.com/jpcy/xatlas [72] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) [73] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. In International Conference on Learning Representations, 2021. International Conference on Learning Representations [74] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021. CVPR [75] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826–5835, 2021. Proceedings of the IEEE/CVF International Conference on Computer Vision [76] Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian. Cips-3d: A 3d-aware generator of gans based on conditionally-independent pixel synthesis. arXiv preprint arXiv:2110.09788, 2021. arXiv preprint arXiv:2110.09788 [77] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018. arXiv:1801.09847 This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. This paper is available on arxiv under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv
