Introduction and Overview.

Traditional 3D modeling pipelines. Computer vision as inverse graphics. Neural 3D representations and neural rendering. Democratization of 3D content creation.with neural 3D synthesis. Generating 3D data for ML training pipelines.

Lecture Slides: Intro

Reading: SyntheticDataforML

Classical 3D Geometry Representations. Volumetric Neural Representations.

Low level: voxel grids, point clouds, triangular and quad meshes. High level: parametric and implicit boundary representations. Review of some classical geometric concepts (normals, curvature). Regular, voxel-based neural representations.

Lecture Slides: ClassicalGeo

Reading: Old survey, normal estimation, VoxNet2, 3D-R2N2, OctGenNet

Neural Architectures for Geometric Data: Point Clouds.

Neural architectures for irregular data. Point cloud methods: PointNet and PointNet++. KPConv and other related methods. Sampling issues. Applications to object detection, classification, and segmentation.

Lecture Slides: PointCloud

Reading: PointNet, PointNet++, DGCNN, VoteNet, FlowNet3D

Generative Models: VAEs and Coordinate-Based Networks.

Introduction to neural generative models. Autoencoders and variational autoencoders. Neural implicits, such as deep signed distance functions.

Lecture Slides: VAEdeepSDF

Reading: VarEncoders1, VarEncoders2, ELBO, DeepMetafunctionals, IMNet, DeepSDF

Student Presentations: PointCloud_S

Homework 1 out.

3D Public Data Sets. Flow and Auto-Regressive Models.

ShapeNet, PartNet, ... Objaverse. PointFlow, Shape Gradient Flow, PolyGen.

Lecture Slides: DataSetsFlowRegress

Reading: ShapeNet, PartNet, Objaverse, PolyGen, PointFlow, SGF

Student Presentations: Sidd, Aniruth

Parametric Models. Generative Adversarial Networks (GANs).

AtlasNet. InfoGAN, 3D-GAN, StyleGAN.

Lecture Slides: ParamGANs

Reading: AtlasNet, GAN1, GAN2, InfoGAN, StyleGAN1

Student Presentations: Rajan, Shunyao

Homework 1 due. Homework 2 out.

Introduction to Denoising Diffusion Models.

Score-based models, Diffusion models, Applications.

Lecture Slides: Diffusion

Reading: DDPM, Gradients, Implicit, TutorialVideo, SongBlogpost

Student Presentations: Tracy, Mahmut

Hierarchical Generation of Structure and Geometry.

Jointly encoding shape part hierarchies and part geometry, including symmetry. Shape abstractions.

Lecture Slides: Hierarchy

Reading: GRASS, StructureNet, DSG_Net, Abstractions

Student Presentations: Jonathan, Jinho

Vector Graphics, Deep Architectures for Meshes.

Vector and mesh geometry generation.

Lecture Slides: VecMesh

Reading: Im2Vec, DMTet, MeshCNN, Sketch2CAD

Student Presentations: Tara. Tianyu

Pose Equivariance and Invariance in 3D Data.

Lecture Slides: InvEquiv

Reading: Vector Neurons, TFNs, EFEM

Student Presentations: Haochen, Samir

Homework 2 due. Homework 3 out.

Conditional Generation: 3D Shape Completion.

Scan completion, part-based generation.

Lecture Slides: CondGenShp

Reading: struct_complete, complement_me, dual_space

Student Presentations: Eric, Judson

Neural Rendering. Neural Radiance Fields (NeRFs) I.

Neural rendering. Neural radiance fields (NeRFs).

Lecture Slides: NeRF

Reading: NeRF

Student Presentations: Brian

Project proposals due.

Conditional Generation: From Image to 3D Shape.

From single image to pintcloud or SDF. GAN and diffusion methods

Lecture Slides: CondGenImg

Reading: deeppointset, DISN, EG3D, 3DiM

Student Presentations: Kevin, Tathagat

Object/Scene Generation and Language I.

Dreamfields and Dreamfusion.

Lecture Slides: Dream

Reading: Dreamfields, Dreamfusion

Student Presentations: Ishan, Po

Homework 3 due.

Object/Scene Generation and Language II.

ShapeGlot, CLIP, DALL-E, StableDiffusion, Imagen

Lecture Slides: Lang

Reading: ShapeGlot, CLIP, DALL-E, StableDiffusion, Imagen

Student Presentations: Haodi

Shape Edits/Deformations.

Continuous and discrete shape editsa/deformations.

Lecture Slides: Deform

Reading: MeshODE, DeformSyncNet, StructEdit, NeuralCages, DeepMetaHandles, DefAwareRetrieval

Student Presentations: Huy, Bernard

Memorial Day (holiday, no classes).

Neural Radiance Fields (NeRFs) II.

Many many NeRF variations, optimizations, and extensions,

Lecture Slides: MoreNeRFs

Reading: iNGP, Mip-NeRF, Nerfies, HyperNerF, VolSDF, PNF, DS-NeRF, NeuralFlowFields

Student Presentations: Tiange, Shuo

Compositional Scene Generation and Object Placement..

Lecture Slides: Scenes

Reading: GRAINS, MetaSim, MetaSim2

Student Presentations: Nikhil

Student Project Presentations.

Slides:

Project due.