Milestone Report: ML-Guided MIS & Adaptive Sampling

CS184/284A – Mid-Project Progress

AI Acknowledgement

⚠️ This milestone report was created with assistance from ChatGPT by OpenAI for organization, editing, and styling.

1. Summary of Accomplishments

  1. Implemented substantial piece of original 2D idea using CLIP’s ViT to blur image regions based on predicted importance; realized this did not involve 3D graphics and pivoted.
  2. Pivoted to improving path tracing sampling with two features: Multiple Importance Sampling (MIS) for direct lighting in non-diffuse materials, and ML-guided adaptive sampling driven by ViT saliency as suggested by mentor C.K. Wolfe.
  3. Set up a GitHub repo and adopted Project 3’s path tracer as our base for implementing MIS and adaptive sampling.
  4. Initial implementation of advanced BSDFs (Mirror, Microfacet, Refraction, Glass). ChatGPT assisted with the implementation and debugging of advanced BSDF functions. After implementing, rendered CBdragon.dae with MirrorBSDF.
  5. Integrated MIS into the direct-light estimator, adjusted reference MIS solution to address PDF mixing across multiple lights and adding a BSDF-sampling branch; debugged with ChatGPT assistance.
  6. Validated MIS by observing proper specular reflection on the rear sphere in CBspheres.dae, confirming improved performance.
  7. Attempted to import custom .dae models (e.g., bunny + Cornell Box, and models downloaded from free sources online); faced manifold issues in Blender/MeshLab and currently render black, indicating further mesh and exporter work is needed.

2. Preliminary Results

ViT blur mask on dog image
ViT blur mask from CLIP attention (Discontinued 2D prototype).
Combined attention map
Composite attention map (Discontinued 2D prototype).
CBdragon.dae with MirrorBSDF
MirrorBSDF rendering of CBdragon.dae.
Adaptive sampling heatmap CBdragon
Adaptive sampling map for CBdragon.dae.
CBspheres.dae before MIS
CBspheres.dae without MIS (missing back-sphere reflection).
CBspheres.dae after MIS
CBspheres.dae with MIS (correct back-sphere reflection).
Custom bunny + Cornell Box rendering
Custom (bunny + CBempty).dae (currently rendering black).

3. Reflection on Progress

Balancing Homework 3, final project pivot, and an exam week has been challenging. We successfully pivoted and built a prototype showing MIS improvements, and have groundwork laid for the rest of the project. Major BSDF and MIS components are implemented (although BSDF is very buggy for everything except MirrorBSDF); remaining work focuses on completing adaptive sampling and asset creation.

4. Updated Work Plan

  1. Week 1: Pivot project and set up repository. (Completed)
  2. Week 2: Implement BSDFs and basic MIS; prepare milestone. (Completed, BSDFs debugging ongoing)
  3. Week 3:
    1. Finalize MIS integration and BSDF fixes
    2. Implement ML-driven adaptive sampling (stretch goal if time permits)
    3. Create or fix complex .dae assets for testing
    4. Benchmark strategies and assemble scene repo to showcase improvements
  4. Week 4: Finalize adaptive sampling (if possible), polish visuals, and prepare presentation.

5. Milestone Video

https://drive.google.com/file/d/1tXc5aLS2BewgGELY8hF3GIsj-xs4jonW/view

6. Presentation Slides

https://docs.google.com/presentation/d/1GJN4pwOsPF3BZFC9wgDqQsBsexMIAlBjLh5UX7oZTZY/edit?usp=sharing