92 lines
3.4 KiB
Markdown
92 lines
3.4 KiB
Markdown
---
|
|
name: materia-nuke-node
|
|
description: >
|
|
Build Nuke VFX plugin nodes for Materia (Biohazard VFX). Use when creating
|
|
ML inference nodes, gizmos, or Cattery plugins for Foundry Nuke. Covers
|
|
TorchScript model wrapping, .cat file compilation, gizmo design, knob
|
|
definitions, and plugin distribution. Triggers: "nuke node", "nuke plugin",
|
|
"nuke gizmo", "materia node", "cattery plugin", "inference node",
|
|
"torchscript for nuke", "depth estimation node", "ML node for nuke",
|
|
"wrap model for nuke", "VFX plugin", "compositing node".
|
|
---
|
|
|
|
# Materia Nuke Node Builder
|
|
|
|
Build ML inference nodes for Foundry Nuke as part of the Materia VFX tools
|
|
suite by Biohazard VFX.
|
|
|
|
## Architecture Decision
|
|
|
|
Always use Cattery + TorchScript + Inference node. NOT nuke-ML-server, NOT
|
|
pure Python pixel processing, NOT custom NDK from scratch.
|
|
|
|
- Zero deps for end users (drop .gizmo + .cat into Cattery/)
|
|
- Nuke's Inference node handles GPU/CPU, precision, batch automatically
|
|
- TorchScript models are portable across platforms
|
|
- Official Foundry-supported pattern
|
|
|
|
## Workflow
|
|
|
|
1. Read the model's source code thoroughly BEFORE writing wrapper code
|
|
2. List state_dict keys from a pretrained checkpoint to understand module hierarchy
|
|
3. Design TorchScript wrapper matching exact module names for weight loading
|
|
4. Write compilation script (download weights, wrap, trace, save .pt)
|
|
5. Build gizmo (node UI + internal processing pipeline)
|
|
6. Convert .pt to .cat via Nuke's CatFileCreator
|
|
7. Test in Nuke
|
|
|
|
## TorchScript Constraints (critical)
|
|
|
|
- ONLY: torch ops, python builtins, math module
|
|
- NO: numpy, opencv, einops, xformers, addict, PIL
|
|
- All class attributes initialized in __init__
|
|
- No dynamic typing (variables can't change type)
|
|
- Use int not bool
|
|
- F.interpolate: size=(h, w) not size=shape[2:]
|
|
- Replace xformers -> torch.nn.functional.scaled_dot_product_attention
|
|
- Replace einops rearrange -> native torch reshape/permute/contiguous
|
|
- Replace addict.Dict -> plain dicts or baked constants
|
|
|
|
## Common Failure Modes
|
|
|
|
- Module hierarchy mismatch: wrapper names don't match checkpoint keys, strict=False silently loads nothing
|
|
- Missing LayerScale: modern ViTs use learnable gamma*x after attn/mlp -- skip it and weights won't load
|
|
- Wrong feature extraction layers: each model variant has different intermediate layer indices, read the yaml config
|
|
- Wrong activation: exp vs sigmoid vs relu -- read the model config, don't guess
|
|
- Gizmo enum expressions: use numeric index (0, 1, 2) not string comparison
|
|
- sRGB conversion: use Nuke Colorspace node, not ColorMatrix (sRGB EOTF is nonlinear)
|
|
|
|
## Reference Implementation Locations
|
|
|
|
- DA2 plugin (working): /mnt/work/dev/materia/Depth-Anything-for-Nuke/
|
|
- DA3 plugin (in progress): /mnt/work/dev/materia/Materia-DepthAnythingV3/
|
|
- Foundry research report: /mnt/work/dev/materia/foundry-nuke-research-report.md
|
|
|
|
## Detailed References
|
|
|
|
- [Cattery pattern and gizmo design](references/cattery-pattern.md)
|
|
- [TorchScript wrapper patterns](references/torchscript-patterns.md)
|
|
|
|
## Repo Template
|
|
|
|
```
|
|
MateriaNodeName/
|
|
├── README.md
|
|
├── SPEC.md
|
|
├── LICENSE (Apache 2.0)
|
|
├── .gitignore
|
|
├── nuke/Cattery/NodeName/
|
|
│ ├── NodeName.gizmo
|
|
│ ├── NodeName.cat
|
|
│ ├── cat.json
|
|
│ └── NodeName.png
|
|
├── compile/
|
|
│ ├── wrapper.py
|
|
│ ├── nuke_compile.py
|
|
│ ├── requirements.txt
|
|
│ └── docker/Dockerfile
|
|
├── models/.gitkeep
|
|
├── tests/
|
|
└── docs/
|
|
```
|