.agents/skills/materia-nuke-node/SKILL.md

name

description

materia-nuke-node

Build Nuke VFX plugin nodes for Materia (Biohazard VFX). Use when creating ML inference nodes, gizmos, or Cattery plugins for Foundry Nuke. Covers TorchScript model wrapping, .cat file compilation, gizmo design, knob definitions, and plugin distribution. Triggers: "nuke node", "nuke plugin", "nuke gizmo", "materia node", "cattery plugin", "inference node", "torchscript for nuke", "depth estimation node", "ML node for nuke", "wrap model for nuke", "VFX plugin", "compositing node".

Materia Nuke Node Builder

Build ML inference nodes for Foundry Nuke as part of the Materia VFX tools suite by Biohazard VFX.

Architecture Decision

Always use Cattery + TorchScript + Inference node. NOT nuke-ML-server, NOT pure Python pixel processing, NOT custom NDK from scratch.

• Zero deps for end users (drop .gizmo + .cat into Cattery/)
• Nuke's Inference node handles GPU/CPU, precision, batch automatically
• TorchScript models are portable across platforms
• Official Foundry-supported pattern

Workflow

1. Read the model's source code thoroughly BEFORE writing wrapper code
2. List state_dict keys from a pretrained checkpoint to understand module hierarchy
3. Design TorchScript wrapper matching exact module names for weight loading
4. Write compilation script (download weights, wrap, trace, save .pt)
5. Build gizmo (node UI + internal processing pipeline)
6. Convert .pt to .cat via Nuke's CatFileCreator
7. Test in Nuke

TorchScript Constraints (critical)

• ONLY: torch ops, python builtins, math module
• NO: numpy, opencv, einops, xformers, addict, PIL
• All class attributes initialized in init
• No dynamic typing (variables can't change type)
• Use int not bool
• F.interpolate: size=(h, w) not size=shape[2:]
• Replace xformers -> torch.nn.functional.scaled_dot_product_attention
• Replace einops rearrange -> native torch reshape/permute/contiguous
• Replace addict.Dict -> plain dicts or baked constants

Common Failure Modes

• Module hierarchy mismatch: wrapper names don't match checkpoint keys, strict=False silently loads nothing
• Missing LayerScale: modern ViTs use learnable gamma*x after attn/mlp -- skip it and weights won't load
• Wrong feature extraction layers: each model variant has different intermediate layer indices, read the yaml config
• Wrong activation: exp vs sigmoid vs relu -- read the model config, don't guess
• Gizmo enum expressions: use numeric index (0, 1, 2) not string comparison
• sRGB conversion: use Nuke Colorspace node, not ColorMatrix (sRGB EOTF is nonlinear)

Reference Implementation Locations

• DA2 plugin (working): /mnt/work/dev/materia/Depth-Anything-for-Nuke/
• DA3 plugin (in progress): /mnt/work/dev/materia/Materia-DepthAnythingV3/
• Foundry research report: /mnt/work/dev/materia/foundry-nuke-research-report.md

Detailed References

• Cattery pattern and gizmo design
• TorchScript wrapper patterns

Repo Template

MateriaNodeName/
├── README.md
├── SPEC.md
├── LICENSE (Apache 2.0)
├── .gitignore
├── nuke/Cattery/NodeName/
│   ├── NodeName.gizmo
│   ├── NodeName.cat
│   ├── cat.json
│   └── NodeName.png
├── compile/
│   ├── wrapper.py
│   ├── nuke_compile.py
│   ├── requirements.txt
│   └── docker/Dockerfile
├── models/.gitkeep
├── tests/
└── docs/