JIGPX
JIGPX

ComfyUI — Guidance

ControlNet, IP-Adapter, and adapters to steer generations

Overview

  • Guidance = extra conditioning (images/structures) to control pose, edges, style, identity.
  • Works with SD 1.5 and SDXL (use model-specific weights).
  • Common tools: ControlNet, IP-Adapter, T2I-Adapter.

ControlNet (structure guidance)

  • Use a preprocessor to extract signals from an image (canny edges, depth, openpose, lineart, tile, etc.).
  • Load a matching ControlNet model and wire it into your sampler alongside your prompt conditioning.
  • Key knobs: strength, start/end (when it influences steps), low/hi-res resize.
Basic wiring:
[Load Image] -> [Preprocessor (e.g., Canny)] -> [ControlNet Loader (canny model)]
[Checkpoint Loader] -> model/clip
[CLIPTextEncode (pos/neg)] -> 
[KSampler with ControlNet conditioning] -> result

IP-Adapter (style/identity guidance)

  • Condition on a reference image via CLIP Vision to transfer style/identity.
  • Variants: IP-Adapter, Plus, FaceID; use SD 1.5 vs SDXL weights accordingly.
  • Knobs: weight(s) for UNet/CLIP; multiple refs can be blended.
Basic wiring:
[Load Image] -> [IP-Adapter Loader] + [Load CLIP Vision (matching SD version)]
[Checkpoint Loader] -> model/clip
[Apply IP-Adapter] -> into KSampler along with prompt conditioning

T2I-Adapter (lightweight control)

  • Similar idea to ControlNet but lighter/faster; supports edge/pose/depth variants.
  • Good when you need guidance but want lower VRAM hit.

File locations

ComfyUI/
  models/
    checkpoints/     # Base/refiner checkpoints
    controlnet/      # ControlNet .safetensors (canny, depth, pose, tile, etc.)
    ipadapter/       # IP-Adapter weights (.bin/.pth) and configs
    clip_vision/     # CLIP-ViT models for IP-Adapter
    t2i-adapter/     # T2I-Adapter weights (if used)

Quick recipes

  • Pose copy: OpenPose preprocessor + ControlNet OpenPose model; strength ~0.7; start 0.0 end 0.8.
  • Edge-to-art: Canny + ControlNet Canny; prompt for style (e.g., watercolor, anime).
  • Style transfer: IP-Adapter with a reference style image; weight ~0.6–0.9; combine with your prompt.
  • Upscale cleanup: Tile ControlNet during high-res fix to keep structure while adding detail.

Tips

  • Use SDXL-specific weights with SDXL; SD 1.5 weights with SD 1.5.
  • If results are “locked,” reduce strength or end earlier; if drifting, increase strength or start earlier.
  • Match preprocessor settings (e.g., canny thresholds) to desired fidelity vs freedom.
  • Combine with LoRAs sparingly; balance weights to avoid fights.