OzzyGT/ideogram4_custom_blocks

Ideogram4 custom modular blocks

Custom Modular Diffusers blocks that extend Ideogram4 with image-to-image and Differential Diffusion, plus a unified AutoBlocks that folds text-to-image, img2img, and differential diffusion into a single pipeline, the workflow is chosen automatically from which inputs you pass.

prompt                          -> text2image
prompt + image                  -> image2image        (optional strength)
prompt + image + diffdiff_map   -> differential diffusion

Loading & running

import torch
from diffusers import ModularPipeline

pipe = ModularPipeline.from_pretrained("OzzyGT/ideogram4_custom_blocks", trust_remote_code=True)
pipe.load_components(
    names=["text_encoder", "tokenizer", "transformer", "unconditional_transformer", "vae", "scheduler"],
    torch_dtype=torch.bfloat16,
)
pipe.to("cuda")

image = pipe(prompt="a photo of a red apple on a black background", height=1024, width=1024, output="images")[0]
image.save("out.png")

image-to-image

Pass an image (and optional strength) to the same pipe:

from diffusers.utils import load_image

image = load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/20240329211129_4024911930.png")

result = pipe(
    prompt="a photo of a snowy mountain landscape at sunset, dramatic clouds",
    image=image,
    strength=0.6,
    output="images",
)[0]
result.save("img2img.png")

differential diffusion

Add a grayscale change map (diffdiff_map) alongside the image — darker regions change more, brighter regions stay closer to the reference:

from diffusers.utils import load_image

image = load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/20240329211129_4024911930.png")
diffdiff_map = load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/gradient_mask.png")

result = pipe(
    prompt="a photo of a snowy mountain landscape at sunset, dramatic clouds",
    image=image,
    diffdiff_map=diffdiff_map,
    output="images",
)[0]
result.save("diffdiff.png")