| --- |
| pipeline_tag: image-feature-extraction |
| license: mit |
| library_name: diffusion-single-file |
| --- |
| |
| # CleanDIFT Model Card |
|
|
|
|
| Diffusion models learn powerful world representations that have proven valuable for tasks like semantic correspondence detection, |
| depth estimation, semantic segmentation, and classification. |
| However, diffusion models require noisy input images, which destroys information and introduces the noise level as a hyperparameter that needs to be tuned for each task. |
|
|
|
|
|
|
|
|
| We introduce CleanDIFT, a novel method to extract noise-free, timestep-independent features by enabling diffusion models to work directly with clean input images. |
| The approach is efficient, training on a single GPU in just 30 minutes. We publish these models alongside our paper ["CleanDIFT: Diffusion Features without Noise"](https://compvis.github.io/cleandift/). |
|
|
| We provide checkpoints for Stable Diffusion 1.5 and Stable Diffusion 2.1. |
|
|
|
|
| ## Usage |
|
|
| For detailed examples on how to extract features with CleanDIFT and how to use them for downstream tasks, please refer to the notebooks provided [here](https://github.com/CompVis/CleanDIFT/tree/main/notebooks). |
|
|
| Our checkpoints are fully compatible with the `diffusers` library. |
| If you already have a pipeline using SD 1.5 or SD 2.1 from `diffusers`, you can simply replace the U-Net state dict: |
| ```python |
| from diffusers import UNet2DConditionModel |
| from huggingface_hub import hf_hub_download |
| |
| unet = UNet2DConditionModel.from_pretrained("stabilityai/stable-diffusion-2-1", subfolder="unet") |
| ckpt_pth = hf_hub_download(repo_id="CompVis/cleandift", filename="cleandift_sd21_unet.safetensors") |
| state_dict = load_file(ckpt_pth) |
| unet.load_state_dict(state_dict, strict=True) |
| ``` |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{stracke2024cleandiftdiffusionfeaturesnoise, |
| title={CleanDIFT: Diffusion Features without Noise}, |
| author={Nick Stracke and Stefan Andreas Baumann and Kolja Bauer and Frank Fundel and Björn Ommer}, |
| year={2024}, |
| eprint={2412.03439}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CV}, |
| url={https://arxiv.org/abs/2412.03439}, |
| } |
| ``` |
