From a3899d56fd33bf31e79324d6ea6f5ee5885c08d8 Mon Sep 17 00:00:00 2001 From: Patrick von Platen Date: Wed, 15 Jun 2022 12:15:33 +0200 Subject: [PATCH] add more readmes --- src/diffusers/models/README.md | 11 +++++++++++ src/diffusers/pipelines/+ | 19 +++++++++++++++++++ src/diffusers/pipelines/README.md | 19 +++++++++++++++++++ src/diffusers/schedulers/README.md | 18 ++++++++++++++++++ 4 files changed, 67 insertions(+) create mode 100644 src/diffusers/models/README.md create mode 100644 src/diffusers/pipelines/+ create mode 100644 src/diffusers/pipelines/README.md create mode 100644 src/diffusers/schedulers/README.md diff --git a/src/diffusers/models/README.md b/src/diffusers/models/README.md new file mode 100644 index 00000000..35f09b29 --- /dev/null +++ b/src/diffusers/models/README.md @@ -0,0 +1,11 @@ +# Models + +- Models: Neural network that models p_θ(x_t-1|x_t) (see image below) and is trained end-to-end to denoise a noisy input to an image. Examples: UNet, Conditioned UNet, 3D UNet, Transformer UNet + +## API + +TODO(Suraj, Patrick) + +## Examples + +TODO(Suraj, Patrick) diff --git a/src/diffusers/pipelines/+ b/src/diffusers/pipelines/+ new file mode 100644 index 00000000..890a27c0 --- /dev/null +++ b/src/diffusers/pipelines/+ @@ -0,0 +1,19 @@ +# Pipelines + +- Pipelines are a collection of end-to-end diffusion systems that can be used out-of-the-box +- Pipelines should stay as close as possible to their original implementation +- Pipelines can include components of other library, such as text-encoders. + +## API + +TODO(Patrick, Anton, Suraj) + +## Examples + +- DDPM for unconditional image generation in [pipeline_ddpm](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_ddpm.py). +- DDIM for unconditional image generation in [pipeline_ddim](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_ddim.py). +- PNDM for unconditional image generation in [pipeline_pndm](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_pndm.py). +- Latent diffusion for text to image generation / conditional image generation in [pipeline_ddpm](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_bddm.py). +- Glide for text to image generation / conditional image generation in [pipeline_ddpm](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_bddm.py). +- BDDM for spectrogram-to-sound vocoding in [pipeline_ddpm](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_bddm.py). +- Grad-TTS for text to audio generation / conditional audio generation in [pipeline_ddpm](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_bddm.py). diff --git a/src/diffusers/pipelines/README.md b/src/diffusers/pipelines/README.md new file mode 100644 index 00000000..61e653a8 --- /dev/null +++ b/src/diffusers/pipelines/README.md @@ -0,0 +1,19 @@ +# Pipelines + +- Pipelines are a collection of end-to-end diffusion systems that can be used out-of-the-box +- Pipelines should stay as close as possible to their original implementation +- Pipelines can include components of other library, such as text-encoders. + +## API + +TODO(Patrick, Anton, Suraj) + +## Examples + +- DDPM for unconditional image generation in [pipeline_ddpm](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_ddpm.py). +- DDIM for unconditional image generation in [pipeline_ddim](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_ddim.py). +- PNDM for unconditional image generation in [pipeline_pndm](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_pndm.py). +- Latent diffusion for text to image generation / conditional image generation in [pipeline_latent_diffusion](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_latent_diffusion.py). +- Glide for text to image generation / conditional image generation in [pipeline_glide](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_glide.py). +- BDDM for spectrogram-to-sound vocoding in [pipeline_bddm](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_bddm.py). +- Grad-TTS for text to audio generation / conditional audio generation in [pipeline_grad_tts](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_grad_tts.py). diff --git a/src/diffusers/schedulers/README.md b/src/diffusers/schedulers/README.md new file mode 100644 index 00000000..70514e07 --- /dev/null +++ b/src/diffusers/schedulers/README.md @@ -0,0 +1,18 @@ +# Schedulers + +- Schedulers are the algorithms to use diffusion models in inference as well as for training. They include the noise schedules and define algorithm-specific diffusion steps. +- Schedulers can be used interchangable between diffusion models in inference to find the preferred tradef-off between speed and generation quality. +- Schedulers are available in numpy, but can easily be transformed into PyTorch. + +## API + +- Schedulers should provide one or more `def step(...)` functions that should be called iteratively to unroll the diffusion loop during +the forward pass. +- Schedulers should be framework-agonstic, but provide a simple functionality to convert the scheduler into a specific framework, such as PyTorch +with a `set_format(...)` method. + +## Examples + +- The DDPM scheduler was proposed in [Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) and can be found in [scheduling_ddpm.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_ddpm.py). An example of how to use this scheduler can be found in [pipeline_ddpm.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_ddpm.py). +- The DDIM scheduler was proposed in [Denoising Diffusion Implicit Models](https://arxiv.org/abs/2010.02502) and can be found in [scheduling_ddim.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_ddim.py). An example of how to use this scheduler can be found in [pipeline_ddim.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_ddim.py). +- The PNMD scheduler was proposed in [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://arxiv.org/abs/2202.09778) and can be found in [scheduling_pndm.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_pndm.py). An example of how to use this scheduler can be found in [pipeline_pndm.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_pndm.py).