From cdc0b39584cccfe3d11075c782df38ad8b883658 Mon Sep 17 00:00:00 2001 From: Victor Hall Date: Wed, 17 May 2023 14:54:16 -0400 Subject: [PATCH] Update OPTIMIZER.md --- doc/OPTIMIZER.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/OPTIMIZER.md b/doc/OPTIMIZER.md index 3932f58..3009bdf 100644 --- a/doc/OPTIMIZER.md +++ b/doc/OPTIMIZER.md @@ -51,7 +51,7 @@ The recommendations are based on "1/10th LR" but "10x the weight decay" compared LR can be set in `optimizer.json` and excluded from the main CLI arg or train.json but if you use the main CLI arg or set it in the main train.json it will override the setting. This was done to make sure existing behavior will not break. To set LR in the `optimizer.json` make sure to delete `"lr": 1.3e-6` in your main train.json and exclude the CLI arg. -The text encoder LR can run at a different value to the Unet LR. This may help prevent over-fitting, especially if you're training from SD2 checkpoints. To set the text encoder LR, add a value for `text_encoder_lr_scale` to `optimizer.json` or set the `text_encoder: lr` to its own value (not null). For example, to train the text encoder with an LR that is half that of the Unet, add `"text_encoder_lr_scale": 0.5` to `optimizer.json`. The default value is `0.5`, meaning the text encoder will be trained at half the learning rate of the unet. +The text encoder LR can run at a different value to the Unet LR. This may help prevent over-fitting, especially if you're training from SD2 checkpoints. ## Text Encoder freezing