Mirrors
/
EveryDream2trainer
mirror of https://github.com/victorchall/EveryDream2trainer.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

update lr recommendation

This commit is contained in:
Victor Hall 2023-03-09 21:02:05 -05:00
parent 6c41be3539
commit e5aa70c54f
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
doc/ATWEAKING.md
View File

@ -46,10 +46,12 @@ The value is defaulted at 0.04, which means 4% conditional dropout. You can set
Learning rate adjustment is a very important part of training. You can use the default settings, or you can tweak it. You should consider increasing this further if you increase your batch size further (10+) using [gradient checkpointing](#gradient_checkpointing).
--lr 1.5e-6 ^
--lr 1.0e-6 ^
By default, the learning rate is constant for the entire training session. However, if you want it to change by itself during training, you can use cosine.
General suggestion is 1e-6 for training SD1.5 at 512 resolution. For SD2.1 at 768, try a much lower value, such as 2e-7. [Validation](VALIDATION.md) can be helpful to tune learning rate.
## Clip skip
Aka "penultimate layer", this takes the output from the text encoder not from its last output layer, but layers before.