2. Fixed findlast option for regular models not to load EMA models by default.
3. findlast can be used to load EMA model too when used with ema_decay_resume_model.
4. Added ema_device variable to store the device in torch type.
5. Cleaned prints and comments.
1. EMA decay. Using EMA decay model, it is updated every ema_decay_interval by (1 - ema_decay_rate), it can be stored on CPU to save VRAM. Only EMA model is saved now.
2. min_snr_gamma - improve converging speed, more info: https://arxiv.org/abs/2303.09556
3. load_settings_every_epoch - Will load 'train.json' at start of every epoch.
commit 0f890f2d6bbccee225f738934f4c4450323f19a2
Merge: c008c40 003b089
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 14 11:47:40 2023 +0200
Merge remote-tracking branch 'upstream/main' into feat_te_last_n_layers_unsquashed
commit c008c404f19ebc6b78085f42a4e39aeb2ba00d04
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 14 11:23:20 2023 +0200
finalize TE layer freezing
commit 7377b10d59e32a6fea5d321a598ae4504e1a9f36
Author: Damian Stewart <d@damianstewart.com>
Date: Thu May 11 20:45:28 2023 +0200
remove zero_lr method
commit 4af13ba816c2811d7b5bd6fbb81a32bca6747e99
Author: Damian Stewart <d@damianstewart.com>
Date: Thu May 11 20:05:01 2023 +0200
Revert "rename parameters"
This reverts commit aa33c61337599ab2d90b34aaf8c3d36fd4edf147.
commit aa33c61337599ab2d90b34aaf8c3d36fd4edf147
Author: Damian Stewart <d@damianstewart.com>
Date: Tue May 9 00:28:00 2023 +0200
rename parameters
commit 1da867e6fadb873da2571371a73b522406d76a18
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 7 22:28:29 2023 +0200
remove silly check
commit 483cb2a635c3fe5a044edf4ea8de095bedc3f0ac
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 7 20:53:43 2023 +0200
use 1e-10 not 0 as 'zero' lr
commit e5d230e6c765a7e25dc6381d09bd0a66a9a54ec2
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 7 20:51:51 2023 +0200
add experimental 'zero_lr' freeze method
commit bcf24ee59a443c0ee71d622e65e1043b547f845e
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 7 17:32:11 2023 +0200
fix layer selection bug
commit 7ee33eff8740e095f85042dcbb792e025b179c6c
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 7 17:25:25 2023 +0200
put back the 'drop' method and make accessible
commit 76dfbf6dd6f43f3aa9a7f4629baa8e86573d9520
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 7 16:39:05 2023 +0200
wip getting final_layer_norm to work
commit a19d43651a87525251106ed57238cd2cd1c3f3ff
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 7 16:15:53 2023 +0200
work around a crash when freeze_final_layer_norm is True
commit c2a44eb25132941b92e2ecd0be3682ae3c6838c2
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 7 15:47:10 2023 +0200
improve logging, add extra freezing controls
commit a31e64c4c0d12dfb6583dd6f22c8c09ba7840410
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 7 13:46:38 2023 +0200
alternative method to freeze early TE layers
commit 095692fd4ea53707c012217898321860d8b9329f
Merge: 876072c 4c5ce81
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 7 11:52:51 2023 +0200
Merge branch 'victorchall:main' into feat_te_last_n_layers
commit 876072c46394fde721a6026f7a6ef72ccb150ddb
Author: Damian Stewart <d@damianstewart.com>
Date: Sun May 7 01:41:50 2023 +0200
implement last N layers training only for TE
Victor Hall