403f7ddf07
fix chunked_shuffle crash with empty list
2023-06-17 11:04:14 +02:00
ded73f088f
cleanup
2023-06-17 11:04:14 +02:00
8885d58efd
create bucket if it doesn't exist
2023-06-17 11:04:14 +02:00
bd89ad96d2
typo
2023-06-17 11:04:14 +02:00
e4872fdc0c
improve shuffle and runt handling for named buckets
2023-06-17 11:04:14 +02:00
a047294676
assert an important precondition
2023-06-08 11:01:16 +02:00
4f98f0bcc9
ensure predictable shuffle behaviour and further cleanup
2023-06-08 10:50:51 +02:00
1874a38663
cleanup
2023-06-08 10:39:32 +02:00
b3c5d656e3
refactor for clarity
2023-06-08 10:31:10 +02:00
86f80a8776
allow empty default_batch
2023-06-07 18:39:13 +02:00
59fc9891d4
shuffle named batches while respecting and accounting for grad_accum
2023-06-07 18:07:37 +02:00
ba95b8c6d1
simplify runt handling
2023-06-05 01:04:21 +02:00
53d0686086
add a batch_id.txt file to subfolders or a batch_id key to local yaml to force images for that folder to be processed in the same batch
2023-06-05 01:02:27 +02:00
aad00eab2e
switch up errored item logic flow
2023-04-29 13:03:10 -04:00
ce85ce30ae
warn on chronically underfilled aspect ratio buckets
2023-04-29 13:03:10 -04:00
8a8a4cf3df
make val optional, revert multiply algo
2023-02-08 13:04:12 -05:00
19347bcaa8
make fractional multiplier logic apply per-directory
2023-02-08 14:15:54 +01:00
a7b00e9ef3
fix multiplier logic
2023-02-08 13:46:58 +01:00
4e37200dda
fix multiplier issues with validation and refactor validation logic
2023-02-08 11:28:45 +01:00
29396ec21b
update EveryDreamValidator for noprompt's changes
2023-02-07 17:46:30 +01:00
41c9f36ed7
GH-36: Add support for validation split (WIP)
...
Co-authored-by: Damian Stewart <office@damianstewart.com>
2023-02-06 22:10:34 -08:00
3fe335f328
Update documentation
2023-01-29 17:47:10 -08:00
326d861a86
Push DLMA into main, pass config to resolve
...
This patch
* passes the configuration (`argparse.Namespace`) to the resolver,
* pushes the DLMA code into the main function,
* makes DLMA take a `list[ImageTrainItem]` instead of `data_root`,
* makes `EveryDreamBatch` take `DLMA` instead of `data_root`, etc.
* allows `data_root` to be a list.
By doing these things, both `EveryDreamBatch` and DLMA can be free from
data resolution logic. It also reduces the number of arguments which
need to be passed down to EDB and DLMA.
2023-01-29 17:08:54 -08:00
9639237762
minor fix to multiply.txt stuff and undersized images txt output new line
2023-01-27 13:58:14 -05:00
e6d1096637
Filter out items with errors
2023-01-24 08:33:20 -08:00
620b157e6a
Fix some name errors
2023-01-23 16:57:02 -08:00
1a0b7994f4
Move target_wh calculation to ImageTrainItem
2023-01-23 12:00:42 -08:00
c1a66317cd
Forgot to set prepared_train_data
2023-01-23 11:16:34 -08:00
b6c7299baf
Merge branch 'main' into refactor-data-resolution
2023-01-23 08:43:23 -08:00
316df2db7e
Use data_resolver.resolve for data loading in data_loader
2023-01-23 00:15:32 -08:00
9c6df69e4e
Use DirectoryResolver.unzip_all
2023-01-22 23:14:16 -08:00
08813eabb5
Use DirectoryResolver.recurse_data_root
2023-01-22 23:13:05 -08:00
18d1da0459
bug fix and multiplytxt fraction stuff
2023-01-22 18:59:59 -05:00
24b00ab35b
add fractional support for multiply.txt
2023-01-22 01:15:50 -05:00
36ece59660
better undersized log file
2023-01-20 16:23:56 -05:00
1c2708dc63
few sanity checks and remove keyboard
2023-01-20 09:42:24 -05:00
711e263e24
Implemented system to train on a subset of the dataset, favouring higher rated images
2023-01-14 16:18:22 +01:00
fb73c16515
Make file name extension check case insensitive, so .JPG or .PNG files will also be detected
2023-01-10 11:54:26 +01:00
f47ceadcc7
Implemented an optimization for the shuffling if all tags have the same weight and added documentation.
2023-01-09 21:53:46 +01:00
3d2709ace9
Implemented loading captions from yaml file
2023-01-09 21:53:07 +01:00
a3618409bc
Support more control regarding caption tag shuffeling using yaml files
2023-01-09 21:53:07 +01:00
b316684bdb
bunch of updates, grad ckpting, no drop bucket, shuffle every epoch
2023-01-01 10:45:18 -05:00
051116a7d9
add gpu id support
2022-12-29 21:11:06 -05:00
4c53f2d55c
various tweaks and bugfixes over holidays
2022-12-27 14:25:32 -05:00
aff2377209
docs, update setup
2022-12-18 13:03:44 -05:00
179fd5395b
hey look ed2
2022-12-17 22:32:48 -05:00
Damian Stewart