1ab27a9059
better check for null manual_data_root
2023-06-17 11:04:14 +02:00
dd6b37840d
permit empty manual_data_root
2023-06-17 11:04:14 +02:00
6c60b76fb6
log combined loss if there are >1 val subsets
2023-06-17 11:04:14 +02:00
403f7ddf07
fix chunked_shuffle crash with empty list
2023-06-17 11:04:14 +02:00
ded73f088f
cleanup
2023-06-17 11:04:14 +02:00
8885d58efd
create bucket if it doesn't exist
2023-06-17 11:04:14 +02:00
bd89ad96d2
typo
2023-06-17 11:04:14 +02:00
e4872fdc0c
improve shuffle and runt handling for named buckets
2023-06-17 11:04:14 +02:00
a047294676
assert an important precondition
2023-06-08 11:01:16 +02:00
4f98f0bcc9
ensure predictable shuffle behaviour and further cleanup
2023-06-08 10:50:51 +02:00
1874a38663
cleanup
2023-06-08 10:39:32 +02:00
b3c5d656e3
refactor for clarity
2023-06-08 10:31:10 +02:00
86f80a8776
allow empty default_batch
2023-06-07 18:39:13 +02:00
59fc9891d4
shuffle named batches while respecting and accounting for grad_accum
2023-06-07 18:07:37 +02:00
ba95b8c6d1
simplify runt handling
2023-06-05 01:04:21 +02:00
53d0686086
add a batch_id.txt file to subfolders or a batch_id key to local yaml to force images for that folder to be processed in the same batch
2023-06-05 01:02:27 +02:00
a96c6e2166
dadapt stuff
2023-06-03 11:26:53 -04:00
9ee2effacd
add 320 res
2023-06-01 21:19:20 -04:00
97f1160496
correct image cropping
2023-06-01 21:06:55 -04:00
c0e7c4adf9
errant print
2023-06-01 19:23:43 -04:00
615fa929e5
errant print
2023-06-01 19:17:43 -04:00
0e0b546ef7
help debug crop error someone got
2023-06-01 19:11:22 -04:00
56deb26a59
early work on shuffle_tags.txt and add try around trimming
2023-06-01 16:18:21 -04:00
7dcfa7acbf
Speed up preloading by not loading pixel data
...
PIL lazy loads pixel data, so image size can be accessed without loading the full image.
This commit makes it so only image size and EXIF metadata are fetched from disk, speeding up the preload stage while still supporting transposing images.
2023-05-31 20:48:48 -04:00
5c98cdee70
update to torch2, xformers 20, bnb 381
2023-05-30 22:15:02 -04:00
f3468fe7e7
Merge remote-tracking branch 'upstream/main' into val_partial_epochs
2023-05-07 02:11:58 +02:00
8e28a74386
add migration code for
2023-04-29 16:55:02 -04:00
360eaa5a42
don't print, using logging
2023-04-29 16:55:02 -04:00
413f981512
implement extra manual validation splits
2023-04-29 16:55:02 -04:00
aad00eab2e
switch up errored item logic flow
2023-04-29 13:03:10 -04:00
ce85ce30ae
warn on chronically underfilled aspect ratio buckets
2023-04-29 13:03:10 -04:00
630ac5e46a
Merge pull request #152 from qslug/yaml-tag-str
...
Always convert YAML tags to strings
2023-04-22 19:03:04 -04:00
2e8056174d
Always convert YAML tags to strings
2023-04-23 00:36:32 +02:00
e3e30a5599
fix exif issues
2023-04-16 14:53:34 -04:00
3697b16344
put exif transpose back in preloading to fix bug
2023-04-16 02:19:27 -04:00
b8f637873f
removing the convert_rgb flag for a simpler design
2023-04-14 20:21:48 -05:00
a839180199
set # of data loaders by the min of batch size or cpu count, do not do an rg b conversion when only loading image metadata
2023-04-14 14:59:28 -05:00
2bb35eaa0a
Use filenames for caption if no main prompt in yaml
2023-04-14 00:59:26 +02:00
dd98ebe080
Add error handling for bad exif, and rotate before sizing
2023-03-28 19:19:08 +02:00
56256ab9ef
attempt to catch some errors reported on github
2023-03-26 11:52:49 -04:00
35d52b56e0
added some resolutions, option for val-loss pos-neg, fix wandb
2023-03-25 20:09:06 -04:00
cdafa2dc43
Avoid unnecessary iteration over files in folder
2023-03-21 12:09:52 +01:00
161e0a563c
Prioritize tags, `image > local > global`, but respect weights
2023-03-21 00:15:53 +01:00
fae0b3c535
Retain original tag order when parsing captions
2023-03-19 23:30:42 +01:00
5afd75fd98
try-except around preloading items to help troubleshoot issues
2023-03-18 22:27:09 -04:00
f01a7354f0
remove sort from dataset due to slowdown on large sets, add contribution readme
2023-03-18 22:24:03 -04:00
48f132554c
Assign default value for MAX_CAPTION_LENGTH
2023-03-15 21:52:32 +01:00
ba687de8b4
add pbar back to preloading, remove cruft from testing loss stuff
2023-03-15 12:06:29 -04:00
605716a646
conf
2023-03-15 11:22:40 -04:00
da3c183cc5
autofix exif orientation
2023-03-15 11:12:59 -04:00
Damian Stewart