This script enables you to webscrape using the Laion parquet files which are available on Huggingface.co.
It has been tested with 2B-en-aesthetic, but may need minor tweaks for some other datasets that contain different columns. Keep in mind some other files are purely sidecar metadata.
The script will rename downloaded files to the best of its ability to the TEXT (caption) of the image with the original file extension, which can be plugged into the new class of caption-capable DreamBooth apps or the EveryDream trainer that will use the filename as the prompt for training.
One suggested use is to take this data and replace regularization images with ground truth data from the Laion dataset.
It should execute quite quickly as it uses async task gathers for the the HTTP and fileio work.
Default folders are /laion for the parquest files and /output for downloaded images relative to the root folder, but consider disk space and point to another location if needed.
## Examples
Query all the parquet files in ./laion for any image with a caption (TEXT) containing "a man" and attempt top stop after downloading (approximately) 50 files:
python scripts/download_laion.py --search_text "a man" --limit 50
Query for person with a leading and trailing space:
python scripts/download_laion.py --search_text " person " --limit 200
Query for both "man" and "photo" anywhere in the caption, and write them to z:/myDumpFolder instead of the default folder. Useful if you need to put them on another drive, NAS, etc. The default limit of 100 images will apply since --limit is omitted:
