2022-11-08 21:00:54 -07:00
{
"cells": [
{
"cell_type": "markdown",
"id": "676114ae",
"metadata": {},
"source": [
"## Every Dream trainer\n",
"\n",
"You will need your data prepared first before starting! Don't waste rental fees if you're not ready to upload your files. Your files should be captioned before you start with either the caption as the filename or in text files for each image alongside the image files. See main README.md for more details. Tools are available to automatically caption your files.\n",
"\n",
"[Instructions](https://github.com/victorchall/EveryDream-trainer/blob/main/README.md)\n",
"\n",
"If you can sign up for Runpod here (shameless referral link): [Runpod](https://runpod.io?ref=oko38cd0)\n",
"\n",
"If you are confused by the wall of text, join the discord here: [EveryDream Discord](https://discord.gg/uheqxU6sXN)\n",
"\n",
"Make sure you have at least 40GB of Runpod **Volume** storage at a minimum so you don't waste training just 1 ckpt that is overtrained and have to start over. Penny pinching on storage is ultimately a waste of your time and money! This is setup to give you more than one ckpt so you don't overtrain.\n",
"\n",
"### Starting model\n",
"Make sure you have your hugging face token ready to download the 1.5 mode. You can get one here: https://huggingface.co/settings/tokens\n",
"If you don't have a User Access Token, create one. Or you can upload a starting checkpoint instead of using the HF download and skip that step, but you'll need to modify the starting model name when you start training (more info below)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bb6d14b7-3c37-4ec4-8559-16b4e9b8dd18",
"metadata": {},
"outputs": [],
"source": [
2022-11-08 23:34:47 -07:00
"I_LIKE_FUN_STUFF = True\n",
"\n",
2022-11-08 21:00:54 -07:00
"!git clone https://github.com/victorchall/everydream-trainer\n",
2022-11-08 23:34:47 -07:00
"%cd everydream-trainer\n",
"\n",
"import scripts.fun\n",
"from IPython.display import Audio\n",
"finish_msg = codecs.encode('QBAR', 'rot_13')\n",
"sound_file = 'demo/wow.mp3'\n",
"\n",
"print(finish_msg)"
2022-11-08 21:00:54 -07:00
]
},
{
"cell_type": "markdown",
"id": "589bfca0",
2022-11-08 23:34:47 -07:00
"metadata": {
"tags": []
},
2022-11-08 21:00:54 -07:00
"source": [
"## Install dependencies\n",
2022-11-08 23:34:47 -07:00
"\n",
"This will take a couple minutes. You can ignore \"warnings.\""
2022-11-08 21:00:54 -07:00
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ab559338",
"metadata": {},
"outputs": [],
"source": [
"!pip install -q omegaconf\n",
"!pip install -q einops\n",
"!pip install -q pytorch-lightning==1.6.5\n",
"!pip install -q test-tube\n",
"!pip install -q transformers==4.19.2\n",
"!pip install -q kornia\n",
"!pip install -e git+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformers\n",
"!pip install -e git+https://github.com/openai/CLIP.git@main#egg=clip\n",
"!pip install -q setuptools==59.5.0\n",
"!pip install -q pillow==9.0.1\n",
"!pip install -q torchmetrics==0.6.0\n",
"!pip install -e .\n",
"#!pip install -qq diffusers[\"training\"]==0.3.0 transformers ftfy\n",
"!pip install huggingface_hub\n",
2022-11-08 23:34:47 -07:00
"!pip install ipywidgets==7.7.1\n",
"import ipywidgets as widgets\n",
"print(finish_msg)\n",
"Audio(sound_file, autoplay=I_LIKE_FUN_STUFF)"
2022-11-08 21:00:54 -07:00
]
},
{
"cell_type": "markdown",
"id": "c230d91a",
"metadata": {},
"source": [
"## Now that dependencies are installed, ready to move on!"
]
},
{
"cell_type": "markdown",
"id": "17affc47",
"metadata": {},
"source": [
"## Log into huggingface\n",
"Run the cell below and paste your token into the prompt. You can get your token from your huggingface account page.\n",
"\n",
"The token will not show on the screen, just press enter after you paste it.\n",
"\n",
"Then run the following cell to download the base checkpoint (may take a minute)."
]
},
{
"cell_type": "code",
2022-11-08 23:34:47 -07:00
"execution_count": 17,
2022-11-08 21:00:54 -07:00
"id": "02c8583e",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
2022-11-08 23:34:47 -07:00
"model_id": "3b16e2951c6549f589afaadc67e5ac9d",
2022-11-08 21:00:54 -07:00
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(HTML(value='<center> <img\\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.sv…"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from huggingface_hub import notebook_login\n",
"\n",
"notebook_login()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "503322f5",
"metadata": {},
"outputs": [],
"source": [
"from huggingface_hub import hf_hub_download\n",
"downloaded_model_path = hf_hub_download(\n",
" repo_id=\"runwayml/stable-diffusion-v1-5\",\n",
" filename=\"v1-5-pruned.ckpt\",\n",
" use_auth_token=True\n",
2022-11-08 23:34:47 -07:00
")\n",
"print(finish_msg)"
2022-11-08 21:00:54 -07:00
]
},
{
"cell_type": "markdown",
"id": "0bf1e8cd",
"metadata": {},
"source": [
"# Upload training files\n",
"\n",
2022-11-08 23:34:47 -07:00
"Ues the navigation on the left to open **\"every-dream-trainer\"** folder, then open **\"input\"** and upload your training files using the **up arrow button** above the file explorer. \n",
2022-11-08 21:00:54 -07:00
"\n",
"You can check there are files in the folder by running the cell below (optional, just prints first 10)."
]
},
{
"cell_type": "code",
2022-11-08 23:34:47 -07:00
"execution_count": null,
"id": "af711a9c-7a50-49a1-a571-439d62a9587c",
2022-11-08 21:00:54 -07:00
"metadata": {},
2022-11-08 23:34:47 -07:00
"outputs": [],
2022-11-08 21:00:54 -07:00
"source": [
2022-11-08 23:34:47 -07:00
"!ls -U input | head -10\n",
"# at least a few image filenames should show below, if not you uploaded to the wrong folder"
2022-11-08 21:00:54 -07:00
]
},
{
"cell_type": "markdown",
"id": "873d9f3f",
"metadata": {},
"source": [
"## Tweak your YAML\n",
2022-11-08 23:34:47 -07:00
"You can adjust the YAML file to change the training parameters. The v1-fine yamls are in the everydream-trainer/configs/stable-diffusion folder. \n",
2022-11-08 21:00:54 -07:00
"\n",
2022-11-08 23:34:47 -07:00
"You can also upload your own in that folder and use it below.\n",
"\n",
"Instructions are here: https://github.com/victorchall/EveryDream-trainer/blob/main/README.md (hopefully you already read this?)\n",
2022-11-08 21:00:54 -07:00
"\n",
"[Runpod YAML](everydream-trainer/configs/stable-diffusion/v1-finetune_runpod.yaml) is a good starting point for small datasets (30-50 images) and is the default in the command below. It will only keep 2 checkpoints.\n",
"\n",
"[EveryDream YAML](workspace/everydream-trainer/configs/stable-diffusion/v1-finetune_everydream.yaml) is a good starting point for large datasets. You will need to change the filename in the --config parameter below to use this. This may create a LOT of large ckpt files while training, so make sure you have enough space in your runpod instance! 60GB+ is recommended."
]
},
{
"cell_type": "markdown",
"id": "25ea006b",
"metadata": {},
"source": [
"# Run the trainer\n",
"This will take a while. Make sure when it finishes you scroll down to run the last cell."
]
},
{
"cell_type": "code",
"execution_count": null,
2022-11-08 23:34:47 -07:00
"id": "65e8a052-bc26-4339-81e0-3b4f78c1d286",
2022-11-08 21:00:54 -07:00
"metadata": {},
"outputs": [],
"source": [
2022-11-08 23:34:47 -07:00
"# change to use a custom yaml, but run this cell no matter what\n",
"my_yaml = \"configs/stable-diffusion/v1-finetune_runpod.yaml\""
2022-11-08 21:00:54 -07:00
]
},
{
2022-11-08 23:34:47 -07:00
"cell_type": "code",
"execution_count": 1,
"id": "c12e7cf3-42be-4537-a4f7-5723c0248562",
2022-11-08 21:00:54 -07:00
"metadata": {},
2022-11-08 23:34:47 -07:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running on GPUs 0,\n",
"Loading model from v1-5-pruned.ckpt\n",
"Instantiating config for: ldm.models.diffusion.ddpm.LatentDiffusion with config:\n",
"{'linear_start': 0.00085, 'linear_end': 0.012, 'num_timesteps_cond': 1, 'log_every_t': 200, 'timesteps': 1000, 'first_stage_key': 'image', 'cond_stage_key': 'caption', 'image_size': 64, 'channels': 4, 'cond_stage_trainable': True, 'conditioning_key': 'crossattn', 'monitor': 'val/loss_simple_ema', 'scale_factor': 0.18215, 'use_ema': False, 'unfreeze_model': True, 'model_lr': 1e-06, 'unet_config': {'target': 'ldm.modules.diffusionmodules.openaimodel.UNetModel', 'params': {'image_size': 32, 'in_channels': 4, 'out_channels': 4, 'model_channels': 320, 'attention_resolutions': [4, 2, 1], 'num_res_blocks': 2, 'channel_mult': [1, 2, 4, 4], 'num_heads': 8, 'use_spatial_transformer': True, 'transformer_depth': 1, 'context_dim': 768, 'use_checkpoint': True, 'legacy': False}}, 'first_stage_config': {'target': 'ldm.models.autoencoder.AutoencoderKL', 'params': {'embed_dim': 4, 'monitor': 'val/rec_loss', 'ddconfig': {'double_z': True, 'z_channels': 4, 'resolution': 512, 'in_channels': 3, 'out_ch': 3, 'ch': 128, 'ch_mult': [1, 2, 4, 4], 'num_res_blocks': 2, 'attn_resolutions': [], 'dropout': 0.0}, 'lossconfig': {'target': 'torch.nn.Identity'}}}, 'cond_stage_config': {'target': 'ldm.modules.encoders.modules.FrozenCLIPEmbedder'}, 'ckpt_path': 'v1-5-pruned.ckpt'}\n",
"LatentDiffusion: Running in eps-prediction mode\n",
"Instantiating config for: ldm.modules.diffusionmodules.openaimodel.UNetModel with config:\n",
"{'image_size': 32, 'in_channels': 4, 'out_channels': 4, 'model_channels': 320, 'attention_resolutions': [4, 2, 1], 'num_res_blocks': 2, 'channel_mult': [1, 2, 4, 4], 'num_heads': 8, 'use_spatial_transformer': True, 'transformer_depth': 1, 'context_dim': 768, 'use_checkpoint': True, 'legacy': False}\n",
"DiffusionWrapper has 859.52 M params.\n",
"Instantiating config for: ldm.models.autoencoder.AutoencoderKL with config:\n",
"{'embed_dim': 4, 'monitor': 'val/rec_loss', 'ddconfig': {'double_z': True, 'z_channels': 4, 'resolution': 512, 'in_channels': 3, 'out_ch': 3, 'ch': 128, 'ch_mult': [1, 2, 4, 4], 'num_res_blocks': 2, 'attn_resolutions': [], 'dropout': 0.0}, 'lossconfig': {'target': 'torch.nn.Identity'}}\n",
"making attention of type 'vanilla' with 512 in_channels\n",
"Working with z of shape (1, 4, 64, 64) = 16384 dimensions.\n",
"making attention of type 'vanilla' with 512 in_channels\n",
"Instantiating config for: torch.nn.Identity with config:\n",
"{}\n",
"Instantiating config for: ldm.modules.encoders.modules.FrozenCLIPEmbedder with config:\n",
"{}\n",
"Restored from v1-5-pruned.ckpt with 0 missing and 688 unexpected keys\n",
"Unexpected Keys: ['model_ema.decay', 'model_ema.num_updates', 'model_ema.diffusion_modeltime_embed0weight', 'model_ema.diffusion_modeltime_embed0bias', 'model_ema.diffusion_modeltime_embed2weight', 'model_ema.diffusion_modeltime_embed2bias', 'model_ema.diffusion_modelinput_blocks00weight', 'model_ema.diffusion_modelinput_blocks00bias', 'model_ema.diffusion_modelinput_blocks10in_layers0weight', 'model_ema.diffusion_modelinput_blocks10in_layers0bias', 'model_ema.diffusion_modelinput_blocks10in_layers2weight', 'model_ema.diffusion_modelinput_blocks10in_layers2bias', 'model_ema.diffusion_modelinput_blocks10emb_layers1weight', 'model_ema.diffusion_modelinput_blocks10emb_layers1bias', 'model_ema.diffusion_modelinput_blocks10out_layers0weight', 'model_ema.diffusion_modelinput_blocks10out_layers0bias', 'model_ema.diffusion_modelinput_blocks10out_layers3weight', 'model_ema.diffusion_modelinput_blocks10out_layers3bias', 'model_ema.diffusion_modelinput_blocks11normweight', 'model_ema.diffusion_modelinput_blocks11normbias', 'model_ema.diffusion_modelinput_blocks11proj_inweight', 'model_ema.diffusion_modelinput_blocks11proj_inbias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0ffnet0projbias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0ffnet2weight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0ffnet2bias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn2to_qweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn2to_kweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn2to_vweight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn2to_out0weight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0attn2to_out0bias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm1weight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm1bias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm2weight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm2bias', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm3weight', 'model_ema.diffusion_modelinput_blocks11transformer_blocks0norm3bias', 'model_ema.diffusion_modelinput_blocks11proj_outweight', 'model_ema.diffusion_modelinput_blocks11proj_outbias', 'model_ema.diffusion_modelinput_blocks20in_layers0weight', 'model_ema.diffusion_modelinput_blocks20in_layers0bias', 'model_ema.diffusion_modelinput_blocks20in_layers2weight', 'model_ema.diffusion_modelinput_blocks20in_layers2bias', 'model_ema.diffusion_modelinput_blocks20emb_layers1weight', 'model_ema.diffusion_modelinput_blocks20emb_layers1bias', 'model_ema.diffusion_modelinput_blocks20out_layers0weight', 'model_ema.diffusion_modelinput_blocks20out_layers0bias', 'model_ema.diffusion_modelinput_blocks20out_layers3weight', 'model_ema.diffusion_modelinput_blocks20out_layers3bias', 'model_ema.diffusion_modelinput_blocks21normweight', 'model_ema.diffusion_modelinput_blocks21normbias', 'model_ema.diffusion_modelinput_blocks21proj_inweight', 'model_ema.diffusion_modelinput_blocks21proj_inbias', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn1to_qweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn1to_kweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn1to_vweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn1to_out0weight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0attn1to_out0bias', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0ffnet0projweight', 'model_ema.diffusion_modelinput_blocks21transformer_blocks0ff
"Instantiating config for: pytorch_lightning.loggers.TestTubeLogger with config:\n",
"{'name': 'testtube', 'save_dir': 'logs\\\\input2022-11-09T00-06-09_test'}\n",
"Monitoring val/loss_simple_ema as checkpoint metric.\n",
"Merged modelckpt-cfg: \n",
"{'target': 'pytorch_lightning.callbacks.ModelCheckpoint', 'params': {'dirpath': 'logs\\\\input2022-11-09T00-06-09_test\\\\checkpoints', 'filename': '{epoch:02d}-{step:05d}', 'verbose': True, 'monitor': 'val/loss_simple_ema', 'every_n_epochs': 1, 'save_top_k': 2, 'save_last': False}}\n",
"Instantiating config for: main.SetupCallback with config:\n",
"{'resume': '', 'now': 'input2022-11-09T00-06-09', 'logdir': 'logs\\\\input2022-11-09T00-06-09_test', 'ckptdir': 'logs\\\\input2022-11-09T00-06-09_test\\\\checkpoints', 'cfgdir': 'logs\\\\input2022-11-09T00-06-09_test\\\\configs', 'config': {'model': {'base_learning_rate': 1e-06, 'target': 'ldm.models.diffusion.ddpm.LatentDiffusion', 'params': {'linear_start': 0.00085, 'linear_end': 0.012, 'num_timesteps_cond': 1, 'log_every_t': 200, 'timesteps': 1000, 'first_stage_key': 'image', 'cond_stage_key': 'caption', 'image_size': 64, 'channels': 4, 'cond_stage_trainable': True, 'conditioning_key': 'crossattn', 'monitor': 'val/loss_simple_ema', 'scale_factor': 0.18215, 'use_ema': False, 'unfreeze_model': True, 'model_lr': 1e-06, 'unet_config': {'target': 'ldm.modules.diffusionmodules.openaimodel.UNetModel', 'params': {'image_size': 32, 'in_channels': 4, 'out_channels': 4, 'model_channels': 320, 'attention_resolutions': [4, 2, 1], 'num_res_blocks': 2, 'channel_mult': [1, 2, 4, 4], 'num_heads': 8, 'use_spatial_transformer': True, 'transformer_depth': 1, 'context_dim': 768, 'use_checkpoint': True, 'legacy': False}}, 'first_stage_config': {'target': 'ldm.models.autoencoder.AutoencoderKL', 'params': {'embed_dim': 4, 'monitor': 'val/rec_loss', 'ddconfig': {'double_z': True, 'z_channels': 4, 'resolution': 512, 'in_channels': 3, 'out_ch': 3, 'ch': 128, 'ch_mult': [1, 2, 4, 4], 'num_res_blocks': 2, 'attn_resolutions': [], 'dropout': 0.0}, 'lossconfig': {'target': 'torch.nn.Identity'}}}, 'cond_stage_config': {'target': 'ldm.modules.encoders.modules.FrozenCLIPEmbedder'}, 'ckpt_path': 'v1-5-pruned.ckpt'}}, 'data': {'target': 'main.DataModuleFromConfig', 'params': {'batch_size': 4, 'num_workers': 1, 'wrap': 'falsegit', 'train': {'target': 'ldm.data.every_dream.EveryDreamBatch', 'params': {'repeats': 50, 'flip_p': 0, 'debug_level': 1}}, 'validation': {'target': 'ldm.data.ed_validate.EDValidateBatch', 'params': {'repeats': 10}}, 'test': {'target': 'ldm.data.ed_validate.EDValidateBatch', 'params': {'repeats': 1}}}}}, 'lightning_config': {'modelcheckpoint': {'params': {'every_n_epochs': 1, 'save_top_k': 2, 'save_last': False, 'filename': '{epoch:02d}-{step:05d}'}}, 'callbacks': {'image_logger': {'target': 'main.ImageLogger', 'params': {'batch_frequency': 150, 'max_images': 16, 'increase_log_steps': False}}}, 'trainer': {'benchmark': True, 'max_epochs': 4, 'max_steps': 99000, 'check_val_every_n_epoch': 1, 'gpus': '0,'}}}\n",
"Instantiating config for: main.ImageLogger with config:\n",
"{'batch_frequency': 150, 'max_images': 16, 'clamp': True, 'increase_log_steps': False}\n",
"Instantiating config for: main.LearningRateMonitor with config:\n",
"{'logging_interval': 'step'}\n",
"Instantiating config for: main.CUDACallback with config:\n",
"{}\n",
"Instantiating config for: pytorch_lightning.callbacks.ModelCheckpoint with config:\n",
"{'dirpath': 'logs\\\\input2022-11-09T00-06-09_test\\\\checkpoints', 'filename': '{epoch:02d}-{step:05d}', 'verbose': True, 'monitor': 'val/loss_simple_ema', 'every_n_epochs': 1, 'save_top_k': 2, 'save_last': False}\n",
"Instantiating config for: main.DataModuleFromConfig with config:\n",
"{'batch_size': 4, 'num_workers': 1, 'wrap': 'falsegit', 'train': {'target': 'ldm.data.every_dream.EveryDreamBatch', 'params': {'repeats': 50, 'flip_p': 0, 'debug_level': 1, 'data_root': 'input'}}, 'validation': {'target': 'ldm.data.ed_validate.EDValidateBatch', 'params': {'repeats': 10, 'data_root': 'input'}}, 'test': {'target': 'ldm.data.ed_validate.EDValidateBatch', 'params': {'repeats': 1, 'data_root': 'input'}}}\n",
" ****** validation: {'target': 'ldm.data.ed_validate.EDValidateBatch', 'params': {'repeats': 10, 'data_root': 'input', 'batch_size': 4, 'set': 'val'}}\n",
"Instantiating config for: ldm.data.every_dream.EveryDreamBatch with config:\n",
"{'repeats': 50, 'flip_p': 0, 'debug_level': 1, 'data_root': 'input', 'batch_size': 4, 'set': 'train'}\n",
" * Creating new dataloader singleton\n",
" Preloading images...\n",
" ** Number of buckets: 0\n",
"Training complete. max_steps or max_epochs reached, or we blew up.\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Global seed set to 23\n",
"Some weights of the model checkpoint at openai/clip-vit-large-patch14 were not used when initializing CLIPTextModel: ['vision_model.encoder.layers.11.mlp.fc2.bias', 'vision_model.encoder.layers.0.self_attn.v_proj.weight', 'visual_projection.weight', 'vision_model.encoder.layers.14.mlp.fc1.bias', 'vision_model.encoder.layers.2.self_attn.out_proj.bias', 'vision_model.encoder.layers.15.self_attn.q_proj.weight', 'vision_model.encoder.layers.13.layer_norm2.weight', 'vision_model.encoder.layers.18.mlp.fc1.weight', 'vision_model.encoder.layers.15.mlp.fc2.weight', 'vision_model.encoder.layers.20.self_attn.k_proj.weight', 'vision_model.encoder.layers.22.self_attn.k_proj.weight', 'vision_model.encoder.layers.2.layer_norm2.bias', 'vision_model.encoder.layers.14.layer_norm2.bias', 'vision_model.encoder.layers.6.self_attn.out_proj.weight', 'vision_model.encoder.layers.14.self_attn.out_proj.bias', 'vision_model.encoder.layers.17.mlp.fc2.bias', 'vision_model.encoder.layers.17.layer_norm2.bias', 'vision_model.encoder.layers.23.layer_norm1.weight', 'vision_model.encoder.layers.3.mlp.fc1.bias', 'vision_model.encoder.layers.14.self_attn.v_proj.weight', 'vision_model.pre_layrnorm.bias', 'vision_model.encoder.layers.0.mlp.fc2.bias', 'vision_model.encoder.layers.3.self_attn.q_proj.weight', 'vision_model.encoder.layers.9.mlp.fc2.weight', 'vision_model.encoder.layers.9.mlp.fc2.bias', 'vision_model.encoder.layers.20.self_attn.out_proj.bias', 'vision_model.encoder.layers.21.self_attn.v_proj.weight', 'vision_model.encoder.layers.14.mlp.fc2.bias', 'vision_model.encoder.layers.10.layer_norm2.weight', 'vision_model.encoder.layers.23.layer_norm1.bias', 'vision_model.encoder.layers.11.layer_norm2.weight', 'vision_model.encoder.layers.15.self_attn.k_proj.bias', 'vision_model.encoder.layers.10.layer_norm2.bias', 'vision_model.encoder.layers.10.self_attn.out_proj.weight', 'vision_model.encoder.layers.2.mlp.fc2.weight', 'vision_model.encoder.layers.8.layer_norm1.bias', 'vision_model.encoder.layers.12.mlp.fc2.bias', 'vision_model.encoder.layers.10.mlp.fc1.weight', 'vision_model.encoder.layers.2.layer_norm1.bias', 'vision_model.encoder.layers.9.self_attn.out_proj.weight', 'vision_model.encoder.layers.1.self_attn.k_proj.weight', 'vision_model.encoder.layers.3.mlp.fc1.weight', 'vision_model.encoder.layers.5.layer_norm1.bias', 'vision_model.encoder.layers.5.mlp.fc2.bias', 'vision_model.encoder.layers.22.self_attn.out_proj.weight', 'vision_model.encoder.layers.1.self_attn.out_proj.bias', 'vision_model.encoder.layers.11.self_attn.q_proj.bias', 'vision_model.encoder.layers.23.self_attn.q_proj.bias', 'vision_model.encoder.layers.2.self_attn.q_proj.bias', 'vision_model.encoder.layers.3.self_attn.out_proj.weight', 'vision_model.encoder.layers.0.layer_norm2.bias', 'vision_model.encoder.layers.10.self_attn.v_proj.weight', 'vision_model.encoder.layers.22.self_attn.v_proj.weight', 'vision_model.encoder.layers.11.mlp.fc1.weight', 'text_projection.weight', 'vision_model.encoder.layers.1.layer_norm1.bias', 'vision_model.encoder.layers.7.self_attn.q_proj.bias', 'vision_model.encoder.layers.16.layer_norm2.bias', 'vision_model.encoder.layers.8.self_attn.v_proj.bias', 'vision_model.encoder.layers.7.self_attn.k_proj.bias', 'vision_model.encoder.layers.13.self_attn.q_proj.weight', 'vision_model.encoder.layers.3.layer_norm2.bias', 'vision_model.encoder.layers.9.layer_norm1.weight', 'vision_model.encoder.layers.5.layer_norm2.bias', 'vision_model.encoder.layers.18.layer_norm1.bias', 'vision_model.encoder.layers.21.mlp.fc1.weight', 'vision_model.encoder.layers.14.layer_norm1.weight', 'vision_model.encoder.layers.4.self_attn.out_proj.weight', 'vision_model.encoder.layers.23.self_attn.q_proj.weight', 'vision_model.encoder.layers.16.self_attn.k_proj.weight', 'vision_model.encoder.layers.19.self_attn.k_proj.weight', 'vision_model.encoder.layers.4.mlp.fc2.weight', 'vision_model.encoder.layers.5.self_attn.q_proj.weight', 'vision_model.encoder.layers.11.mlp.fc2.weight', 'vision_model.encoder.layers.5.self_attn.q_proj.bias', 'vision_model.encoder.layers.1.self_attn.v_proj.weight', 'v
"- This IS expected if you are initializing CLIPTextModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
"- This IS NOT expected if you are initializing CLIPTextModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
"c:\\Users\\Freon\\.conda\\envs\\everydream\\lib\\site-packages\\pytorch_lightning\\loggers\\test_tube.py:104: LightningDeprecationWarning: The TestTubeLogger is deprecated since v1.5 and will be removed in v1.7. We recommend switching to the `pytorch_lightning.loggers.TensorBoardLogger` as an alternative.\n",
" rank_zero_deprecation(\n",
"GPU available: True, used: True\n",
"TPU available: False, using: 0 TPU cores\n",
"IPU available: False, using: 0 IPUs\n",
"Traceback (most recent call last):\n",
" File \"main.py\", line 692, in <module>\n",
" data.prepare_data()\n",
" File \"c:\\Users\\Freon\\.conda\\envs\\everydream\\lib\\site-packages\\pytorch_lightning\\core\\datamodule.py\", line 474, in wrapped_fn\n",
" fn(*args, **kwargs)\n",
" File \"r:\\everydream-trainer\\main.py\", line 244, in prepare_data\n",
" instantiate_from_config(data_cfg)\n",
" File \"r:\\everydream-trainer\\ldm\\util.py\", line 89, in instantiate_from_config\n",
" return get_obj_from_str(config[\"target\"])(**config.get(\"params\", dict()), **kwargs)\n",
" File \"r:\\everydream-trainer\\ldm\\data\\every_dream.py\", line 22, in __init__\n",
" dls.shared_dataloader = dlma(data_root=data_root, debug_level=debug_level, batch_size=self.batch_size, flip_p=flip_p)\n",
" File \"r:\\everydream-trainer\\ldm\\data\\data_loader.py\", line 28, in __init__\n",
" print(f\" * DLMA Example {self.image_caption_pairs[0]} images\")\n",
"IndexError: list index out of range\n"
]
},
{
"ename": "NameError",
"evalue": "name 'Audio' is not defined",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32mr:\\everydream-trainer\\Train-Runpod.ipynb Cell 16\u001b[0m in \u001b[0;36m<cell line: 3>\u001b[1;34m()\u001b[0m\n\u001b[0;32m <a href='vscode-notebook-cell:/r%3A/everydream-trainer/Train-Runpod.ipynb#X52sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m \u001b[39m# run the trainer, wait until it finishes then SCROLL DOWN to the next cell\u001b[39;00m\n\u001b[0;32m <a href='vscode-notebook-cell:/r%3A/everydream-trainer/Train-Runpod.ipynb#X52sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m get_ipython()\u001b[39m.\u001b[39msystem(\u001b[39m'\u001b[39m\u001b[39mpython main.py --base configs/stable-diffusion/v1-finetune_runpod.yaml -t --actual_resume \u001b[39m\u001b[39m\"\u001b[39m\u001b[39mv1-5-pruned.ckpt\u001b[39m\u001b[39m\"\u001b[39m\u001b[39m -n test --data_root input\u001b[39m\u001b[39m'\u001b[39m)\n\u001b[1;32m----> <a href='vscode-notebook-cell:/r%3A/everydream-trainer/Train-Runpod.ipynb#X52sZmlsZQ%3D%3D?line=2'>3</a>\u001b[0m Audio(sound_file, autoplay\u001b[39m=\u001b[39mI_LIKE_FUN_STUFF)\n\u001b[0;32m <a href='vscode-notebook-cell:/r%3A/everydream-trainer/Train-Runpod.ipynb#X52sZmlsZQ%3D%3D?line=3'>4</a>\u001b[0m \u001b[39mprint\u001b[39m(finish_msg)\n",
"\u001b[1;31mNameError\u001b[0m: name 'Audio' is not defined"
]
}
],
2022-11-08 21:00:54 -07:00
"source": [
2022-11-08 23:34:47 -07:00
"# run the trainer, wait until it finishes then SCROLL DOWN to the next cell\n",
"!python main.py --base {my_yaml} -t --actual_resume \"v1-5-pruned.ckpt\" -n test --data_root input\n",
"Audio(sound_file, autoplay=I_LIKE_FUN_STUFF)\n",
"print(finish_msg)"
2022-11-08 21:00:54 -07:00
]
},
{
"cell_type": "markdown",
"id": "e8c93085",
"metadata": {},
"source": [
"## Prune your checkpoints\n",
"This will create 2GB pruned files for all your checkpoints. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5e70ae7e",
"metadata": {},
"outputs": [],
"source": [
"# prune the ckpts\n",
"!python auto_prune_all.py --delete"
]
},
{
"cell_type": "markdown",
"id": "51456afe",
"metadata": {},
"source": [
"## Download your checkpoints\n",
"\n",
"Use the file explorer on the left, go into the \"every-dream-trainer\" folder.\n",
"\n",
"Look for all the ckpt files that say \"-pruned\" on the end. Download them and you're done! \n",
"\n",
"[EveryDream Discord](https://discord.gg/uheqxU6sXN)"
]
}
],
"metadata": {
"kernelspec": {
2022-11-08 23:34:47 -07:00
"display_name": "Python 3 (ipykernel)",
2022-11-08 21:00:54 -07:00
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"vscode": {
"interpreter": {
"hash": "2e677f113ff5b533036843965d6e18980b635d0aedc1c5cebd058006c5afc92a"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
