Mirrors
/
stable-diffusion-paperspace
mirror of https://github.com/Engineer-of-Stuff/stable-diffusion-paperspace.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

storage options 

Former-commit-id: 796b4a9b0c
This commit is contained in:
Cyberes 2022-09-10 14:24:12 -06:00 committed by GitHub
parent 4aee2a6943
commit 5bd7aeeac0
1 changed files with 107 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

144
StableDiffusionUI_Voldemort_paperspace.ipynb
View File

@ -22,15 +22,31 @@
"You must reinstall everything each time you restart the machine. If already downloaded dependencies will be auto-updated."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Where to store the models**\n",
"\n",
"`/storage/models/` is for the free tier.\n",
"\n",
"`/notebooks/models/` is for the pro tier.\n",
"\n",
"_You must run the block below or else the variable won't be accessable in the notebook._"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OL82Y4rBjZIV"
},
"metadata": {},
"outputs": [],
"source": [
"!nvidia-smi"
"model_storage_dir = '/storage/models' # no trailing slash\n",
"\n",
"# ===============================================================\n",
"# Save the variable to Jupiter's temp storage to access it when the kernel restarts.\n",
"# To reset your storage directory, rerun this cell.\n",
"%store model_storage_dir"
]
},
{
@ -52,7 +68,7 @@
"outputs": [],
"source": [
"import os.path\n",
"\n",
"%store -r model_storage_dir\n",
"if not os.path.exists('/notebooks/stable-diffusion-webui'):\n",
" %cd /notebooks/\n",
" !git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui\n",
@ -83,7 +99,7 @@
"outputs": [],
"source": [
"%cd /notebooks/stable-diffusion-webui\n",
"\n",
"%store -r model_storage_dir\n",
"!pip install -r requirements.txt\n",
"\n",
"if not os.path.exists('/notebooks/stable-diffusion-webui/repositories'): # download repositories if they don't exist\n",
@ -104,7 +120,7 @@
"else:\n",
" print('GFPGANv1.3.pth already downloaded')\n",
" \n",
"!mkdir -p /notebooks/models/\n",
"!mkdir -p $model_storage_dir\n",
"\n",
"# Get some storage back\n",
"!pip cache purge\n",
@ -115,6 +131,7 @@
"cell_type": "markdown",
"metadata": {
"id": "F0EINk5M0s-w",
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
@ -131,7 +148,9 @@
"\n",
"**Filesize and Storage Disclaimer**\n",
"\n",
"Paperspace free tier has only 5GB of storage space. The standard model is 4GB and waifu is 7.2GB. If you run out of storage space, you have two options:\n",
"Paperspace free tier has only 5GB of storage space. The standard model is 4GB and waifu is 3.5GB/7.2GB. To get around this limited storage capacity we will download the models to `/storage/`, a non-persistent directory. Each time you restart the machine you will have to redownload the models.\n",
"\n",
"If you don't want to have to redownload the models every time you restart your machine you have three options:\n",
"1. Add a payment method to your account. Storage overages are billed at \\$0.29/GB and billing occurs monthly and runs at midnight on the first of each month. With a payment method on file, Paperspace will let you use more storage and if you time it right you shouldn't actually be charged for it.\n",
"2. Upgrade to a Pro account. They'll give you 15GB and you'll get longer runtimes and more powerful free GPUs.\n",
"3. Create an account using my referral code `KQLRH37` or [signup link](https://console.paperspace.com/signup?R=KQLRH37). You'll get \\$10 credit that you should be able to put towards the storage overage charges. If you've already created your account you can still redeem the code at the bottom of the Billing page."
@ -160,9 +179,10 @@
"metadata": {},
"outputs": [],
"source": [
"%store -r model_storage_dir\n",
"!apt update\n",
"!apt install -y aria2\n",
"%cd /notebooks/models\n",
"%cd $model_storage_dir\n",
"!aria2c --seed-time=0 --max-overall-upload-limit=1K \"magnet:?xt=urn:btih:3A4A612D75ED088EA542ACAC52F9F45987488D1C&tr=udp://tracker.opentrackr.org:1337\""
]
},
@ -176,7 +196,7 @@
"\n",
"[https://drive.google.com/file/d/1wHFgl0ivCmIZv88hVZXkb8oy9qCuaBGA/view](https://drive.google.com/file/d/1wHFgl0ivCmIZv88hVZXkb8oy9qCuaBGA/view)\n",
"\n",
"Download it to your computer then upload it to `/notebooks/models` (make sure it's named `sd-v1-4.ckpt`).\n",
"Download it to your computer then upload it to your model storage directory (make sure it's named `sd-v1-4.ckpt`).\n",
"\n",
"HuggingFace is much faster and reliable but you need to get access to the repo and provide your user token."
]
@ -189,17 +209,17 @@
},
"outputs": [],
"source": [
"user_token = \"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\"\n",
"user_token = \"<enter your user token here>\"\n",
"\n",
"# ===============================================================================================\n",
"\n",
"user_header = f\"'Authorization: Bearer {user_token}'\"\n",
"!wget --header={user_header} https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/resolve/main/sd-v1-4.ckpt -O /notebooks/models/sd-v1-4.ckpt"
"%store -r model_storage_dir\n",
"!wget --header=\"'Authorization: Bearer {user_token}'\" https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/resolve/main/sd-v1-4.ckpt -O $model_storage_dir/sd-v1-4.ckpt"
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
@ -221,7 +241,7 @@
"source": [
"!apt update\n",
"!apt install -y aria2\n",
"%cd /notebooks/models\n",
"%cd $model_storage_dir\n",
"!aria2c --seed-time=0 --max-overall-upload-limit=1K \"magnet:?xt=urn:btih:F45CECF4E9DE86DA83A78DD2CCCD7F27D5557A52&tr=udp://nyquist.localghost.org:6969\""
]
},
@ -242,14 +262,15 @@
"metadata": {},
"outputs": [],
"source": [
"!wget https://thisanimedoesnotexist.ai/downloads/wd-v1-2-full-ema.ckpt -O /notebooks/models/wd-v1-2-full-ema.ckpt"
"%store -r model_storage_dir\n",
"!wget https://thisanimedoesnotexist.ai/downloads/wd-v1-2-full-ema.ckpt -O $model_storage_dir/wd-v1-2-full-ema.ckpt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Half-size Model**\n",
"**Half-Size Model**\n",
"\n",
"If you're on free tier try this first, it's 3.5GB."
]
@ -260,9 +281,10 @@
"metadata": {},
"outputs": [],
"source": [
"%store -r model_storage_dir\n",
"!apt update\n",
"!apt install -y aria2\n",
"%cd /notebooks/models\n",
"%cd $model_storage_dir\n",
"!aria2c --seed-time=0 --max-overall-upload-limit=1K \"magnet:?xt=urn:btih:153590FD7E93EE11D8DB951451056C362E3A9150&dn=wd-v1-2-full-ema-pruned.ckpt&tr=udp://glotorrents.pw:6969/announce&tr=udp://tracker.opentrackr.org:1337/announce&tr=udp://torrent.gresille.org:80/announce&tr=udp://tracker.openbittorrent.com:80&tr=udp://tracker.coppersurfer.tk:6969&tr=udp://tracker.leechers-paradise.org:6969&tr=udp://p4p.arenabg.ch:1337&tr=udp://tracker.internetwarriors.net:1337&tr=wss://tracker.openwebtorrent.com\""
]
},
@ -291,19 +313,22 @@
"metadata": {},
"outputs": [],
"source": [
"mounted_dataset_path = '/datasets/<name>'\n",
"mounted_dataset_path = '/datasets/models'\n",
"\n",
"# ============================================================\n",
"\n",
"%store -r model_storage_dir\n",
"import os\n",
"for file in os.listdir(mounted_dataset_path):\n",
" if file.endswith(\"ckpt\"):\n",
" !ln -s $mounted_dataset_path/$file /notebooks/models/$file\n",
" !ls -la --block-size=GB /notebooks/models/$file"
" !ln -s $mounted_dataset_path/$file $model_storage_dir$file\n",
" !ls -la --block-size=GB $model_storage_dir/$file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"metadata": {
"tags": []
},
"source": [
"### Clean up and restart the kernel"
]
@ -314,7 +339,8 @@
"metadata": {},
"outputs": [],
"source": [
"!rm /notebooks/models/*.aria2\n",
"%store -r model_storage_dir\n",
"!rm $model_storage_dir/*.aria2\n",
"!apt autoremove --purge -y aria2\n",
"import os\n",
"os.kill(os.getpid(), 9)"
@ -340,26 +366,44 @@
"If you have any issues, restart the kernel."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Choose your Model**\n",
"|Name||\n",
"|-|-|\n",
"|`sd-v1-4.ckpt`|Standard diffusion v1.4|\n",
"|`wd-v1-2-full-ema.ckpt`|waifu-diffusion v1.2|\n",
"|`wd-v1-2-full-ema-pruned.ckpt`|waifu-diffusion v1.2 half-size|\n",
"\n",
"Enter the name of the model you want to use in `model_name` below.\n",
"\n",
"To change models you must restart the kernel for it to take effect."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Start the WebUI**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "R-xAdMA5wxXd",
"scrolled": true,
"tags": []
},
"outputs": [],
"source": [
"# Possible models:\n",
"# sd-v1-4.ckpt (standard diffusion v1.4)\n",
"# wd-v1-2-full-ema.ckpt (waifu-diffusion v1.2)\n",
"# wd-v1-2-full-ema-pruned.ckpt (waifu-diffusion v1.2 half-size)\n",
"\n",
"model_name = ''\n",
"\n",
"# ========================================================================================================\n",
"\n",
"model_file = f'/notebooks/models/{model_name}'\n",
"# ===============================================================================================================\n",
"# Don't change anything in here\n",
"%store -r model_storage_dir\n",
"model_file = f'{model_storage_dir}/{model_name}'\n",
"\n",
"import sys\n",
"import os\n",
@ -369,10 +413,13 @@
" sys.exit(1)\n",
"\n",
"if model_name == '':\n",
" print('You forgot to fill out the variable model_file above.')\n",
" print('You forgot to fill out the variable model_name above.')\n",
" sys.exit(1)\n",
"\n",
"!python /notebooks/stable-diffusion-webui/webui.py --share --ckpt $model_file --show-negative-prompt --no-progressbar-hiding"
"# ===============================================================================================================\n",
" \n",
"# You can modify the args if nessesary\n",
"!python /notebooks/stable-diffusion-webui/webui.py --share --ckpt $model_file --show-negative-prompt # --no-progressbar-hiding"
]
},
{
@ -389,13 +436,16 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"This block will compress the outputs with 7zip max compression then delete the originals. It expects you to have `log/` and `outputs/` in `/notebooks/`."
"This block will rename and compress the outputs with 7zip max compression. It expects you to have `log/` and `outputs/` in `/notebooks/`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"metadata": {
"scrolled": true,
"tags": []
},
"outputs": [],
"source": [
"!apt update\n",
@ -407,6 +457,7 @@
"!mkdir $datetime_str\n",
"!mv log $datetime_str\n",
"!mv outputs $datetime_str\n",
"print('Compressing, please wait...')\n",
"os.system(f'7z a -t7z -m0=lzma2 -mx=9 -mfb=64 -md=32m -ms=on {datetime_str}.7z {datetime_str}/')"
]
},
@ -430,6 +481,25 @@
"!rm -rf /notebooks/$datetime_str/\n",
"!echo Deleted /notebooks/$datetime_str/"
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"# Tools"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!nvidia-smi"
]
}
],
"metadata": {