clarify docker notebook instructions for HF upload

This commit is contained in:
Victor Hall 2023-06-07 16:59:44 -04:00
parent 1155a28867
commit e255aec653
1 changed files with 40 additions and 7 deletions

View File

@ -1,6 +1,7 @@
{ {
"cells": [ "cells": [
{ {
"attachments": {},
"cell_type": "markdown", "cell_type": "markdown",
"id": "2c831b5b-3025-4177-bef5-25aaec89573a", "id": "2c831b5b-3025-4177-bef5-25aaec89573a",
"metadata": {}, "metadata": {},
@ -33,6 +34,7 @@
] ]
}, },
{ {
"attachments": {},
"cell_type": "markdown", "cell_type": "markdown",
"id": "ffff47f7", "id": "ffff47f7",
"metadata": {}, "metadata": {},
@ -55,6 +57,18 @@
] ]
}, },
{ {
"cell_type": "code",
"execution_count": null,
"id": "3034c47a",
"metadata": {},
"outputs": [],
"source": [
"# Check your GPU\n",
"!nvidia-smi"
]
},
{
"attachments": {},
"cell_type": "markdown", "cell_type": "markdown",
"id": "7c73894e-3b5e-4268-9f83-ed89bd4569f2", "id": "7c73894e-3b5e-4268-9f83-ed89bd4569f2",
"metadata": { "metadata": {
@ -72,6 +86,7 @@
] ]
}, },
{ {
"attachments": {},
"cell_type": "markdown", "cell_type": "markdown",
"id": "f15fcd56-0418-4be1-a5c3-38aa679b1aaf", "id": "f15fcd56-0418-4be1-a5c3-38aa679b1aaf",
"metadata": {}, "metadata": {},
@ -114,6 +129,7 @@
] ]
}, },
{ {
"attachments": {},
"cell_type": "markdown", "cell_type": "markdown",
"id": "ed464c6b-1a8d-48e4-9787-265e8acaac43", "id": "ed464c6b-1a8d-48e4-9787-265e8acaac43",
"metadata": {}, "metadata": {},
@ -134,21 +150,28 @@
] ]
}, },
{ {
"attachments": {},
"cell_type": "markdown", "cell_type": "markdown",
"id": "3c506e79-bf03-4e34-bf06-9371963d4d7d", "id": "3c506e79-bf03-4e34-bf06-9371963d4d7d",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# HuggingFace download (Optional)\n", "# HuggingFace Upload/Download (Optional)\n",
"Run the cell below and paste your token into the prompt. You can get your token from your [huggingface account page](https://huggingface.co/settings/tokens).\n", "Run the cell below and paste your token into the prompt. You can get your token from your [huggingface account page](https://huggingface.co/settings/tokens).\n",
"\n", "\n",
"The token will not show on the screen, just press enter after you paste it." "The token will not show on the screen, just press enter after you paste it.\n",
"\n",
"### Make sure to run this login cell for any Huggingface uploads or private downloads."
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": null, "execution_count": null,
"id": "138b7776-8783-4e1d-920d-cf358809b802", "id": "138b7776-8783-4e1d-920d-cf358809b802",
"metadata": {}, "metadata": {
"tags": [
"hflogin"
]
},
"outputs": [], "outputs": [],
"source": [ "source": [
"from huggingface_hub import notebook_login, hf_hub_download\n", "from huggingface_hub import notebook_login, hf_hub_download\n",
@ -157,11 +180,13 @@
] ]
}, },
{ {
"attachments": {},
"cell_type": "markdown", "cell_type": "markdown",
"id": "b252a308-49cf-443f-abbb-d08b471411fb", "id": "b252a308-49cf-443f-abbb-d08b471411fb",
"metadata": {}, "metadata": {},
"source": [ "source": [
"Then run the following cell to download the base checkpoint (may take a minute)." "### HuggingFace Download a checkpoint\n",
"Make sure you are logged in using the above login cell first. "
] ]
}, },
{ {
@ -196,16 +221,21 @@
] ]
}, },
{ {
"attachments": {},
"cell_type": "markdown", "cell_type": "markdown",
"id": "f24eee3d-f5df-45f3-9acc-ee0206cfe6b1", "id": "f24eee3d-f5df-45f3-9acc-ee0206cfe6b1",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# HuggingFace upload\n", "### HuggingFace upload all your checkpoints\n",
"Make sure you are **logged in** using the above login cell first. \n",
"\n",
"Use the cell below to upload one or more checkpoints to your personal HuggingFace account, if you want, instead of manually downloading. You should already be authorized to Huggingface by token if you used the download/token cells above.\n", "Use the cell below to upload one or more checkpoints to your personal HuggingFace account, if you want, instead of manually downloading. You should already be authorized to Huggingface by token if you used the download/token cells above.\n",
"\n", "\n",
"* You can get your account name from your [HuggingFace account page](https://huggingface.co/settings/account). Look for your \"username\" field and paste it below.\n", "* You can get your account name from your [HuggingFace account page](https://huggingface.co/settings/account). Look for your \"username\" field and paste it below.\n",
"\n", "\n",
"* You only need to setup a repository one time. You can create it here: [Create New HF Model](https://huggingface.co/new) Make sure you write down the repo name you make for future use. You can reuse it later." "* You only need to setup a repository one time. You can create it here: [Create New HF Model](https://huggingface.co/new) Make sure you write down the repo name you make for future use. You can reuse it later.\n",
"\n",
"When you run this, **CLICK** to select which .ckpt files are marked for upload. This allows you to select which ones to upload. If you don't click of the ckpts, nothing will happen."
] ]
}, },
{ {
@ -233,6 +263,8 @@
"def upload_ckpts(_):\n", "def upload_ckpts(_):\n",
" repo_id=f\"{hfuser.value}/{hfrepo.value}\"\n", " repo_id=f\"{hfuser.value}/{hfrepo.value}\"\n",
" with out:\n", " with out:\n",
" if ckpt_picker is None or len(ckpt_picker) < 1:\n",
" print(\"Nothing selected for upload, make sure to click one of the ckpt files in the list, or, you have no ckpt files in the current directory.\")\n",
" for ckpt in ckpt_picker.value:\n", " for ckpt in ckpt_picker.value:\n",
" print(f\"Uploading to HF: huggingface.co/{repo_id}/{ckpt}\")\n", " print(f\"Uploading to HF: huggingface.co/{repo_id}/{ckpt}\")\n",
" response = api.upload_file(\n", " response = api.upload_file(\n",
@ -253,6 +285,7 @@
] ]
}, },
{ {
"attachments": {},
"cell_type": "markdown", "cell_type": "markdown",
"id": "c1a00d16-9b84-492f-8e6a-defe71e82b43", "id": "c1a00d16-9b84-492f-8e6a-defe71e82b43",
"metadata": {}, "metadata": {},
@ -387,7 +420,7 @@
"name": "python", "name": "python",
"nbconvert_exporter": "python", "nbconvert_exporter": "python",
"pygments_lexer": "ipython3", "pygments_lexer": "ipython3",
"version": "3.10.6" "version": "3.10.10"
}, },
"vscode": { "vscode": {
"interpreter": { "interpreter": {