Kaggle是一个机器学习和数据科学社区,可以利用它提供的环境来搭建StableDiffusion(每周30个小时)
注册Kaggle与启动StableDiffusion
-
打开Kaggle官网(www.kaggle.com/)
-
注册一个Kaggle账号
-
注册好Kaggle账号后,打开链接 www.kaggle.com/code/xinsia…
-
进入你的代码(上一步点击按钮后会跳转入你的代码页面),进行GPU配置,StableDiffusion需要GPU的算力,Kaggle的CPU算力很弱很弱
手机号验证成功后,就可以开启GPU了
-
配置好后就可以运行代码,生成一个StableDiffusion
代码执行过程中,如果看到红色的警告不用太担心,只要代码没停止执行,就继续等等,直到看到输出链接
解决人机身份验证的问题
在注册和验证手机号的过程中,会需要一些方法,来显示人机交互验证(谷歌提供的),你可以选择魔法或者用插件重定向访问
下面以微软浏览器Edge来介绍一个插件Header Editor来解决人机身份验证问题,不一定能成功,仅供参考
插件地址(microsoftedge.microsoft.com/addons/deta…
-
打开微软Edge浏览器的扩展管理
-
搜索安装插件Header Editor
-
配置Header Edit
用到的规则URL:github.azurezeng.com/static/HE-G…
-
配置好后,刷新对应网页,就可以看到人机身份验证了
更换StableDiffusion模型(Kaggle版)
默认的代码下载的模型只有两个,当你想用一下其他模型的时候,应该怎么做呢?
参考下面代码分析
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/57618 -d /kaggle/working/stable-diffusion-webui/models/Stable-diffusion -o Counterfeit-V3.safetensors
##########
#更换上面??这句代码里的链接和输出的文件名,贴会Kaggle里或者直接在Kaggle里修改,再执行代码即可下载模型
获取模型下载链接的方式,这里以 civitai.com为例
建议不要下载太多模型,只有20GB空间,所以我把不需要的ControlNet模型,都注释了下载代码,不让它下载,减少空间占用
如何下载 Lora 和 VEA
与更换模型的原理类似,除了修改下载链接和输出文件名,还要修改模型存放位置
# Lora
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/32988 -d /kaggle/working/stable-diffusion-webui/models/Lora -o blindbox_v1_mix.safetensors
# VAE
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/gemasai/vae-ft-mse-840000-ema-pruned/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt -d /kaggle/working/stable-diffusion-webui/models/VAE -o vae-ft-mse-840000-ema-pruned.ckpt
代码下载太复杂?来试试civitai 的SD插件
-
检查一下,代码里有没有安装这个civitai浏览插件,可以通过ctrl + F,打开查找框,输入civitai,快速查找相关代码
-
在运行的StableDiffusionWebUI中,切换到civitAi选项卡
-
刷新并读取模型
Kaggle与HuggingFace同步数据
-
注册HuggingFace账号
-
创建Dataset,来存储配置文件Config(我创建了两个一个公开的用来存储配置文件,一个私有的存储生成图的压缩文件)
-
生成Access Tokens,用于Kaggle 代码上传文件到HuggingFace
-
上传token到Kaggle里
复制token到txt文件
然后就可以看到token文件
将文件路径写到代码里,用于代码上传文件到HuggingFace使用token
#####################################################################
# 同步配置
# 使用huggingface保存和载入webui配置文件
huggingface_use = True
zip_output=True
huggingface_token_file = '/kaggle/input/hgtoken/huggingfacetoken.text'
huggiingface_repo_id = 'MSa/sd-output-images'
huggiingface_config_repo_id = 'MSa/sd-config'
# 将会同步的文件
config_files = [
'ui-config.json',
'config.json',
# 'styles.csv'
]
#功能函数
from pathlib import Path
from huggingface_hub import HfApi, login
# config 上传
def hugface_upload_config(huggingface_token_file, config_files, repo_id):
if Path(huggingface_token_file).exists():
with open(huggingface_token_file, encoding="utf-8") as nkfile:
hugToken = nkfile.readline()
if hugToken != '':
# 使用您的 Hugging Face 访问令牌登录
login(token=hugToken)
# 实例化 HfApi 类
api = HfApi()
print("HfApi 类已实例化")
%cd /kaggle/working/stable-diffusion-webui
# 使用 upload_file() 函数上传文件
print("开始上传文件...")
for config_file in config_files:
if Path(config_file).exists():
response = api.upload_file(
path_or_fileobj=config_file,
path_in_repo=config_file,
repo_id=repo_id,
repo_type="dataset"
)
print("文件上传完成")
print(f"响应: {response}")
else:
print(f'Error: File {config_file} does not exist')
else:
print(f'Error: File {huggingface_token_file} does not exist')
代码使用的注意事项
代码太复杂?来试试HuggingFace 的SD插件
确保代码里有下载HuggingFace 的SD插件
## Kaggle下载HuggingFace 的SD插件的代码
!git clone https://github.com/camenduru/stable-diffusion-webui-huggingface /kaggle/working/stable-diffusion-webui/extensions/stable-diffusion-webui-huggingface
插件安装成功后,StableDiffusion WebUI 里会有HuggingFace选项卡
获得Kaggle 输出文件夹路径的方法
# 其实Kaggle的文件夹路径都是相对固定的(如果你没做修改的话),可以直接用下面?这个路径
/kaggle/working/stable-diffusion-webui/outputs
在HuggingFace创建Dataset
复制HuggingFace的Dataset路径
生成HuggingFace的写入token
复制token,然后粘贴到插件的token输入框里
推送到HuggingFace的效果
如何导出导入代码
导出代码
如果担心Kaggle封禁项目代码,大家可以下载自己的代码备份
新建项目,导入代码
有了代码文件,可以新建项目,再导入代码,这样,你就获得一个新的StableDiffusion项目了(配置文件会是旧的,因为是保存在HuggingFace上,不过只要不同步,就是新的配置文件)
获得一个新的项目
完整的代码文件
「zh-stable-diffusion-webui-kaggle.ipynb」
个人经验与建议
-
更换模型需要注意的事项
在Kaggle编辑器中找到下载代码的位置
下载代码应该编写的位置
关于代码运行的注意事项
注意,不要下载太多模型,个人建议,每次运行Kaggle(开机关机算一次),最多两个模型就好了,太多模型占用空间过多,容易死机(其实问题也不大,只是死机的话,会来不及保存生成的图片)
附录:
完整代码:
# ? Please follow me for new updates https://twitter.com/camenduru
# ? Please join our discord server https://discord.gg/k5BwmmvJJU
!apt -y update -qq
!apt -y install -qq aria2
!pip install -q torch==1.13.1+cu117 torchvision==0.14.1+cu117 -f https://download.pytorch.org/whl/torch_stable.html
!pip install -q https://github.com/camenduru/stable-diffusion-webui-colab/releases/download/0.0.16/xformers-0.0.16+814314d.d20230118-cp38-cp38-linux_x86_64.whl
!pip install -q huggingface-hub==0.11.0 -U
#!pip install -q gradio==3.23.0 -U
!git clone -b v2.1 https://github.com/camenduru/stable-diffusion-webui
!wget https://raw.githubusercontent.com/camenduru/stable-diffusion-webui-scripts/main/run_n_times.py -O /kaggle/working/stable-diffusion-webui/scripts/run_n_times.py
!git clone https://github.com/AlUlkesh/stable-diffusion-webui-images-browser /kaggle/working/stable-diffusion-webui/extensions/stable-diffusion-webui-images-browser
# 切换图片浏览插件的版本
%cd /kaggle/working/stable-diffusion-webui/extensions/stable-diffusion-webui-images-browser
!git checkout 1d5c2e7
%cd ~
!git clone https://github.com/camenduru/stable-diffusion-webui-huggingface /kaggle/working/stable-diffusion-webui/extensions/stable-diffusion-webui-huggingface
!git clone -b v2.0 https://github.com/camenduru/sd-civitai-browser /kaggle/working/stable-diffusion-webui/extensions/sd-civitai-browser
!git clone https://github.com/Mikubill/sd-webui-controlnet /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet
!git clone https://github.com/camenduru/openpose-editor /kaggle/working/stable-diffusion-webui/extensions/openpose-editor
!git clone https://github.com/kohya-ss/sd-webui-additional-networks /kaggle/working/stable-diffusion-webui/extensions/sd-webui-additional-networks
!git clone https://github.com/DominikDoom/a1111-sd-webui-tagcomplete /kaggle/working/stable-diffusion-webui/extensions/a1111-sd-webui-tagcomplete
!git clone https://github.com/toriato/stable-diffusion-webui-wd14-tagger /kaggle/working/stable-diffusion-webui/extensions/stable-diffusion-webui-wd14-tagger
!git clone https://jihulab.com/hunter0725/stable-diffusion-webui-localization-zh_CN.git
%cd /kaggle/working/stable-diffusion-webui
# !wget https://temp-1257364845.cos.ap-nanjing.myqcloud.com/config.json
!wget https://jihulab.com/hunter0726/stable-diffusion-webui-localization-zh_CN/-/raw/582ca24da2f5ca7a155b1950471a61d38b1c9b62/localizations/zh_CN.json
!mv zh_CN.json localizations
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/control_canny-fp16.safetensors -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/models -o control_canny-fp16.safetensors
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/control_depth-fp16.safetensors -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/models -o control_depth-fp16.safetensors
#!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/control_hed-fp16.safetensors -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/models -o control_hed-fp16.safetensors
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/control_mlsd-fp16.safetensors -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/models -o control_mlsd-fp16.safetensors
#!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/control_normal-fp16.safetensors -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/models -o control_normal-fp16.safetensors
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/control_openpose-fp16.safetensors -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/models -o control_openpose-fp16.safetensors
#!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/control_scribble-fp16.safetensors -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/models -o control_scribble-fp16.safetensors
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/webui/ControlNet-modules-safetensors/resolve/main/control_seg-fp16.safetensors -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/models -o control_seg-fp16.safetensors
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/ckpt/ControlNet/resolve/main/hand_pose_model.pth -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose -o hand_pose_model.pth
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/ckpt/ControlNet/resolve/main/body_pose_model.pth -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose -o body_pose_model.pth
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/ckpt/ControlNet/resolve/main/dpt_hybrid-midas-501f0c75.pt -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/midas -o dpt_hybrid-midas-501f0c75.pt
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/ckpt/ControlNet/resolve/main/mlsd_large_512_fp32.pth -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/mlsd -o mlsd_large_512_fp32.pth
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/ckpt/ControlNet/resolve/main/mlsd_tiny_512_fp32.pth -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/mlsd -o mlsd_tiny_512_fp32.pth
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/ckpt/ControlNet/resolve/main/network-bsds500.pth -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/hed -o network-bsds500.pth
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/ckpt/ControlNet/resolve/main/upernet_global_small.pth -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/uniformer -o upernet_global_small.pth
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11f1e_sd15_tile.pth -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/models -o control_v11f1e_sd15_tile.pth
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11f1e_sd15_tile.yaml -d /kaggle/working/stable-diffusion-webui/extensions/sd-webui-controlnet/models
# CheckPoint
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/55203 -d /kaggle/working/stable-diffusion-webui/models/Stable-diffusion -o QteaMix-通用Q版模型.safetensors
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/57618 -d /kaggle/working/stable-diffusion-webui/models/Stable-diffusion -o Counterfeit-V3.safetensors
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/46846 -d /kaggle/working/stable-diffusion-webui/models/Stable-diffusion -o revAnimated_v122.safetensors
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/13069 -d /kaggle/working/stable-diffusion-webui/models/Stable-diffusion -o fantasyWorld_v1.safetensors
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/25571 -d /kaggle/working/stable-diffusion-webui/models/Stable-diffusion -o CheckpointYesmix_v16Original.safetensors
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/69832 -d /kaggle/working/stable-diffusion-webui/models/Stable-diffusion -o disneyPixarCartoon_v10.safetensors
# Lora
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/32988 -d /kaggle/working/stable-diffusion-webui/models/Lora -o blindbox_v1_mix.safetensors
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/63077 -d /kaggle/working/stable-diffusion-webui/models/Lora -o hanfu_tangstyle_v31.safetensors
# !aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/50709 -d /kaggle/working/stable-diffusion-webui/models/Lora -o fashigirl-v5.4-lora-64dim-naivae-5ep.safetensors
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/85137 -d /kaggle/working/stable-diffusion-webui/models/Lora -o ChineseDragon_v2.safetensors
# embeddings
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/datasets/gsdf/EasyNegative/resolve/main/EasyNegative.safetensors -d /kaggle/working/stable-diffusion-webui/embeddings -o EasyNegative.safetensors
# VAE
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/gemasai/vae-ft-mse-840000-ema-pruned/resolve/main/vae-ft-mse-840000-ema-pruned.ckpt -d /kaggle/working/stable-diffusion-webui/models/VAE -o vae-ft-mse-840000-ema-pruned.ckpt
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://civitai.com/api/download/models/14459?type=VAE -d /kaggle/working/stable-diffusion-webui/models/VAE
!sed -i -e 's/numpy==1.23.3/numpy==1.21.6/g' requirements_versions.txt
!sed -i -e 's/blendmodes==2022/blendmodes==2021/g' requirements_versions.txt
!sed -i -e 's/fastapi==0.90.1/fastapi==0.89.1/g' requirements_versions.txt
!sed -i -e '''/ prepare_environment()/a\ os.system(f"""sed -i -e ''"s/dict()))/dict())).cuda()/g"'' /kaggle/working/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/util.py""")''' /kaggle/working/stable-diffusion-webui/launch.py
!npm install -g localtunnel
import subprocess
import threading
import time
import socket
def iframe_thread(port):
while True:
time.sleep(0.5)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
result = sock.connect_ex(('127.0.0.1', port))
if result == 0:
break
sock.close()
p = subprocess.Popen(["lt", "--port", "{}".format(port)], stdout=subprocess.PIPE)
for line in p.stdout:
print(line.decode(), end='')
threading.Thread(target=iframe_thread, daemon=True, args=(7860,)).start()
#####################################################################
# 同步配置
# 使用huggingface保存和载入webui配置文件
huggingface_use = True
zip_output=True
huggingface_token_file = '/kaggle/input/hgtoken/huggingfacetoken.text'
huggiingface_repo_id = 'MSa/sd-output-images'
huggiingface_config_repo_id = 'MSa/sd-config'
# 将会同步的文件
config_files = [
'ui-config.json',
'config.json',
# 'styles.csv'
]
#功能函数
from pathlib import Path
from huggingface_hub import HfApi, login
import subprocess
import pandas as pd
import shutil
import os
import re
import gc
import requests
import datetime
import pytz
# 生成时间字符串
def timeStamp():
# 获取当前本地时间
local_time = datetime.datetime.now()
# 创建北京时区对象
beijing_tz = pytz.timezone('Asia/Shanghai')
# 将任意时区时间转换为北京时间
beijing_time = local_time.astimezone(beijing_tz)
timeStr = beijing_time.strftime('%Y%m%d-%H:%M:%S')
return timeStr
# config 上传
def hugface_upload_config(huggingface_token_file, config_files, repo_id):
if Path(huggingface_token_file).exists():
with open(huggingface_token_file, encoding="utf-8") as nkfile:
hugToken = nkfile.readline()
if hugToken != '':
# 使用您的 Hugging Face 访问令牌登录
login(token=hugToken)
# 实例化 HfApi 类
api = HfApi()
print("HfApi 类已实例化")
%cd /kaggle/working/stable-diffusion-webui
# 使用 upload_file() 函数上传文件
print("开始上传文件...")
for config_file in config_files:
if Path(config_file).exists():
response = api.upload_file(
path_or_fileobj=config_file,
path_in_repo=config_file,
repo_id=repo_id,
repo_type="dataset"
)
print("文件上传完成")
print(f"响应: {response}")
else:
print(f'Error: File {config_file} does not exist')
else:
print(f'Error: File {huggingface_token_file} does not exist')
# 上传图片压缩文件
def hugface_upload_zipfile(huggingface_token_file, zipfile, repo_id):
if Path(huggingface_token_file).exists():
with open(huggingface_token_file, encoding="utf-8") as nkfile:
hugToken = nkfile.readline()
if hugToken != '':
# 使用您的 Hugging Face 访问令牌登录
login(token=hugToken)
# 实例化 HfApi 类
api = HfApi()
print("HfApi 类已实例化")
%cd /kaggle/working/
# 使用 upload_file() 函数上传文件
print("开始上传文件...")
if Path(zipfile).exists():
response = api.upload_file(
path_or_fileobj=zipfile,
path_in_repo='output-images/' + zipfile,
repo_id=repo_id,
repo_type="dataset"
)
print("文件上传完成")
print(f"响应: {response}")
else:
print(f'Error: File {zipfile} does not exist')
else:
print(f'Error: File {huggingface_token_file} does not exist')
# 将生成的图片压缩
clear_output = True # 是否清理输出文件夹
def zip_updata():
if zip_output:
output_folder = f'/kaggle/working/stable-diffusion-webui/outputs/'
if os.path.exists(output_folder):
fileName = timeStamp()
filePath = '/kaggle/working/' + fileName
zip_file_path = shutil.make_archive(filePath, 'zip', output_folder)
print('图片已压缩到output')
if os.path.exists(zip_file_path):
# 上传压缩包
hugface_upload_zipfile(huggingface_token_file, fileName + ".zip", huggiingface_repo_id)
print("------hugface_upload_zipfile------")
else:
print(f'文件夹 {output_folder} 不存在,跳过压缩操作')
if clear_output:
%cd /kaggle/working/stable-diffusion-webui/outputs/
clean_folder('img2img-images')
clean_folder('txt2img-images')
clean_folder('img2img-grids')
clean_folder('txt2img-grids')
clean_folder('extras-images')
print('清理完毕')
if huggingface_use == True:
hugface_upload_config(huggingface_token_file,config_files,huggiingface_config_repo_id)
# 清理文件夹
def clean_folder(folder_path):
if not os.path.exists(folder_path):
return
for filename in os.listdir(folder_path):
file_path = os.path.join(folder_path, filename)
if os.path.isfile(file_path):
os.remove(file_path)
elif os.path.isdir(file_path):
shutil.rmtree(file_path)
#####################################################################
####################################################################
# 下载配置文件
def webui_config_download(config_files, huggiingface_repo_id):
print("--------开始同步Config--------")
%cd /kaggle/working/stable-diffusion-webui/
for config_file in config_files:
url = f'https://huggingface.co/datasets/{huggiingface_repo_id}/resolve/main/{config_file}'
response = requests.head(url)
if response.status_code == 200:
result = subprocess.run(['wget', '-O', config_file, url, '-q'], capture_output=True)
if result.returncode != 0:
print(f'Error: Failed to download {config_file} from {url}')
else:
print(f'{config_file} download success !!!')
else:
print(f'Error: Invalid URL {url}')
# 执行下载
webui_config_download(config_files, huggiingface_config_repo_id)
#######################################################
#启动参数
args = [
'--share',
'--xformers',
'--lowram',
'--enable-insecure-extension-access',
'--theme dark',
'--gradio-queue',
]
##################################
!python launch.py {' '.join(args)}
# 执行config上传(需要的时候才执行)
hugface_upload_config(huggingface_token_file, config_files, huggiingface_config_repo_id)
# 执行上传成品图片压缩包(需要的时候才执行)
zip_updata()