nodes/city96-ComfyUI_ExtraModels
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

T5 offload w/ bnb

Browse Source 
Mentioned in #22
This commit is contained in:
City
2024-04-21 20:41:25 +02:00
parent 92323c62e7
commit f9e1178de7
3 changed files with 5 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1 -1
View File
@@ -140,7 +140,7 @@ If you have a second GPU, selecting "cuda:1" as the device will allow you to use
Loaded in bnb4bit mode, it only takes around 6GB VRAM, making it work with 12GB cards. The only drawback is that it'll constantly stay in VRAM since BitsAndBytes doesn't allow moving the weights to the system RAM temporarily. Switching to a different workflow *should* still release the VRAM as expected. Pascal cards (1080ti, P40) seem to struggle with 4bit. Select "cpu" if you encounter issues.
On windows, you may need a newer version of bitsandbytes for 4bit. Try `python -m pip install bitsandbytes --prefer-binary --extra-index-url=https://jllllll.github.io/bitsandbytes-windows-webui`
On windows, you may need a newer version of bitsandbytes for 4bit. Try `python -m pip install bitsandbytes`
> [!IMPORTANT]
> You may also need to upgrade transformers and install spiece for the tokenizer. `pip install -r requirements.txt`