diff --git a/README.md.bak b/README.md.bak index d2b07a8..01194c9 100644 --- a/README.md.bak +++ b/README.md.bak @@ -1,3 +1,5 @@ +
+ **A GPT-4V Level Multimodal LLM on Your Phone** @@ -5,6 +7,9 @@ [ไธญๆ–‡](./README_zh.md) | English +Join our ๐Ÿ’ฌ WeChat + +

MiniCPM-Llama3-V 2.5 ๐Ÿค— ๐Ÿค– | MiniCPM-V 2.0 ๐Ÿค— ๐Ÿค– | @@ -25,14 +30,14 @@ #### ๐Ÿ“Œ Pinned -* [2024.05.28] ๐Ÿš€๐Ÿš€๐Ÿš€ MiniCPM-Llama3-V 2.5 now fully supports its feature in [llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md) and [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5)! Please pull the latest code for llama.cpp & ollama. We also release GGUF in various sizes [here](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf/tree/main). FAQ list for ollama usage is comming within a day. Please stay tuned! +* [2024.05.28] ๐Ÿš€๐Ÿš€๐Ÿš€ MiniCPM-Llama3-V 2.5 now fully supports its feature in llama.cpp and ollama! Please pull the latest code **of our provided forks** ([llama.cpp](https://github.com/OpenBMB/llama.cpp/blob/minicpm-v2.5/examples/minicpmv/README.md), [ollama](https://github.com/OpenBMB/ollama/tree/minicpm-v2.5/examples/minicpm-v2.5)). GGUF models in various sizes are available [here](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf/tree/main). MiniCPM-Llama3-V 2.5 series is **not supported by the official repositories yet**, and we are working hard to merge PRs. Please stay tuned! * [2024.05.28] ๐Ÿ’ซ We now support LoRA fine-tuning for MiniCPM-Llama3-V 2.5, using only 2 V100 GPUs! See more statistics [here](https://github.com/OpenBMB/MiniCPM-V/tree/main/finetune#model-fine-tuning-memory-usage-statistics). * [2024.05.23] ๐Ÿ” We've released a comprehensive comparison between Phi-3-vision-128k-instruct and MiniCPM-Llama3-V 2.5, including benchmarks evaluations, multilingual capabilities, and inference efficiency ๐ŸŒŸ๐Ÿ“Š๐ŸŒ๐Ÿš€. Click [here](./docs/compare_with_phi-3_vision.md) to view more details. * [2024.05.23] ๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅ MiniCPM-V tops GitHub Trending and Hugging Face Trending! Our demo, recommended by Hugging Face Gradioโ€™s official account, is available [here](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5). Come and try it out!
- +* [2024.06.03] Now, you can run MiniCPM-Llama3-V 2.5 on multiple low VRAM GPUs(12 GB or 16 GB) by distributing the model's layers across multiple GPUs. For more details, Check this [link](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/inference_on_multiple_gpus.md). * [2024.05.25] MiniCPM-Llama3-V 2.5 now supports streaming outputs and customized system prompts. Try it [here](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5#usage)! * [2024.05.24] We release the MiniCPM-Llama3-V 2.5 [gguf](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5-gguf), which supports [llama.cpp](#inference-with-llamacpp) inference and provides a 6~8 token/s smooth decoding on mobile phones. Try it now! * [2024.05.20] We open-soure MiniCPM-Llama3-V 2.5, it has improved OCR capability and supports 30+ languages, representing the first end-side MLLM achieving GPT-4V level performance! We provide [efficient inference](#deployment-on-mobile-phone) and [simple fine-tuning](./finetune/readme.md). Try it now! @@ -51,14 +56,13 @@ - [MiniCPM-Llama3-V 2.5](#minicpm-llama3-v-25) - [MiniCPM-V 2.0](#minicpm-v-20) -- [Online Demo](#online-demo) +- [Chat with Our Demo on Gradio](#chat-with-our-demo-on-gradio) - [Install](#install) - [Inference](#inference) - [Model Zoo](#model-zoo) - [Multi-turn Conversation](#multi-turn-conversation) - [Inference on Mac](#inference-on-mac) - [Deployment on Mobile Phone](#deployment-on-mobile-phone) - - [WebUI Demo](#webui-demo) - [Inference with llama.cpp](#inference-with-llamacpp) - [Inference with vLLM](#inference-with-vllm) - [Fine-tuning](#fine-tuning) @@ -456,9 +460,31 @@ We deploy MiniCPM-V 2.0 on end devices. The demo video is the raw screen recordi | OmniLMM-12B | [Document](./omnilmm_en.md) | +## Chat with Our Demo on Gradio + +We provide online and local demos powered by HuggingFace [Gradio](https://github.com/gradio-app/gradio), the most popular model deployment framework nowadays. It supports streaming outputs, progress bars, queuing, alerts, and other useful features. + + +### Online Demo + +Click here to try out the online demo of [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) ๏ฝœ [MiniCPM-V 2.0](https://huggingface.co/spaces/openbmb/MiniCPM-V-2) on HuggingFace Spaces. + +### Local WebUI Demo + +You can easily build your own local WebUI demo with Gradio using the following commands. + +```shell +pip install -r requirements.txt +``` + +```shell +# For NVIDIA GPUs, run: +python web_demo_2.5.py --device cuda + +# For Mac with MPS (Apple silicon or AMD GPUs), run: +PYTORCH_ENABLE_MPS_FALLBACK=1 python web_demo_2.5.py --device mps +``` -## Online Demo -Click here to try out the Demo of [MiniCPM-Llama3-V 2.5](https://huggingface.co/spaces/openbmb/MiniCPM-Llama3-V-2_5) ๏ฝœ [MiniCPM-V 2.0](https://huggingface.co/spaces/openbmb/MiniCPM-V-2). ## Install @@ -578,25 +604,7 @@ PYTORCH_ENABLE_MPS_FALLBACK=1 python test.py ### Deployment on Mobile Phone -MiniCPM-V 2.0 can be deployed on mobile phones with Android operating systems. ๐Ÿš€ Click [here](https://github.com/OpenBMB/mlc-MiniCPM) to install apk. MiniCPM-Llama3-V 2.5 coming soon. - -### WebUI Demo - -

-Click to see how to deploy WebUI demo on different devices - -```shell -pip install -r requirements.txt -``` - -```shell -# For NVIDIA GPUs, run: -python web_demo_2.5.py --device cuda - -# For Mac with MPS (Apple silicon or AMD GPUs), run: -PYTORCH_ENABLE_MPS_FALLBACK=1 python web_demo_2.5.py --device mps -``` -
+MiniCPM-Llama3-V 2.5 and MiniCPM-V 2.0 can be deployed on mobile phones with Android operating systems. ๐Ÿš€ Click [MiniCPM-Llama3-V 2.5](http://minicpm.modelbest.cn/android/modelbest-release-20240528_182155.apk) / [MiniCPM-V 2.0](https://github.com/OpenBMB/mlc-MiniCPM) to install apk. ### Inference with llama.cpp MiniCPM-Llama3-V 2.5 can run with llama.cpp now! See our fork of [llama.cpp](https://github.com/OpenBMB/llama.cpp/tree/minicpm-v2.5/examples/minicpmv) for more detail. This implementation supports smooth inference of 6~8 token/s on mobile phones (test environment๏ผšXiaomi 14 pro + Snapdragon 8 Gen 3). @@ -604,7 +612,7 @@ MiniCPM-Llama3-V 2.5 can run with llama.cpp now! See our fork of [llama.cpp](htt ### Inference with vLLM
-Click to see how to inference with vLLM +Click to see how to inference MiniCPM-V 2.0 with vLLM (MiniCPM-Llama3-V 2.5 coming soon) Because our pull request to vLLM is still waiting for reviewing, we fork this repository to build and test our vLLM demo. Here are the steps: 1. Clone our version of vLLM: @@ -618,7 +626,7 @@ pip install -e . ``` 3. Install timm: ```shell -pip install timm=0.9.10 +pip install timm==0.9.10 ``` 4. Run our demo: ```shell @@ -649,13 +657,12 @@ Best Practices๏ผš[MiniCPM-V 1.0](https://github.com/modelscope/swift/blob/main/d ## Model License -The code in this repo is released according to [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) +* This repository is released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) License. -The usage of MiniCPM-V's and OmniLMM's parameters is subject to "[General Model License Agreement - Source Notes - Publicity Restrictions - Commercial License](https://github.com/OpenBMB/General-Model-License/blob/main/้€š็”จๆจกๅž‹่ฎธๅฏๅ่ฎฎ-ๆฅๆบ่ฏดๆ˜Ž-ๅฎฃไผ ้™ๅˆถ-ๅ•†ไธšๆŽˆๆƒ.md)" +* The usage of MiniCPM-V model weights must strictly follow [MiniCPM Model License.md](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%20Model%20License.md). -The parameters are fully open to academic research - -Please contact cpm@modelbest.cn to obtain written authorization for commercial uses. Free commercial use is also allowed after registration. +* The models and weights of MiniCPM are completely free for academic research. after filling out a ["questionnaire"](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g) for registration, are also available for free commercial use. + ## Statement @@ -680,9 +687,31 @@ This project is developed by the following institutions: ## ๐ŸŒŸ Star History -
- -
+ + +

+ +

+
+ + ## Citation