diff --git a/README.md b/README.md index 4221719..b6398f9 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,7 @@
+
 
InternLM @@ -17,6 +18,7 @@
+ [![evaluation](https://github.com/InternLM/InternLM/assets/22529082/f80a2a58-5ddf-471a-8da4-32ab65c8fd3b)](https://github.com/internLM/OpenCompass/) [💻Github Repo](https://github.com/InternLM/InternLM) • [🤔Reporting Issues](https://github.com/InternLM/InternLM/issues/new) • [📜Technical Report](https://arxiv.org/abs/2403.17297) @@ -29,14 +31,15 @@ + ## Introduction InternLM3 has open-sourced an 8-billion parameter instruction model, InternLM3-8B-Instruct, designed for general-purpose usage and advanced reasoning. This model has the following characteristics: - **Enhanced performance at reduced cost**: -State-of-the-art performance on reasoning and knowledge-intensive tasks surpass models like Llama3.1-8B and Qwen2.5-7B. Remarkably, InternLM3 is trained on only 4 trillion high-quality tokens, saving more than 75% of the training cost compared to other LLMs of similar scale. + State-of-the-art performance on reasoning and knowledge-intensive tasks surpass models like Llama3.1-8B and Qwen2.5-7B. Remarkably, InternLM3 is trained on only 4 trillion high-quality tokens, saving more than 75% of the training cost compared to other LLMs of similar scale. - **Deep thinking capability**: -InternLM3 supports both the deep thinking mode for solving complicated reasoning tasks via the long chain-of-thought and the normal response mode for fluent user interactions. + InternLM3 supports both the deep thinking mode for solving complicated reasoning tasks via the long chain-of-thought and the normal response mode for fluent user interactions. ## InternLM3-8B-Instruct @@ -67,7 +70,9 @@ We conducted a comprehensive evaluation of InternLM using the open-source evalua - The evaluation data may have numerical differences due to the version iteration of [OpenCompass](https://github.com/internLM/OpenCompass/), so please refer to the latest evaluation results of [OpenCompass](https://github.com/internLM/OpenCompass/). **Limitations:** Although we have made efforts to ensure the safety of the model during the training process and to encourage the model to generate text that complies with ethical and legal requirements, the model may still produce unexpected outputs due to its size and probabilistic generation paradigm. For example, the generated responses may contain biases, discrimination, or other harmful content. Please do not propagate such content. We are not responsible for any consequences resulting from the dissemination of harmful information. + ### Requirements + ```python transformers >= 4.48 ``` @@ -81,7 +86,7 @@ To load the InternLM3 8B Instruct model using Transformers, use the following co ```python import torch -from transformers import AutoTokenizer, AutoModelForCausalLM +from modelscope import AutoTokenizer, AutoModelForCausalLM model_dir = "internlm/internlm3-8b-instruct" tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True) @@ -108,11 +113,14 @@ generated_ids = model.generate(tokenized_chat, max_new_tokens=1024, temperature= generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(tokenized_chat, generated_ids) ] +prompt = tokenizer.batch_decode(tokenized_chat)[0] +print(prompt) response = tokenizer.batch_decode(generated_ids)[0] print(response) ``` #### LMDeploy inference + LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by the MMRazor and MMDeploy teams. ```bash @@ -153,6 +161,10 @@ Find more details in the [LMDeploy documentation](https://lmdeploy.readthedocs.i +#### Ollama inference + +TODO + #### vLLM inference We are still working on merging the PR(https://github.com/vllm-project/vllm/pull/12037) into vLLM. In the meantime, please use the following PR link to install it manually. @@ -195,6 +207,7 @@ print(outputs) ### Thinking Mode + #### Thinking Demo @@ -206,6 +219,7 @@ print(outputs) #### Thinking system prompt + ```python thinking_system_prompt = """You are an expert mathematician with extensive experience in mathematical competitions. You approach problems through systematic thinking and rigorous reasoning. When solving problems, follow these thought processes: ## Deep Understanding @@ -253,10 +267,12 @@ When you're ready, present your complete solution with: Focus on clear, logical progression of ideas and thorough explanation of your mathematical reasoning. Provide answers in the same language as the user asking the question, repeat the final answer using a '\\boxed{}' without any units, you have [[8192]] tokens to complete the answer. """ ``` + #### Transformers inference + ```python import torch -from transformers import AutoTokenizer, AutoModelForCausalLM +from modelscope import AutoTokenizer, AutoModelForCausalLM model_dir = "internlm/internlm3-8b-instruct" tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True) @@ -280,9 +296,12 @@ generated_ids = model.generate(tokenized_chat, max_new_tokens=8192) generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(tokenized_chat, generated_ids) ] +prompt = tokenizer.batch_decode(tokenized_chat)[0] +print(prompt) response = tokenizer.batch_decode(generated_ids)[0] print(response) ``` + #### LMDeploy inference LMDeploy is a toolkit for compressing, deploying, and serving LLM. @@ -308,9 +327,14 @@ response = pipe(messages, gen_config=GenerationConfig(max_new_tokens=2048)) print(response) ``` +#### Ollama inference + +TODO + #### vLLM inference We are still working on merging the PR(https://github.com/vllm-project/vllm/pull/12037) into vLLM. In the meantime, please use the following PR link to install it manually. + ```python git clone https://github.com/RunningLeon/vllm.git pip install -e . @@ -369,9 +393,9 @@ The code is licensed under Apache-2.0, while model weights are fully open for ac InternLM3，即书生·浦语大模型第3代，开源了80亿参数，面向通用使用与高阶推理的指令模型（InternLM3-8B-Instruct）。模型具备以下特点： - **更低的代价取得更高的性能**: -在推理、知识类任务上取得同量级最优性能，超过Llama3.1-8B和Qwen2.5-7B. 值得关注的是InternLM3只用了4万亿词元进行训练，对比同级别模型训练成本节省75%以上。 + 在推理、知识类任务上取得同量级最优性能，超过Llama3.1-8B和Qwen2.5-7B。值得关注的是InternLM3只用了4万亿词元进行训练，对比同级别模型训练成本节省75%以上。 - **深度思考能力**: -InternLM3支持通过长思维链求解复杂推理任务的深度思考模式，同时还兼顾了用户体验更流畅的通用回复模式。 + InternLM3支持通过长思维链求解复杂推理任务的深度思考模式，同时还兼顾了用户体验更流畅的通用回复模式。 #### 性能评测 @@ -418,7 +442,7 @@ transformers >= 4.48 ```python import torch -from transformers import AutoTokenizer, AutoModelForCausalLM +from modelscope import AutoTokenizer, AutoModelForCausalLM model_dir = "internlm/internlm3-8b-instruct" tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True) @@ -445,6 +469,8 @@ generated_ids = model.generate(tokenized_chat, max_new_tokens=1024, temperature= generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(tokenized_chat, generated_ids) ] +prompt = tokenizer.batch_decode(tokenized_chat)[0] +print(prompt) response = tokenizer.batch_decode(generated_ids)[0] print(response) ``` @@ -491,7 +517,12 @@ curl http://localhost:23333/v1/chat/completions \ +##### Ollama 推理 + +TODO + ##### vLLM 推理 + 我们还在推动PR(https://github.com/vllm-project/vllm/pull/12037) 合入vllm，现在请使用以下PR链接手动安装 ```python @@ -592,7 +623,7 @@ Focus on clear, logical progression of ideas and thorough explanation of your ma ```python import torch -from transformers import AutoTokenizer, AutoModelForCausalLM +from modelscope import AutoTokenizer, AutoModelForCausalLM model_dir = "internlm/internlm3-8b-instruct" tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True) @@ -616,9 +647,12 @@ generated_ids = model.generate(tokenized_chat, max_new_tokens=8192) generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(tokenized_chat, generated_ids) ] +prompt = tokenizer.batch_decode(tokenized_chat)[0] +print(prompt) response = tokenizer.batch_decode(generated_ids)[0] print(response) ``` + ##### LMDeploy 推理 LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by the MMRazor and MMDeploy teams. @@ -644,6 +678,10 @@ response = pipe(messages, gen_config=GenerationConfig(max_new_tokens=2048)) print(response) ``` +##### Ollama 推理 + +TODO + ##### vLLM 推理 我们还在推动PR(https://github.com/vllm-project/vllm/pull/12037) 合入vllm，现在请使用以下PR链接手动安装 @@ -681,10 +719,6 @@ print(outputs) - - - - ## 开源许可证 本仓库的代码依照 Apache-2.0 协议开源。模型权重对学术研究完全开放，也可申请免费的商业使用授权（[申请表](https://wj.qq.com/s2/12725412/f7c1/)）。其他问题与合作请联系 。 @@ -700,5 +734,4 @@ print(outputs) archivePrefix={arXiv}, primaryClass={cs.CL} } -``` - +``` \ No newline at end of file