mirror of
https://git.mirrors.martin98.com/https://github.com/infiniflow/ragflow.git
synced 2025-04-23 22:50:17 +08:00
111501af5e
### What problem does this PR solve?  make `<xxxx>` visiable it was misinterpreted as part of the HTML tags  Issue link:None ### Type of change - [ ] Bug Fix (non-breaking change which fixes an issue) - [ ] New Feature (non-breaking change which adds functionality) - [ ] Breaking Change (fix or feature that could cause existing functionality not to work as expected) - [x] Documentation Update - [ ] Refactoring - [ ] Performance Improvement - [ ] Test cases - [ ] Python SDK impacted, Need to update PyPI - [ ] Other (please describe):
2.0 KiB
2.0 KiB
Xinference
Xorbits Inference(Xinference) empowers you to unleash the full potential of cutting-edge AI models.
Install
To start a local instance of Xinference, run the following command:
$ xinference-local --host 0.0.0.0 --port 9997
Launch Xinference
Decide which LLM you want to deploy (here's a list for supported LLM), say, mistral. Execute the following command to launch the model, remember to replace ${quantization} with your chosen quantization method from the options listed above:
$ xinference launch -u mistral --model-name mistral-v0.1 --size-in-billions 7 --model-format pytorch --quantization ${quantization}
Use Xinference in RAGFlow
- Go to 'Settings > Model Providers > Models to be added > Xinference'.
Base URL: Enter the base URL where the Xinference service is accessible, like,
http://<your-xinference-endpoint-domain>:9997/v1.
- Use Xinference Models.