59 Commits

Author SHA1 Message Date
黄腾
35b7d17d97
fix SILICONFLOW embedding error (#2363)
### What problem does this PR solve?

#2335  fix SILICONFLOW embedding error

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-09-11 12:17:44 +08:00
黄腾
12975cf128
Fix some security vulnerabilities. (#2160)
### What problem does this PR solve?

Fix some security vulnerabilities

### Type of change

- [x] Performance Improvement

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-29 16:21:32 +08:00
黄腾
99993e5026
add support for Voyage AI (#2159)
### What problem does this PR solve?

#1853  #2138 add support for Voyage AI

### Type of change
- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-08-29 16:14:49 +08:00
黄腾
733219cc3f
add support for Baidu yiyan (#2049)
### What problem does this PR solve?

add support for Baidu yiyan

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-22 16:45:15 +08:00
黄腾
79426fc41f
add support for Replicate (#1980)
### What problem does this PR solve?

#1853  add support for Replicate

### Type of change


- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-19 10:36:57 +08:00
黄腾
b4ef50bdb5
fix OpenAI Embedding length error (#1972)
### What problem does this PR solve?
 
#1958   fix OpenAI Embedding length error

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-16 09:49:27 +08:00
黄腾
e013ac52af
add support for SILICONFLOW (#1926)
### What problem does this PR solve?

#1853 add support for SILICONFLOW

### Type of change


- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-13 16:09:10 +08:00
黄腾
e9e39d57ce
add support for Upstage (#1902)
### What problem does this PR solve?

#1853  add support for Upstage

### Type of change


- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-12 11:06:25 +08:00
黄腾
94cb66ba80
add support for TogetherAI (#1890)
### What problem does this PR solve?

#1853 add support for TogetherAI

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-08-12 10:15:21 +08:00
黄腾
9a6dc89156
add support for PerfXCloud (#1883)
### What problem does this PR solve?

#1853  add support for PerfXCloud

### Type of change


- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-12 10:11:50 +08:00
黄腾
e34817c2a9
add support for cohere (#1849)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-07 18:40:51 +08:00
黄腾
60428c4ad2
fix LocalAI add bug (#1851)
### What problem does this PR solve?

#1848 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-07 18:10:42 +08:00
黄腾
b67484e77d
add supprot for OpenAI-API-Compatible llm (#1787)
### What problem does this PR solve?

#1771  add supprot for OpenAI-API-Compatible 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-06 16:20:21 +08:00
江不江
bc50f68127
fix embedding_model (#1698)
### What problem does this PR solve?
fix embedding_model #1692
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Signed-off-by: seaver <zhudan187@qq.com>
2024-07-25 11:43:43 +08:00
黄腾
e67bfca552
refactor some llm api using openai api format (#1692)
### What problem does this PR solve?

refactor some llm api using openai api format

### Type of change

- [x] Refactoring

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-07-25 10:23:35 +08:00
黄腾
d96348eb22
add support for LM Studio (#1663)
### What problem does this PR solve?

#1602 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-07-24 12:46:43 +08:00
黄腾
b4a281eca1
add support for NVIDIA llm (#1645)
### What problem does this PR solve?

add support for NVIDIA llm
### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-07-23 10:43:09 +08:00
黄腾
3fcdba1683
add support for LocalAI (#1608)
### What problem does this PR solve?

#762 

### Type of change
- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-07-19 15:50:28 +08:00
Kevin Hu
be6d5b76c3
fix embeding model for Azure (#1601)
### What problem does this PR solve?

#1599

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-07-19 09:22:59 +08:00
黄腾
3e9f444e6b
add support for Gemini (#1465)
### What problem does this PR solve?

#1036

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-07-11 15:41:00 +08:00
H
6144a109ab
Add Support for AWS Bedrock (#1408)
### What problem does this PR solve?

#308 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: KevinHuSh <kevinhu.sh@gmail.com>
2024-07-08 09:37:34 +08:00
LiYongHui
a6765e9ca4
Integrates LLM Azure OpenAI (#1318)
### What problem does this PR solve?

feat: Integrates LLM Azure OpenAI #716 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

### Other
It's just the back-end code, the front-end needs to provide the Azure
OpenAI model addition form.
   
#### Required parameters

- base_url
- api_key

---------

Co-authored-by: yonghui li <yonghui.li@bondex.com.cn>
2024-07-04 09:57:16 +08:00
zhuhao
47926a95ae
Fix ragflow may encounter an OOM (Out Of Memory) when there are a lot of conversations (#1292)
### What problem does this PR solve?

Fix ragflow may encounter an OOM (Out Of Memory) when there are a lot of
conversations.
#1288

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: zhuhao <zhuhao@linklogis.com>
2024-06-27 14:48:49 +08:00
KevinHuSh
7dc39cbfa6
add support for mistral (#1153)
### What problem does this PR solve?

#433 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-06-14 11:32:58 +08:00
KevinHuSh
2023fdc13e
fix file preview in file management (#1151)
### What problem does this PR solve?

fix file preview in file management

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-06-14 10:33:59 +08:00
KevinHuSh
abcd3d2469
refactor (#1124)
### What problem does this PR solve?


### Type of change

- [x] Refactoring
2024-06-12 11:02:15 +08:00
Michael Feil
68a698655a
infinity: Update embedding_model.py (#1109)
### What problem does this PR solve?

I implemented infinity, a fast vector embeddings engine. 

### Type of change


- [x] Performance Improvement
- [X] Other (please describe):
2024-06-11 08:23:58 +08:00
KevinHuSh
923c3b8cac
fix bug in api (#1088)
### What problem does this PR solve?

#1075 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-06-07 09:56:14 +08:00
KevinHuSh
b9bb11879f
fix #994 (#1006)
### What problem does this PR solve?

#994 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-31 09:46:22 +08:00
KevinHuSh
758eb03ccb
fix jina adding issure and term weight refinement (#974)
### What problem does this PR solve?

#724 #162

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
2024-05-29 19:38:57 +08:00
KevinHuSh
614defec21
add rerank model (#969)
### What problem does this PR solve?

feat: add rerank models to the project #724 #162

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-29 16:50:02 +08:00
yungongzi
9ffd7ae321
Added support for Baichuan LLM (#934)
### What problem does this PR solve?

- Added support for Baichuan LLM

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: 海贼宅 <stu_xyx@163.com>
2024-05-28 09:09:37 +08:00
KevinHuSh
4153a36683
truncate text to fitin embedding model (#692)
### What problem does this PR solve?


### Type of change

- [x] Refactoring
2024-05-09 11:35:08 +08:00
Fakai Zhao
de839fc3f0
optimize srv broker and executor logic (#630)
### What problem does this PR solve?

Optimize task broker and executor for reduce memory usage and deployment
complexity.

### Type of change
- [x] Performance Improvement
- [x] Refactoring

### Change Log
- Enhance redis utils for message queue(use stream)
- Modify task broker logic via message queue (1.get parse event from
message queue 2.use ThreadPoolExecutor async executor )
- Modify the table column name of document and task (process_duation ->
process_duration maybe just a spelling mistake)
- Reformat some code style(just what i see)
- Add requirement_dev.txt for developer
- Add redis container on docker compose

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-05-07 11:43:33 +08:00
KevinHuSh
674b3aeafd
fix disable and enable llm setting in dialog (#616)
### What problem does this PR solve?
#614 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-30 11:04:14 +08:00
KevinHuSh
2af74cc494
refine docker layers (#606)
### What problem does this PR solve?


### Type of change

- [x] Performance Improvement
2024-04-29 17:57:40 +08:00
KevinHuSh
9d60a84958
refactor code (#583)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-28 13:19:54 +08:00
KevinHuSh
66f8d35632
Refactor (#537)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-25 14:14:28 +08:00
KevinHuSh
453c29170f
make sure the models will not be load twice (#422)
### What problem does this PR solve?

#381 
### Type of change

- [x] Refactoring
2024-04-18 09:37:23 +08:00
KevinHuSh
890561703b
Add bce-embedding and fastembed (#383)
### What problem does this PR solve?


Issue link:#326

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-04-16 16:42:19 +08:00
Anush
826ad6a33a
feat: FastEmbed embedding support (#291)
### Description

Following up on https://github.com/infiniflow/ragflow/pull/275, this PR
adds support for FastEmbed model configurations.

The options are not exhaustive. You can find the full list
[here](https://qdrant.github.io/fastembed/examples/Supported_Models/).

P.S. I ran into OOM issues when building the image.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: KevinHuSh <kevinhu.sh@gmail.com>
2024-04-15 15:58:06 +08:00
KevinHuSh
91068edf16
Support Xinference (#320)
### What problem does this PR solve?

Issue link:#299

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-04-11 18:22:25 +08:00
KevinHuSh
3708b97db9
Support Ollama (#261)
### What problem does this PR solve?

Issue link:#221

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-04-08 19:20:57 +08:00
KevinHuSh
e876f58b4c
refine readme (#170) 2024-03-29 14:38:15 +08:00
KevinHuSh
38e5737067
add base url for OpenAI (#166) 2024-03-28 19:15:16 +08:00
KevinHuSh
a5384446e3
let's load model from local (#163) 2024-03-28 16:10:47 +08:00
KevinHuSh
bf2e3d7fc1
refine OpenAi Api (#159) 2024-03-27 17:55:45 +08:00
KevinHuSh
fd7fcb5baf
apply pep8 formalize (#155) 2024-03-27 11:33:46 +08:00
KevinHuSh
979b3a5b4b
support snapshot download from local (#153)
* support snapshot download from local

* let snapshot download from local
2024-03-27 09:53:42 +08:00
KevinHuSh
da21320b88
fix plainPdf bugs (#152) 2024-03-26 15:11:07 +08:00