703 Commits

Author SHA1 Message Date
Zhedong Cen
8dd45459be
Add support for HTML file (#973)
### What problem does this PR solve?

Add support for HTML file

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-30 09:12:55 +08:00
CKLogic
dded365b8d
Fix:After being idle for a while, new tasks need to be cancel and redo (#958)
### What problem does this PR solve?

After being idle for a while (When Redis Queue exceeds the
SVR_QUEUE_RETENTION(60*60) expiration time), new tasks need to be cancel
and redo.

When use xgroup_create to create a consumer group, set the ID to "$",
meaning that only messages added to the stream after the group is
created will be visible to new consumers. If the application scenario
requires processing messages that already exist in the queue, you might
need to change this ID to "0", so that the new consumer group can read
all messages from the beginning.


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-30 09:03:11 +08:00
writinwaters
9fdd517af6
Update README.md (#978)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update
2024-05-29 20:22:41 +08:00
writinwaters
2604ded2e4
Update README.md (#976)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update
2024-05-29 20:02:16 +08:00
KevinHuSh
758eb03ccb
fix jina adding issure and term weight refinement (#974)
### What problem does this PR solve?

#724 #162

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
2024-05-29 19:38:57 +08:00
balibabu
e0d05a3895
fix: if the conversation name is too long, it will overflow the current item. #607 (#972)
### What problem does this PR solve?

fix: if the conversation name is too long, it will overflow the current
item. #607

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-29 18:32:03 +08:00
KevinHuSh
614defec21
add rerank model (#969)
### What problem does this PR solve?

feat: add rerank models to the project #724 #162

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-29 16:50:02 +08:00
balibabu
e1f0644deb
feat: add jina (#967)
### What problem does this PR solve?
feat: add jina #650 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-29 16:48:52 +08:00
balibabu
a135f9f5b6
feat: add rerank models to the project #724 #162 (#966)
### What problem does this PR solve?

Vector similarity weight is displayed incorrectly #965
feat: add rerank models to the project #724 #162
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-29 16:19:08 +08:00
KevinHuSh
daa4799385
limit the system context length of conversation messages. (#962)
### What problem does this PR solve?

#951 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-29 10:40:07 +08:00
balibabu
495a6434ec
feat: add FlowHeader and delete edge (#959)
### What problem does this PR solve?
feat: add FlowHeader and delete edge #918 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-29 10:01:39 +08:00
writinwaters
21aac545d9
Expanded the supported LLM list (#960)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update
2024-05-28 20:13:03 +08:00
Jin Hai
0f317221b4
Update README (#956)
### What problem does this PR solve?

Update README due to support new LLMs.

### Type of change

- [x] Documentation Update

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-05-28 20:05:02 +08:00
writinwaters
a427672229
Fixed a docusaurus display issue (#954)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update

---------

Co-authored-by: KevinHuSh <kevinhu.sh@gmail.com>
2024-05-28 17:26:13 +08:00
balibabu
196f2b445f
fix: fixed the issue of 404 error in the user settings page of the demo site (#948)
### What problem does this PR solve?

fix: fixed the issue of 404 error in the user settings page of the demo
site #947

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-28 11:22:56 +08:00
Zhedong Cen
5041677f11
Add umap-learn, fasttext and volcengine in requirements_arm.txt (#945)
### What problem does this PR solve?

Complete the requirements for ARM

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-28 11:13:48 +08:00
KevinHuSh
7eee193956
fix #917 #915 (#946)
### What problem does this PR solve?

#917 
#915

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-28 11:13:02 +08:00
yungongzi
9ffd7ae321
Added support for Baichuan LLM (#934)
### What problem does this PR solve?

- Added support for Baichuan LLM

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: 海贼宅 <stu_xyx@163.com>
2024-05-28 09:09:37 +08:00
writinwaters
ec6ae744a1
minor editorial updates for clarity (#941)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update
2024-05-27 20:35:08 +08:00
balibabu
d9bc093df1
feat: test buildNodesAndEdgesFromDSLComponents (#940)
### What problem does this PR solve?
 feat: test buildNodesAndEdgesFromDSLComponents #918

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-27 19:35:14 +08:00
Zhedong Cen
571aaaff22
Add Dockerfile and requirements.txt for arm (#936)
### What problem does this PR solve?

#253 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-27 19:34:49 +08:00
GYH
7d8e03ec38
Update docnm_kwd to doc_name (#939)
### What problem does this PR solve?

Update docnm_kwd to doc_name 
#908 

### Type of change


- [x] Refactoring
2024-05-27 19:14:04 +08:00
writinwaters
65677f65c9
Updated RESTful API Reference (#908)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update
2024-05-27 18:34:16 +08:00
Camus Gao
89d296feab
Remove duplicated FROM. (#935)
### What problem does this PR solve?
Remove duplicated FROM in Dockerfile.cuda.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-27 17:16:47 +08:00
writinwaters
3ae8a87986
Expanded list of locally deployed embedding models (#930)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update
2024-05-27 14:01:52 +08:00
KevinHuSh
46454362d7
fix raptor bugs (#928)
### What problem does this PR solve?

#922 
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-27 11:01:20 +08:00
balibabu
55fb96131e
feat: build react flow nodes and edges from mock data #918 (#919)
### What problem does this PR solve?
feat: build react flow nodes and edges from mock data #918

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-27 08:21:30 +08:00
dashi6174
20b57144b0
syntax error (#924)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-27 08:20:32 +08:00
dashi6174
9e3a0e4d03
The fasttext library is missing, and it is used in the operators.py file. (#925)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-27 08:18:47 +08:00
yungongzi
c0d71adaa2
Bug fix for volcengine (#909)
### What problem does this PR solve?
Bug fixes for the VolcEngine

- Bug fix for front-end configuration code of VolcEngine

- Bug fix for tokens counting logic of VolcEngine


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: 海贼宅 <stu_xyx@163.com>
2024-05-24 11:34:39 +08:00
Jin Hai
735bdf06a4
Update README (#901)
### What problem does this PR solve?

Update README due to implement RAPTOR.

### Type of change

- [x] Documentation Update

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-05-24 08:30:08 +08:00
dashi6174
fe18627ebc
Fix some syntax errors, re not import (#904)
re not import

### What problem does this PR solve?

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-23 19:38:13 +08:00
balibabu
4cda40c3ef
feat: fixed issue with threshold translation #882 and add NodeContextMenu (#906)
### What problem does this PR solve?

feat: fixed issue with threshold translation #882
feat: add NodeContextMenu

### Type of change


- [ ] New Feature (non-breaking change which adds functionality)
2024-05-23 18:53:04 +08:00
GYH
1e5c5abe58
Update api_md document/rm (#894)
### What problem does this PR solve?

Update api_md document/rm
#717 

### Type of change

- [x] Documentation Update
2024-05-23 15:19:58 +08:00
KevinHuSh
6f99bbbb08
add raptor (#899)
### What problem does this PR solve?

#882 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-23 14:31:16 +08:00
xinzhuang
3bbdf3b770
fixbug for computing 'not concating feature' (#896)
### What problem does this PR solve?

When pdfparser call `_naive_vertical_merge` method,there is a "not
concating feature " value by computing difference between `b` and `b_`'s
layoutno ,but actually is `b` and `b`. I think it's a bug, so fix it.
Please check again.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-23 14:29:42 +08:00
balibabu
070b53f3bf
feat: RAPTOR is not displayed when the parsing method is picture. (#897)
### What problem does this PR solve?

Implements RAPTOR for better chunking #882

### Type of change


- [x] New Feature (non-breaking change which adds functionality)
2024-05-23 14:13:09 +08:00
yungongzi
eb51ad73d6
Add support for VolcEngine - the current version supports SDK2 (#885)
- The main idea is to assemble **ak**, **sk**, and **ep_id** into a
dictionary and store it in the database **api_key** field
- I don’t know much about the front-end, so I learned from Ollama, which
may be redundant.

### Configuration method

- model name

- Format requirements: {"VolcEngine model name":"endpoint_id"}
    - For example: {"Skylark-pro-32K":"ep-xxxxxxxxx"}
    
- Volcano ACCESS_KEY
- Format requirements: VOLC_ACCESSKEY of the volcano engine
corresponding to the model

- Volcano SECRET_KEY
- Format requirements: VOLC_SECRETKEY of the volcano engine
corresponding to the model
    
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-23 11:15:29 +08:00
GYH
fbd0d74053
Add /api/document/rm function (#887)
### What problem does this PR solve?

Delete files from a knowledge base.

#717 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-23 10:32:56 +08:00
balibabu
170186ee4d
feat: remove the space before promptText (#886)
### What problem does this PR solve?

feat: remove the space before promptText #882 


### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-22 18:36:55 +08:00
balibabu
ed184ed87e
Implements RAPTOR for better chunking #882 (#883)
### What problem does this PR solve?

Implements RAPTOR for better chunking #882

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-22 18:04:18 +08:00
GYH
43412571f7
Add api.md:/api/list_kb_docs/ description (#881)
### What problem does this PR solve?

Add api.md:/api/list_kb_docs/ description
#717 

### Type of change

- [x] Documentation Update
2024-05-22 17:37:11 +08:00
LiYongHui
17489e6c6c
fix import error (#877)
Fix import error for user_app.py

---------

Co-authored-by: yonghui li <yonghui.li@bondex.com.cn>
2024-05-22 16:14:53 +08:00
dashi6174
21453ffff0
fixed: The choices may be empty. (#876)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-22 15:29:07 +08:00
GYH
be13429d05
Add api/list_kb_docs function and modify api/list_chunks (#874)
### What problem does this PR solve?
#717 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-22 14:58:56 +08:00
writinwaters
5178daeeaf
Fixed a format issue (#872)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update
2024-05-22 13:39:38 +08:00
writinwaters
d5b8d8e647
fixed a format issue for docusaurus publication (#871)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update
2024-05-22 12:45:34 +08:00
balibabu
b62a20816e
fix: display specific error message when previewing file error #868 (#869)
### What problem does this PR solve?

fix: display specific error message when previewing file error  #868


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-22 11:54:32 +08:00
writinwaters
3cae87a902
Reorganized docs for docusaurus publish (#860)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update
2024-05-21 20:53:55 +08:00
balibabu
1797f5ce31
fix: the site domain name in the Chat Bot API is hardcoded. #776 (#859)
### What problem does this PR solve?

fix: the site domain name in the Chat Bot API is hardcoded. #776

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-21 17:37:19 +08:00