137 Commits

Author SHA1 Message Date
KevinHuSh
6f99bbbb08
add raptor (#899)
### What problem does this PR solve?

#882 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-23 14:31:16 +08:00
yungongzi
eb51ad73d6
Add support for VolcEngine - the current version supports SDK2 (#885)
- The main idea is to assemble **ak**, **sk**, and **ep_id** into a
dictionary and store it in the database **api_key** field
- I don’t know much about the front-end, so I learned from Ollama, which
may be redundant.

### Configuration method

- model name

- Format requirements: {"VolcEngine model name":"endpoint_id"}
    - For example: {"Skylark-pro-32K":"ep-xxxxxxxxx"}
    
- Volcano ACCESS_KEY
- Format requirements: VOLC_ACCESSKEY of the volcano engine
corresponding to the model

- Volcano SECRET_KEY
- Format requirements: VOLC_SECRETKEY of the volcano engine
corresponding to the model
    
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-23 11:15:29 +08:00
GYH
fbd0d74053
Add /api/document/rm function (#887)
### What problem does this PR solve?

Delete files from a knowledge base.

#717 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-23 10:32:56 +08:00
LiYongHui
17489e6c6c
fix import error (#877)
Fix import error for user_app.py

---------

Co-authored-by: yonghui li <yonghui.li@bondex.com.cn>
2024-05-22 16:14:53 +08:00
GYH
be13429d05
Add api/list_kb_docs function and modify api/list_chunks (#874)
### What problem does this PR solve?
#717 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-22 14:58:56 +08:00
KevinHuSh
bae376a479
Update db_models.py 2024-05-21 12:02:22 +08:00
dashi6174
7e74546b73
Set the language default value of the language based on the LANG envi… (#853)
…ronment variable at the initial creation.

1. Set the User's default language based on LANG;
2. Set the Knowledgebase's default language based on LANG; 
3. Set the default language of the Dialog based on LANG;

### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2024-05-21 11:05:41 +08:00
dashi6174
9a01d1b876
The default max tokens of 215 is too small, answers are often cut off.I will modify it to 512 to address this issue. (#845)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-05-20 17:25:19 +08:00
KevinHuSh
2b36283712
fix english query bug (#840)
### What problem does this PR solve?

#834 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-20 12:23:51 +08:00
KevinHuSh
6683179d6a
fix bug about removing KB. (#839)
### What problem does this PR solve?

#838 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-20 09:23:57 +08:00
KevinHuSh
673a28e492
fix bug of chat without stream (#830)
### What problem does this PR solve?

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-17 20:03:00 +08:00
KevinHuSh
b3c923da6b
add doc ids in API: completion (#827)
### What problem does this PR solve?
#808 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-17 17:51:54 +08:00
KevinHuSh
a1586e0af9
correct mismatched kb doc number (#826)
### What problem does this PR solve?

#620

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-17 17:27:39 +08:00
KevinHuSh
f6a599461f
fix zhipuAI stream issue (#825)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-17 17:07:33 +08:00
GYH
081f922ee6
0517 list chunks (#821)
### What problem does this PR solve?

#717 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-17 15:58:05 +08:00
KevinHuSh
a2a6a35e94
fix doc number miss-match issue (#822)
### What problem does this PR solve?

#620 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-17 15:35:09 +08:00
KevinHuSh
9e5d501e83
fix data init error (#820)
### What problem does this PR solve?

#810 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-17 14:33:19 +08:00
KevinHuSh
c3bc72dfd9
fix too large thumbnail issue (#817)
### What problem does this PR solve?

#709

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-17 14:04:21 +08:00
LiYongHui
2dd705fe68
feat: add feishu oauth (#815)
### What problem does this PR solve?

The back-end code adds Feishu oauth

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: yonghui li <yonghui.li@bondex.com.cn>
2024-05-17 13:47:05 +08:00
KevinHuSh
d1614107e2
fix stream chat for ollama (#816)
### What problem does this PR solve?

#709

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-17 12:07:00 +08:00
KevinHuSh
e73ce39b66
Add 2 embeding models from OpenAI (#812)
### What problem does this PR solve?

#810 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-17 08:51:29 +08:00
KevinHuSh
95f809187e
add stream chat (#811)
### What problem does this PR solve?

#709 
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-16 20:14:53 +08:00
KevinHuSh
d6772f5dd7
add version (#807)
### What problem does this PR solve?
#709 
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-16 16:17:48 +08:00
KevinHuSh
7b144cc086
fix: can't capitalize file or folder name (#798)
### What problem does this PR solve?


#792 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-16 09:10:29 +08:00
KevinHuSh
1c4e92ed35
Knowledge base search is case sensitive (#797)
### What problem does this PR solve?
#793 
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-16 09:00:12 +08:00
dashi6174
6ff63ee2ba
Support for code files parse (#789)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-15 16:34:28 +08:00
KevinHuSh
d8c080ee52
fix bugs in searching file using keywords (#780)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-15 12:51:57 +08:00
GYH
626ace8639
Updated document upload method (#777)
### What problem does this PR solve?

api_app.py
/document/upload 
add two non mandatory parameters
parser_id:
[naive,qaresume,manual,table,paper,book,laws,presentation,picture,one]
run: 1

### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-05-15 12:22:11 +08:00
KevinHuSh
aa1c915d6e
support gpt-4o (#773)
### What problem does this PR solve?
#771 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-15 11:16:08 +08:00
Jin Hai
d7744f5870
Refactor method name (#760)
### What problem does this PR solve?

#757

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-05-14 14:48:15 +08:00
KevinHuSh
aa03dfa453
fix bug of get file (#746)
### What problem does this PR solve?

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-13 14:02:38 +08:00
Sawyer
69b7c61498
fix: typo in user_app.py (#740)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Bug Fix (non-breaking change 
- [x] Other (please describe): Fix typo
2024-05-13 09:25:45 +08:00
KevinHuSh
04a9e95161
let file in knowledgebases visible in file manager (#714)
### What problem does this PR solve?

Let file in knowledgebases visible in file manager.
#162 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-11 16:04:28 +08:00
KevinHuSh
7ddb2f19be
make sure to raise exception if redis is not there (#674)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-05-08 15:20:45 +08:00
KevinHuSh
eb27a4309e
add support for deepseek (#668)
### What problem does this PR solve?

#666 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-08 10:30:02 +08:00
KevinHuSh
8d6d7f6887
fix task losting isssue (#665)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-07 20:46:45 +08:00
KevinHuSh
a5aed2412f
fix bugs (#662)
### What problem does this PR solve?

Fix import error for task_service.py

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-07 16:41:56 +08:00
Fakai Zhao
de839fc3f0
optimize srv broker and executor logic (#630)
### What problem does this PR solve?

Optimize task broker and executor for reduce memory usage and deployment
complexity.

### Type of change
- [x] Performance Improvement
- [x] Refactoring

### Change Log
- Enhance redis utils for message queue(use stream)
- Modify task broker logic via message queue (1.get parse event from
message queue 2.use ThreadPoolExecutor async executor )
- Modify the table column name of document and task (process_duation ->
process_duration maybe just a spelling mistake)
- Reformat some code style(just what i see)
- Add requirement_dev.txt for developer
- Add redis container on docker compose

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-05-07 11:43:33 +08:00
KevinHuSh
c6b6c748ae
fix file encoding detection bug (#653)
### What problem does this PR solve?

#651 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-07 10:01:24 +08:00
Moonlit
ca5acc151a
Refactor: Use TaskStatus enum for task status handling (#646)
### What problem does this PR solve?

This commit changes the status 'not started' from being hard-coded to
being maintained by the TaskStatus enum. This enhancement ensures
consistency across the codebase and improves maintainability.

### Type of change

- [x] Refactoring
2024-05-06 18:39:17 +08:00
KevinHuSh
5f03a4de11
remove redis (#629)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-30 19:00:41 +08:00
KevinHuSh
cab274f560
remove PyMuPDF (#618)
### What problem does this PR solve?
#613 

### Type of change


- [x] Other (please describe):
2024-04-30 12:38:09 +08:00
KevinHuSh
674b3aeafd
fix disable and enable llm setting in dialog (#616)
### What problem does this PR solve?
#614 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-30 11:04:14 +08:00
KevinHuSh
2af74cc494
refine docker layers (#606)
### What problem does this PR solve?


### Type of change

- [x] Performance Improvement
2024-04-29 17:57:40 +08:00
KevinHuSh
6874c6f3a7
refine document upload (#602)
### What problem does this PR solve?

#567 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-29 15:45:08 +08:00
KevinHuSh
8c07992b6c
refine code (#595)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-28 19:13:33 +08:00
KevinHuSh
9d60a84958
refactor code (#583)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-28 13:19:54 +08:00
KevinHuSh
038822f3bd
make cites in conversation API configurable (#576)
### What problem does this PR solve?

#566 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-28 11:56:17 +08:00
KevinHuSh
944776f207
fix bug about fetching file from minio (#574)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-28 09:57:40 +08:00
KevinHuSh
ab06f502d7
fix bug of file management (#565)
### What problem does this PR solve?

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-26 19:59:21 +08:00