83 Commits

Author SHA1 Message Date
Zhichang Yu
85083ad400
Validate returned chunk at list_chunks and add_chunk (#4153)
### What problem does this PR solve?

Validate returned chunk at list_chunks and add_chunk

### Type of change

- [x] Refactoring
2024-12-20 22:55:45 +08:00
liuhua
35580af875
Update the component of the agent API with parameters. (#4131)
### What problem does this PR solve?

Update the  component of the agent API with parameters.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [x] Refactoring

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Co-authored-by: writinwaters <93570324+writinwaters@users.noreply.github.com>
2024-12-20 17:34:16 +08:00
liuhua
b35e811fe7
Add parameters for ask_chat and fix bugs in list_sessions (#4119)
### What problem does this PR solve?

Add parameters for ask_chat and fix bugs in list_sessions
#4105
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-12-19 17:24:26 +08:00
liuhua
1d65299791
Fix rerank_model bug in chat and markdown bug (#4061)
### What problem does this PR solve?

Fix rerank_model bug in chat and markdown bug
#4000
#3992
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-12-17 16:03:37 +08:00
liuhua
1ecb687c51
Fix bugs in agent api and update api document (#3996)
### What problem does this PR solve?

Fix bugs in agent api and update api document

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-12-13 10:25:52 +08:00
Zhichang Yu
c8b1a564aa
Replaced md5 with xxhash64 for chunk id (#4009)
### What problem does this PR solve?

Replaced md5 with xxhash64 for chunk id

### Type of change

- [x] Refactoring
2024-12-12 17:47:39 +08:00
Kevin Hu
e9b8c30a38
Support iframe chatbot. (#3961)
### What problem does this PR solve?

#3909

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-10 17:03:24 +08:00
Zhichang Yu
03f00c9e6f
Rename page_num_list, top_list, position_list (#3940)
### What problem does this PR solve?

Rename page_num_list, top_list, position_list to page_num_int, top_int,
position_int

### Type of change

- [x] Refactoring
2024-12-10 16:32:58 +08:00
Kevin Hu
255f4ccffc
Fix session API issues. (#3939)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-09 17:37:36 +08:00
Kevin Hu
3d735dca87
Add support to iframe chatbot (#3929)
### What problem does this PR solve?

#3909

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-09 12:38:04 +08:00
Zhichang Yu
1254ecf445
Added static check at PR CI (#3921)
### What problem does this PR solve?

Added static check at PR CI

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
2024-12-08 21:23:51 +08:00
Zhichang Yu
0d68a6cd1b
Fix errors detected by Ruff (#3918)
### What problem does this PR solve?

Fix errors detected by Ruff

### Type of change

- [x] Refactoring
2024-12-08 14:21:12 +08:00
Kevin Hu
56f473b680
Feat: Add question parameter to edit chunk modal (#3875)
### What problem does this PR solve?

Close #3873

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-05 14:51:19 +08:00
Kevin Hu
78601ee1bd
Fix open AI compatible rerank issue. (#3866)
### What problem does this PR solve?
#3700
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-05 10:26:21 +08:00
liuhua
87455d79e4
Add api for list agents and agent seesions (#3835)
### What problem does this PR solve?

Add api for list agents and agent seesions

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-12-03 19:03:16 +08:00
liuhua
95da6de9e1
Fix the agent reference bug and the session prologue (#3823)
### What problem does this PR solve?

Fix the agent reference bug and the session prologue
#3285 #3819
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-12-03 14:49:26 +08:00
liuhua
9654e64a0a
Fix the bug that the agent could not find the context (#3795)
### What problem does this PR solve?

Fix the bug that the agent could not find the context
#3682
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-12-02 19:05:18 +08:00
liuhua
3ee9ca749d
Fix the bug that prevented modifying dataset_ids (#3784)
### What problem does this PR solve?

Fix the bug that prevented modifying `dataset_ids`.
 #3772

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-12-02 11:40:20 +08:00
Kevin Hu
7058ac0041
Fix out of boundary. (#3786)
### What problem does this PR solve?

#3769
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-02 11:38:53 +08:00
Kevin Hu
b89f7c69ad
Fix image_id absence issue (#3735)
### What problem does this PR solve?

#3731

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-29 10:37:09 +08:00
kunkeji
fcdc6ad085
Fix the issue where the agent interface cannot call the context (#3725)
### What problem does this PR solve?

Fix the context of the agent interface call to the context during web
testing, and change it to the context record of user chat
Remove duplicate messages and add them, which can be viewed in the
messages section of database api_4_comversation

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2024-11-29 10:36:48 +08:00
liuhua
5c59651bda
Fix the bug causing garbled text (#3640)
### What problem does this PR solve?

Fix the bug causing garbled text #3613

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-26 12:06:56 +08:00
liuhua
d8b150a34c
Add support for folder deletion (#3635)
### What problem does this PR solve?

Add support for folder deletion.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-25 16:59:43 +08:00
liuhua
ce6b4c0e05
Fix a bug in completions (#3632)
### What problem does this PR solve?

Fix a bug in completions

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-25 16:21:28 +08:00
liuhua
86e48179a1
Fix bugs in api (#3624)
### What problem does this PR solve?

#3488

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-11-25 15:25:48 +08:00
Kevin Hu
cc219ff648
Fix agent session API (#3589)
### What problem does this PR solve?

#3585
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-22 16:19:00 +08:00
Kevin Hu
d56f52eef8
Fix agent api invokation issue (#3581)
### What problem does this PR solve?

#3570 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-22 12:37:27 +08:00
liuhua
c7c8b3812f
Add test for document (#3548)
### What problem does this PR solve?

Add test for document

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-21 12:02:26 +08:00
Kevin Hu
e559cebcdc fix: keyerror issue (#3512)
### What problem does this PR solve?

#3511

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-20 20:53:20 +08:00
liuhua
d42362deb6
Add api for sessions and add max_tokens for tenant_llm (#3472)
### What problem does this PR solve?

Add api for sessions and add max_tokens for tenant_llm

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-19 14:51:33 +08:00
Zhichang Yu
dec9b3e540
Fix logs. Use dict.pop instead of del. Close #3473 (#3484)
### What problem does this PR solve?

Fix logs. Use dict.pop instead of del.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-19 14:15:25 +08:00
Jin Hai
1e90a1bf36
Move settings initialization after module init phase (#3438)
### What problem does this PR solve?

1. Module init won't connect database any more.
2. Config in settings need to be used with settings.CONFIG_NAME

### Type of change

- [x] Refactoring

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-15 17:30:56 +08:00
Mohammed Tawileh
95d21e5d9f
fix: standardize property name from 'chat' to 'chat_id' (#3383)
### What problem does this PR solve?

This PR addresses the inconsistency in property naming within the
codebase by renaming the 'chat' property to 'chat_id' in the session.py
file. This change aims to align the naming convention with other parts
of the application that refer to chat identifiers as 'chat_id', thereby
improving code clarity and maintainability.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-14 15:49:03 +08:00
Kevin Hu
ccf189cb7f
mv service_conf.yaml to conf/ and fix: add 'answer' as a parameter to 'generate' (#3379)
### What problem does this PR solve?
#3373

### Type of change

- [x] Refactoring
- [x] Bug fix
2024-11-13 15:56:40 +08:00
Zhichang Yu
a2a5631da4
Rework logging (#3358)
Unified all log files into one.

### What problem does this PR solve?

Unified all log files into one.

### Type of change

- [x] Refactoring
2024-11-12 17:35:13 +08:00
liuhua
567a7563e7
Fix bugs in api and add examples (#3353)
### What problem does this PR solve?

Fix bugs in api.
Add simple examples for api.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-11-12 17:14:33 +08:00
Zhichang Yu
f4c52371ab
Integration with Infinity (#2894)
### What problem does this PR solve?

Integration with Infinity

- Replaced ELASTICSEARCH with dataStoreConn
- Renamed deleteByQuery with delete
- Renamed bulk to upsertBulk
- getHighlight, getAggregation
- Fix KGSearch.search
- Moved Dealer.sql_retrieval to es_conn.py


### Type of change

- [x] Refactoring
2024-11-12 14:59:41 +08:00
liuhua
48ab6d7a45
Update authorization for team (#3262)
### What problem does this PR solve?

Update authorization for team.
#3253 #3233
### Type of change

- [x] Refactoring

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-07 19:26:03 +08:00
Mohammed Tawileh
5038552ed9
fix: improve embedding model validation logic for dataset operations (#3235)
What problem does this PR solve?
When creating or updating datasets with custom embedding models (e.g.,
Ollama), the validation logic was too restrictive and prevented valid
models from being used. The previous implementation would reject valid
custom models if they weren't in the predefined list, even when they
existed in TenantLLMService.

Changes:
- Simplify and improve the embedding model validation flow in
create/update endpoints
- Check TenantLLMService for custom models before rejecting
- Make validation logic more consistent between create and update
operations

### What problem does this PR solve?

This fix allows users to successfully create and update datasets with
custom embedding models while maintaining proper validation checks.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Jin Hai <haijin.chn@gmail.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Co-authored-by: liuhua <10215101452@stu.ecnu.edu.cn>
2024-11-07 10:36:28 +08:00
liuhua
f3aaa0d453
Add sdk for Agent API (#3220)
### What problem does this PR solve?

Add sdk for Agent API

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-06 18:03:45 +08:00
liuhua
a7bf4ca8fc
Fix bugs in API (#3204)
### What problem does this PR solve?

Fix bugs in API

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-05 14:07:31 +08:00
Zhichang Yu
185c6a0c71
Unified API response json schema (#3170)
### What problem does this PR solve?

Unified API response json schema

### Type of change

- [x] Refactoring
2024-11-05 11:02:31 +08:00
liuhua
cbca7dfce6
fix bugs in test (#3196)
### What problem does this PR solve?

fix bugs in test

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-04 20:03:14 +08:00
Kevin Hu
8305632852
add agent completion API (#3192)
### What problem does this PR solve?

#3105

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-11-04 17:20:16 +08:00
Matej Horník
dd1146ec64
feat: docs for api endpoints to generate openapi specification (#3109)
### What problem does this PR solve?

**Added openapi specification for API routes. This creates swagger UI
similar to FastAPI to better use the API.**
Using python package `flasgger`

### Type of change
- [x] New Feature (non-breaking change which adds functionality)

Not all routes are included since this is a work in progress.

Docs can be accessed on: `{host}:{port}/apidocs`
2024-11-04 15:35:36 +08:00
liuhua
44ad9a6cd7
Add test for API (#3134)
### What problem does this PR solve?

Add test for API

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
2024-11-01 22:59:17 +08:00
liuhua
9aeb07d830
Add test for CI (#3114)
### What problem does this PR solve?

Add test for CI

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-10-31 14:07:23 +08:00
Yangong
3fa570f49b
fix: remove useless test code (#3122)
### What problem does this PR solve?

remove useless test code

### Type of change

- [X] Refactoring
2024-10-31 11:56:46 +08:00
liuhua
18dfa2900c
Fix bugs in API (#3103)
### What problem does this PR solve?

Fix bugs in API


- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-10-30 16:15:42 +08:00
knew
c5a3146a8c
fix: modify the response of metadata in Dify retrieval api (#3076)
### What problem does this PR solve?

Modify the response of metadata in Dify retrieval api

resolve   #2914

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-29 11:06:02 +08:00