654 Commits

Author SHA1 Message Date
Kevin Hu
b89f7c69ad
Fix image_id absence issue (#3735)
### What problem does this PR solve?

#3731

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-29 10:37:09 +08:00
kunkeji
fcdc6ad085
Fix the issue where the agent interface cannot call the context (#3725)
### What problem does this PR solve?

Fix the context of the agent interface call to the context during web
testing, and change it to the context record of user chat
Remove duplicate messages and add them, which can be viewed in the
messages section of database api_4_comversation

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2024-11-29 10:36:48 +08:00
Zhichang Yu
112ef42a19
Ensure thumbnail be smaller than 64K (#3722)
### What problem does this PR solve?

Ensure thumbnail be smaller than 64K. Close #1443
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-11-28 19:15:31 +08:00
Kevin Hu
91f1814a87
Fix error response (#3719)
### What problem does this PR solve?



### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2024-11-28 18:56:10 +08:00
Jin Hai
9fcad0500d
Add more web test cases (#3702)
### What problem does this PR solve?

Test cases about dataset

### Type of change

- [x] Other (please describe): test cases

---------

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-28 15:46:35 +08:00
balibabu
ec560cc99d
Feat: Scrolling knowledge base list and set the number of entries per page to 30 #3695 (#3712)
### What problem does this PR solve?

Feat: Scrolling knowledge base list and set the number of entries per
page to 30 #3695

### Type of change


- [x] New Feature (non-breaking change which adds functionality)
2024-11-28 15:25:38 +08:00
Zhichang Yu
bc701d7b4c
Edit chunk shall update instead of insert it (#3709)
### What problem does this PR solve?

Edit chunk shall update instead of insert it. Close #3679 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-28 13:00:38 +08:00
Kevin Hu
9f57534843
Revert "Feat: Scrolling knowledge base list #3695" (#3708)
Reverts infiniflow/ragflow#3703
2024-11-28 11:44:23 +08:00
balibabu
52b3492b18
Feat: Scrolling knowledge base list #3695 (#3703)
### What problem does this PR solve?

Feat: Scrolling knowledge base list #3695

### Type of change


- [x] New Feature (non-breaking change which adds functionality)
2024-11-28 10:51:30 +08:00
Zhichang Yu
2249d5d413
Always open text file for write with UTF-8 (#3688)
### What problem does this PR solve?

Always open text file for write with UTF-8. Close #932 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-27 16:24:16 +08:00
Jin Hai
b6f3f15f0b
Fix KB list bugs and add web api test (#3649)
### What problem does this PR solve?

1. Read KB list path parameter, page_number and page_size, which type
isn't int
2. Add cases on create / list / delete datasets.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Test cases

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-26 18:21:15 +08:00
Zhichang Yu
5337cad7e4
Check model id when set dialog. Close #849 (#3655)
### What problem does this PR solve?

Check model id when set dialog. Close #849

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-26 16:32:46 +08:00
liuhua
5c59651bda
Fix the bug causing garbled text (#3640)
### What problem does this PR solve?

Fix the bug causing garbled text #3613

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-26 12:06:56 +08:00
liuhua
d8b150a34c
Add support for folder deletion (#3635)
### What problem does this PR solve?

Add support for folder deletion.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-25 16:59:43 +08:00
liuhua
ce6b4c0e05
Fix a bug in completions (#3632)
### What problem does this PR solve?

Fix a bug in completions

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-25 16:21:28 +08:00
liuhua
86e48179a1
Fix bugs in api (#3624)
### What problem does this PR solve?

#3488

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-11-25 15:25:48 +08:00
Zhichang Yu
08ead81dde
Bump infinity to v0.5.0-dev5 (#3520)
### What problem does this PR solve?

Bump infinity to v0.5.0-dev5

### Type of change

- [x] Refactoring
2024-11-25 11:53:58 +08:00
Zhichang Yu
646ac1f2b4
Improved image build instructions (#3580)
### What problem does this PR solve?

Improved arm64 image build instructions

### Type of change

- [x] Documentation Update
- [x] Refactoring
2024-11-22 20:24:32 +08:00
Kevin Hu
cc219ff648
Fix agent session API (#3589)
### What problem does this PR solve?

#3585
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-22 16:19:00 +08:00
Kevin Hu
d56f52eef8
Fix agent api invokation issue (#3581)
### What problem does this PR solve?

#3570 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-22 12:37:27 +08:00
Kevin Hu
9f3141804f
Fix chunk enable/disable issue (#3579)
### What problem does this PR solve?

#3576

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-22 12:25:42 +08:00
Kevin Hu
ee50f78d99
Add component 'Template' (#3562)
### What problem does this PR solve?

#3560

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-11-21 18:26:22 +08:00
Kevin Hu
30f111edb3
Fixs for translation agent (#3557)
### What problem does this PR solve?

#3556 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
2024-11-21 16:22:25 +08:00
liuhua
c7c8b3812f
Add test for document (#3548)
### What problem does this PR solve?

Add test for document

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-21 12:02:26 +08:00
Kevin Hu
d9c882399d
Ensure LIGHTEN work (#3542)
### What problem does this PR solve?
#3531

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-21 09:56:28 +08:00
Jin Hai
8930bfcff8 Fix bugs (#3535)
### What problem does this PR solve?

1. system monitor icon and text missing
2. Team knowledge base can't be search

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-20 20:55:42 +08:00
Kevin Hu
362db857d0 New: a new interpretor based on Andrew Ng theory. (#3532)
### What problem does this PR solve?

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-11-20 20:55:22 +08:00
Kevin Hu
541272eb99 Fix: authorization issue (#3530)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-20 20:55:10 +08:00
Zhichang Yu
5b44b99cfd Removed beartype (#3528)
### What problem does this PR solve?

The beartype configuration of
main(64f50992e0fc4dce73e79f8b951a02e31cb2d638) is:
```
from beartype import BeartypeConf
from beartype.claw import beartype_all  # <-- you didn't sign up for this
beartype_all(conf=BeartypeConf(violation_type=UserWarning))    # <-- emit warnings from all code
```

ragflow_server failed at a third-party package:

```
(ragflow-py3.10) zhichyu@iris:~/github.com/infiniflow/ragflow$ rm -rf logs/* && bash docker/launch_backend_service.sh 
Starting task_executor.py for task 0 (Attempt 1)
Starting ragflow_server.py (Attempt 1)
Traceback (most recent call last):
  File "/home/zhichyu/github.com/infiniflow/ragflow/api/ragflow_server.py", line 22, in <module>
    from api.utils.log_utils import initRootLogger
  File "/home/zhichyu/github.com/infiniflow/ragflow/api/utils/__init__.py", line 25, in <module>
    import requests
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/urllib3/__init__.py", line 15, in <module>
    from ._base_connection import _TYPE_BODY
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/urllib3/_base_connection.py", line 5, in <module>
    from .util.connection import _TYPE_SOCKET_OPTIONS
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .connection import is_connection_dropped
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 7, in <module>
    from .timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/urllib3/util/timeout.py", line 20, in <module>
    _DEFAULT_TIMEOUT: Final[_TYPE_DEFAULT] = _TYPE_DEFAULT.token
NameError: name 'Final' is not defined
Traceback (most recent call last):
  File "/home/zhichyu/github.com/infiniflow/ragflow/rag/svr/task_executor.py", line 22, in <module>
    from api.utils.log_utils import initRootLogger
  File "/home/zhichyu/github.com/infiniflow/ragflow/api/utils/__init__.py", line 25, in <module>
    import requests
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/urllib3/__init__.py", line 15, in <module>
    from ._base_connection import _TYPE_BODY
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/urllib3/_base_connection.py", line 5, in <module>
    from .util.connection import _TYPE_SOCKET_OPTIONS
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/urllib3/util/__init__.py", line 4, in <module>
    from .connection import is_connection_dropped
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 7, in <module>
    from .timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT
  File "/home/zhichyu/github.com/infiniflow/ragflow/.venv/lib/python3.10/site-packages/urllib3/util/timeout.py", line 20, in <module>
    _DEFAULT_TIMEOUT: Final[_TYPE_DEFAULT] = _TYPE_DEFAULT.token
NameError: name 'Final' is not defined
```

This third-package is out of our control. I have to remove beartype
entirely.


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-20 20:54:57 +08:00
Zhichang Yu
6be7901df2 Warning instead of exception on type mismatch (#3523)
### What problem does this PR solve?

Warning instead of exception on type mismatch.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-20 20:54:42 +08:00
Kevin Hu
d02a2b131a Fix: potential risk (#3515)
### What problem does this PR solve?


### Type of change

- [x] Refactoring
2024-11-20 20:53:58 +08:00
Kevin Hu
e559cebcdc fix: keyerror issue (#3512)
### What problem does this PR solve?

#3511

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-20 20:53:20 +08:00
Kevin Hu
361cff34fc
add input variables to begin component (#3498)
### What problem does this PR solve?

#3355 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-11-19 18:41:48 +08:00
liuhua
d42362deb6
Add api for sessions and add max_tokens for tenant_llm (#3472)
### What problem does this PR solve?

Add api for sessions and add max_tokens for tenant_llm

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-19 14:51:33 +08:00
Zhichang Yu
dec9b3e540
Fix logs. Use dict.pop instead of del. Close #3473 (#3484)
### What problem does this PR solve?

Fix logs. Use dict.pop instead of del.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-19 14:15:25 +08:00
Zhichang Yu
4413683898
Introduced beartype (#3460)
### What problem does this PR solve?

Introduced [beartype](https://github.com/beartype/beartype) for runtime
type-checking.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-11-18 17:38:17 +08:00
shizzgar
4b3eeaa6ef
Added LocalAI support for rerank models (#3446)
### What problem does this PR solve?

Hi there!
LocalAI added support of rerank models
https://localai.io/features/reranker/

I've implemented LocalAIRerank class (typically copied it from
OpenAI_APIRerank class).
Also, LocalAI model response with 500 error code if len of "documents"
is less than 2 in similarity check.
So I've added the second "document" on RERANK model connection check in
`api/apps/llm_app.py`.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-11-18 12:05:52 +08:00
Jin Hai
70cd5c1599
Remove unused code (#3448)
### What problem does this PR solve?

1. Remove unused code.
2. Move some codes from settings to constants

### Type of change

- [x] Refactoring

---------

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-18 12:05:38 +08:00
Zhichang Yu
f9643adc43
Adjusted heartbeat format (#3459)
### What problem does this PR solve?

Adjusted heartbeat format

### Type of change

- [x] Refactoring
2024-11-18 12:03:28 +08:00
Kevin Hu
a1d01a1b2f
enlarge the default token length of RAPTOR summarization (#3454)
### What problem does this PR solve?

#3426

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-11-18 10:15:26 +08:00
Zhichang Yu
4ed5ca2666
handle_task catch all exception (#3441)
### What problem does this PR solve?

handle_task catch all exception
Report heartbeats

### Type of change

- [x] Refactoring
2024-11-15 18:51:09 +08:00
Jin Hai
1e90a1bf36
Move settings initialization after module init phase (#3438)
### What problem does this PR solve?

1. Module init won't connect database any more.
2. Config in settings need to be used with settings.CONFIG_NAME

### Type of change

- [x] Refactoring

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-15 17:30:56 +08:00
Kevin Hu
cb3b9d7ada
refine the message of queuing a task (#3437)
### What problem does this PR solve?



### Type of change

- [x] Refactoring
2024-11-15 15:59:54 +08:00
Kevin Hu
ca9e97d2f2
Enlarge the term weight difference (#3435)
### What problem does this PR solve?


### Type of change

- [x] Performance Improvement
2024-11-15 15:41:50 +08:00
Zhichang Yu
a854bc22d1
Rework task executor heartbeat (#3430)
### What problem does this PR solve?

Rework task executor heartbeat, and print in console.

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2024-11-15 14:43:55 +08:00
Jin Hai
47abfc32d4
Remove unused settings (#3427)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Refactoring

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-15 13:18:16 +08:00
Jin Hai
6878d23a57
Print configs when startup RAGFlow server (#3414)
### What problem does this PR solve?

Print configs at the RAGFlow startup phase.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

```
2024-11-14 21:27:53,090 INFO     962231 Current configs, from /home/weilongma/Documents/development/ragflow/conf/service_conf.yaml:
2024-11-14 21:27:53,090 INFO     962231 ragflow: {'host': '0.0.0.0', 'http_port': 9380}
2024-11-14 21:27:53,090 INFO     962231 mysql: {'name': 'rag_flow', 'user': 'root', 'password': 'infini_rag_flow', 'host': 'mysql', 'port': 5455, 'max_connections': 100, 'stale_timeout': 30}
2024-11-14 21:27:53,090 INFO     962231 minio: {'user': 'rag_flow', 'password': 'infini_rag_flow', 'host': 'minio:9000'}
2024-11-14 21:27:53,090 INFO     962231 es: {'hosts': 'http://es01:1200', 'username': 'elastic', 'password': 'infini_rag_flow'}
2024-11-14 21:27:53,090 INFO     962231 redis: {'db': 1, 'password': 'infini_rag_flow', 'host': 'redis:6379'}
```

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-15 09:29:40 +08:00
Jin Hai
e4c4fdabbd
Update version display on web UI (#3405)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Refactoring

---------

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-14 17:51:21 +08:00
Zhichang Yu
30f6421760
Use consistent log file names, introduced initLogger (#3403)
### What problem does this PR solve?

Use consistent log file names, introduced initLogger

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2024-11-14 17:13:48 +08:00
Kevin Hu
ab4384e011
Updates on parsing progress, including more detailed time cost inform… (#3402)
### What problem does this PR solve?

#3401 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-11-14 16:28:10 +08:00