81 Commits

Author SHA1 Message Date
Jin Hai
8f070c3d56
Fix 'SCORE' not found bug (#4178)
### What problem does this PR solve?

As title

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-12-23 14:50:12 +08:00
Kevin Hu
31d67c850e
Fetch chunk by batches. (#4177)
### What problem does this PR solve?

#4173

### Type of change

- [x] Performance Improvement
2024-12-23 12:12:15 +08:00
Kevin Hu
2cbe064080
Add Llama3.3 (#4174)
### What problem does this PR solve?

#4168

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-23 11:18:01 +08:00
Kenny Dizi
f13f503952
Use s3 configuration from settings module (#4167)
### What problem does this PR solve?

Fix the issue when retrieving AWS credentials from the S3 configuration
from the settings module instead of getting from the environment
variables.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-12-23 10:22:45 +08:00
Zhichang Yu
85083ad400
Validate returned chunk at list_chunks and add_chunk (#4153)
### What problem does this PR solve?

Validate returned chunk at list_chunks and add_chunk

### Type of change

- [x] Refactoring
2024-12-20 22:55:45 +08:00
Zhichang Yu
a0dc9e1bdf
Fix position_int on infinity (#4144)
### What problem does this PR solve?

Fix position_int on infinity

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-20 11:30:33 +08:00
Zhichang Yu
57c99dd811
Fixed infinity exception SCORE() / SCORE_FACTORS() requires Fusion or MATCH TEXT or MATCH TENSOR (#4110)
### What problem does this PR solve?

Fixed infinity exception SCORE() / SCORE_FACTORS() requires Fusion or
MATCH TEXT or MATCH TENSOR. Close #4109

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-19 13:49:36 +08:00
Zhichang Yu
bcccaccc2b
Added pagerank support to infinity (#4059)
### What problem does this PR solve?

Added pagerank support to infinity

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-17 15:45:01 +08:00
Zhichang Yu
0bca46ac3a
Migrate infinity at startup (#3858)
### What problem does this PR solve?

Migrate infinity at startup

#3809
https://github.com/infiniflow/infinity/issues/2321

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-13 13:43:56 +08:00
Zhichang Yu
3d76f10a91
Fixed retrieval TypeError: unhashable type: 'list' (#3966)
### What problem does this PR solve?

Fixed retrieval TypeError: unhashable type: 'list'

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-10 18:28:56 +08:00
Zhichang Yu
03f00c9e6f
Rename page_num_list, top_list, position_list (#3940)
### What problem does this PR solve?

Rename page_num_list, top_list, position_list to page_num_int, top_int,
position_int

### Type of change

- [x] Refactoring
2024-12-10 16:32:58 +08:00
Zhichang Yu
0d68a6cd1b
Fix errors detected by Ruff (#3918)
### What problem does this PR solve?

Fix errors detected by Ruff

### Type of change

- [x] Refactoring
2024-12-08 14:21:12 +08:00
Kevin Hu
74b28ef1b0
Add pagerank to KB. (#3809)
### What problem does this PR solve?

#3794

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-03 14:30:35 +08:00
Zhichang Yu
c5f13629af
Set Log level by env (#3798)
### What problem does this PR solve?

Set Log level by env

### Type of change

- [x] Refactoring
2024-12-02 17:24:39 +08:00
Jin Hai
d00297a763
Fix chunk creation using Infinity (#3763)
### What problem does this PR solve?

1. Store error type in Infinity
2. position list value read from Infinity isn't correct.

Fix issue: #3729

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-30 00:10:14 +08:00
Zhichang Yu
80af3cc2d4
Don't log exception if object doesn't exist (#3724)
### What problem does this PR solve?

Don't log exception if object doesn't exist. Close #1483

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-28 19:37:01 +08:00
Jin Hai
9fcad0500d
Add more web test cases (#3702)
### What problem does this PR solve?

Test cases about dataset

### Type of change

- [x] Other (please describe): test cases

---------

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-28 15:46:35 +08:00
Zhichang Yu
bc701d7b4c
Edit chunk shall update instead of insert it (#3709)
### What problem does this PR solve?

Edit chunk shall update instead of insert it. Close #3679 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-28 13:00:38 +08:00
Zhichang Yu
90ffcb4ddb
Fix graphrag + infinity bugs (#3681)
### What problem does this PR solve?

Fix graphrag + infinity bugs

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-27 12:45:43 +08:00
liuhua
7f48acb3fd
Fix enable/disable bug (#3662)
### What problem does this PR solve?

Fix enable/disable bug   #3628

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-11-27 09:37:11 +08:00
Zhichang Yu
7669fc8f52
Fix es get NotFoundError (#3659)
### What problem does this PR solve?

Fix es get NotFoundError

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-26 18:07:07 +08:00
Zhichang Yu
9348616659
Handle infinity empty response (#3627)
### What problem does this PR solve?

Handle infinity empty response. Close #3623
Show version in docker build log

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-25 14:09:42 +08:00
Zhichang Yu
08ead81dde
Bump infinity to v0.5.0-dev5 (#3520)
### What problem does this PR solve?

Bump infinity to v0.5.0-dev5

### Type of change

- [x] Refactoring
2024-11-25 11:53:58 +08:00
Kevin Hu
9f3141804f
Fix chunk enable/disable issue (#3579)
### What problem does this PR solve?

#3576

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-22 12:25:42 +08:00
Zhichang Yu
9541d7e7bc
Added TRACE_MALLOC_DELTA and TRACE_MALLOC_FULL (#3555)
### What problem does this PR solve?

Added TRACE_MALLOC_DELTA and TRACE_MALLOC_FULL to debug task_executor.py
heap. Relates to #3518

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-11-22 12:00:25 +08:00
Kevin Hu
ee50f78d99
Add component 'Template' (#3562)
### What problem does this PR solve?

#3560

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-11-21 18:26:22 +08:00
Kevin Hu
0ac6dc8f8c
Cut down the attempt times of ES (#3550)
### What problem does this PR solve?

#3541
### Type of change


- [x] Refactoring
- [x] Performance Improvement
2024-11-21 11:37:45 +08:00
Zhichang Yu
cad341e794 Added kb_id filter to knn. Fix #3458 (#3513)
### What problem does this PR solve?

Added kb_id filter to knn. Fix #3458

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-20 20:53:30 +08:00
Jin Hai
2044bb0039 Fix bugs (#3502)
### What problem does this PR solve?

1. Remove unused code
2. Fix type mismatch, in nlp search and infinity search interface
3. Fix chunk list, get all chunks of this user.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-20 20:52:23 +08:00
Zhichang Yu
0cd5b64c3b
Changed requirement to python 3.10 (#3496)
### What problem does this PR solve?

Changed requirement to python 3.10.
Changed image base to Ubuntu 22.04 since it contains python 3.10.

### Type of change

- [x] Refactoring
2024-11-19 18:25:04 +08:00
Jin Hai
883fafde72
Fix elasticsearch status display (#3487)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-19 14:40:58 +08:00
Zhichang Yu
dec9b3e540
Fix logs. Use dict.pop instead of del. Close #3473 (#3484)
### What problem does this PR solve?

Fix logs. Use dict.pop instead of del.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-19 14:15:25 +08:00
Zhichang Yu
4413683898
Introduced beartype (#3460)
### What problem does this PR solve?

Introduced [beartype](https://github.com/beartype/beartype) for runtime
type-checking.

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-11-18 17:38:17 +08:00
Zhichang Yu
4ed5ca2666
handle_task catch all exception (#3441)
### What problem does this PR solve?

handle_task catch all exception
Report heartbeats

### Type of change

- [x] Refactoring
2024-11-15 18:51:09 +08:00
Zhichang Yu
a854bc22d1
Rework task executor heartbeat (#3430)
### What problem does this PR solve?

Rework task executor heartbeat, and print in console.

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2024-11-15 14:43:55 +08:00
Jin Hai
996c94a8e7
Move clk100k_base tokenizer to docker image (#3411)
### What problem does this PR solve?

Move the tiktoken of cl100k_base into docker image

issue: #3338 

### Type of change

- [x] Refactoring

Signed-off-by: jinhai <haijin.chn@gmail.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-11-15 10:18:40 +08:00
Zhichang Yu
30f6421760
Use consistent log file names, introduced initLogger (#3403)
### What problem does this PR solve?

Use consistent log file names, introduced initLogger

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2024-11-14 17:13:48 +08:00
Kevin Hu
4caf932808
fix bug about fetching knowledge graph (#3394)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-14 12:29:15 +08:00
Zhichang Yu
9d395ab74e
Added doc for switching elasticsearch to infinity (#3370)
### What problem does this PR solve?

Added doc for switching elasticsearch to infinity

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
- [x] Documentation Update
2024-11-14 00:08:55 +08:00
Kevin Hu
ccf189cb7f
mv service_conf.yaml to conf/ and fix: add 'answer' as a parameter to 'generate' (#3379)
### What problem does this PR solve?
#3373

### Type of change

- [x] Refactoring
- [x] Bug fix
2024-11-13 15:56:40 +08:00
Zhichang Yu
a2a5631da4
Rework logging (#3358)
Unified all log files into one.

### What problem does this PR solve?

Unified all log files into one.

### Type of change

- [x] Refactoring
2024-11-12 17:35:13 +08:00
Zhichang Yu
f4c52371ab
Integration with Infinity (#2894)
### What problem does this PR solve?

Integration with Infinity

- Replaced ELASTICSEARCH with dataStoreConn
- Renamed deleteByQuery with delete
- Renamed bulk to upsertBulk
- getHighlight, getAggregation
- Fix KGSearch.search
- Moved Dealer.sql_retrieval to es_conn.py


### Type of change

- [x] Refactoring
2024-11-12 14:59:41 +08:00
Zhichang Yu
37d71dfa90
Replaced redis with Valkey (#3164)
### What problem does this PR solve?

Replaced redis with Valkey. Close #3070

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2024-11-02 20:05:12 +08:00
Kevin Hu
2d1fbefdb5
search between multiple indiices for team function (#3079)
### What problem does this PR solve?

#2834 
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-10-29 13:19:01 +08:00
Jake Stride
d92acdcf1d
Update azure_sas_conn.py - fixing container_url typo (#2740)
### What problem does this PR solve?

Fixes #2739

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-08 18:26:30 +08:00
yqkcn
34abcf7704
style: fix typo and format code (#2618)
### What problem does this PR solve?

- Fix typo
- Remove unused import
- Format code

### Type of change

- [x] Other (please describe): typo and format
2024-09-27 13:17:25 +08:00
Fachuan Bai
ddb8be9219
Web: Display the icon of the currently used storage. (#2504)
https://github.com/infiniflow/ragflow/issues/2503


### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Before:

<img width="611" alt="image"
src="https://github.com/user-attachments/assets/02a3a1ee-7bfb-4fe0-9b15-11ced69cc8a3">

After:

<img width="796" alt="image"
src="https://github.com/user-attachments/assets/371136af-8d16-47aa-909b-26609d3ad60e">

<img width="557" alt="image"
src="https://github.com/user-attachments/assets/9268362f-2b6a-4ea1-9fe7-659f7292e5e1">
2024-09-20 09:49:16 +08:00
Kevin Hu
9d4bb5767c
make highlight friendly to English (#2417)
### What problem does this PR solve?

#2415

### Type of change

- [x] Performance Improvement
2024-09-13 17:03:51 +08:00
Fachuan Bai
8dd3adc443
Storage: Support the s3, azure blob as the object storage of ragflow. (#2278)
### What problem does this PR solve?

issue: https://github.com/infiniflow/ragflow/issues/2277

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-09-09 09:41:14 +08:00
Kevin Hu
6c1f1a9f53
remove alter log (#2140)
### What problem does this PR solve?


### Type of change
- [x] Refactoring
2024-08-28 16:29:23 +08:00