451 Commits

Author SHA1 Message Date
Zhichang Yu
0bca46ac3a
Migrate infinity at startup (#3858)
### What problem does this PR solve?

Migrate infinity at startup

#3809
https://github.com/infiniflow/infinity/issues/2321

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-13 13:43:56 +08:00
Luo Pan
68d46b2a1e
Fix bug in hierarchical_merge function (#4006)
### What problem does this PR solve?

Fix hierarchical_merge function. From idx vs. actual value to actual
value vs. actual value.
Related issue #4003 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: luopan <luopan@example.com>
2024-12-13 08:50:58 +08:00
Zhichang Yu
c8b1a564aa
Replaced md5 with xxhash64 for chunk id (#4009)
### What problem does this PR solve?

Replaced md5 with xxhash64 for chunk id

### Type of change

- [x] Refactoring
2024-12-12 17:47:39 +08:00
Zhichang Yu
301f95837c
Try to reuse existing chunks (#3983)
### What problem does this PR solve?

Try to reuse existing chunks. Close #3793
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-12 16:38:03 +08:00
Kevin Hu
6d19294ddc
Support debug components. (#3994)
### What problem does this PR solve?

#3993

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-11 19:23:59 +08:00
Zhichang Yu
9a6d976252
Add back beartype (#3967)
### What problem does this PR solve?

Add back beartype

### Type of change

- [x] Refactoring
2024-12-10 18:43:43 +08:00
Zhichang Yu
3d76f10a91
Fixed retrieval TypeError: unhashable type: 'list' (#3966)
### What problem does this PR solve?

Fixed retrieval TypeError: unhashable type: 'list'

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-10 18:28:56 +08:00
Zhichang Yu
03f00c9e6f
Rename page_num_list, top_list, position_list (#3940)
### What problem does this PR solve?

Rename page_num_list, top_list, position_list to page_num_int, top_int,
position_int

### Type of change

- [x] Refactoring
2024-12-10 16:32:58 +08:00
so95
d5a322a352
Theme switch support (#3568)
### What problem does this PR solve?
- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Yingfeng <yingfeng.zhang@gmail.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2024-12-10 11:42:04 +08:00
Kevin Hu
927873bfa6
Fix syn error. (#3953)
### What problem does this PR solve?

Close #3696
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-10 10:54:54 +08:00
Zhichang Yu
7a6bf4326e
Fixed log not displaying (#3946)
### What problem does this PR solve?

Fixed log not displaying

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-10 09:36:59 +08:00
Kevin Hu
044afa83d1
Fix transformers dependencies for slim. (#3934)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-09 14:21:37 +08:00
Kevin Hu
215e9361ea
Fix field missing issue. (#3931)
### What problem does this PR solve?

#3905
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-09 13:20:58 +08:00
Zhichang Yu
0d68a6cd1b
Fix errors detected by Ruff (#3918)
### What problem does this PR solve?

Fix errors detected by Ruff

### Type of change

- [x] Refactoring
2024-12-08 14:21:12 +08:00
Kevin Hu
56f473b680
Feat: Add question parameter to edit chunk modal (#3875)
### What problem does this PR solve?

Close #3873

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-05 14:51:19 +08:00
Kevin Hu
593ffc4067
Fix HuggingFace model error. (#3870)
### What problem does this PR solve?

#3865

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-05 13:28:42 +08:00
Kevin Hu
78601ee1bd
Fix open AI compatible rerank issue. (#3866)
### What problem does this PR solve?
#3700
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-05 10:26:21 +08:00
Kevin Hu
1b817a5b4c
Refine synonym query. (#3855)
### What problem does this PR solve?

### Type of change

- [x] Performance Improvement
2024-12-04 17:20:12 +08:00
Kevin Hu
3f3469130b
Fix preview issue in file manager. (#3846)
### What problem does this PR solve?

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-04 11:53:23 +08:00
Jin Hai
6657ca7cde
Change default error message to English (#3838)
### What problem does this PR solve?

As title

### Type of change

- [x] Refactoring

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-12-04 09:34:49 +08:00
Jin Hai
821fdf02b4
Fix parsing JSON file error (#3829)
### What problem does this PR solve?

Close issue: #3828

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-12-03 19:02:03 +08:00
Zhichang Yu
92ab7ef659
Refactor embedding batch_size (#3825)
### What problem does this PR solve?

Refactor embedding batch_size. Close #3657

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
2024-12-03 16:22:39 +08:00
Kevin Hu
74b28ef1b0
Add pagerank to KB. (#3809)
### What problem does this PR solve?

#3794

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-03 14:30:35 +08:00
Kevin Hu
6a0583f5ad
Fix voyage embedding. (#3818)
### What problem does this PR solve?

#3816 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-03 09:33:54 +08:00
Zhichang Yu
c5f13629af
Set Log level by env (#3798)
### What problem does this PR solve?

Set Log level by env

### Type of change

- [x] Refactoring
2024-12-02 17:24:39 +08:00
Zhichang Yu
d19f059f34
Detect invalid response from api.siliconflow.cn (#3792)
### What problem does this PR solve?

Detect invalid response from api.siliconflow.cn. Close #2643

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-12-02 12:55:05 +08:00
devMls
59a5813f1b
add jina new models in jina connector (#3770)
### What problem does this PR solve?

add new models in jinna connector, to allow use models that support
multilingual models

### Type of change

- [X] Other (please describe): new connectors no breaking change
2024-12-02 10:06:39 +08:00
Jin Hai
08c1a5e1e8
Refactor parse progress (#3781)
### What problem does this PR solve?

Refactor parse file progress

### Type of change

- [x] Refactoring

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-12-01 22:28:00 +08:00
Jin Hai
ea84cc2e33
Update file parsing progress info (#3780)
### What problem does this PR solve?

Refine the file parsing progress info

### Type of change

- [x] Refactoring

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-12-01 17:03:00 +08:00
Jin Hai
e079656473
Update progress info and start welcome info (#3768)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Refactoring

---------

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-30 18:48:06 +08:00
Jin Hai
d00297a763
Fix chunk creation using Infinity (#3763)
### What problem does this PR solve?

1. Store error type in Infinity
2. position list value read from Infinity isn't correct.

Fix issue: #3729

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-30 00:10:14 +08:00
Kevin Hu
0f08b0f053
Weight up title and keywords for chunks in terms of retrieval (#3750)
### What problem does this PR solve?


### Type of change

- [x] Performance Improvement
2024-11-29 16:39:55 +08:00
Zhichang Yu
d94386e00a
Pass top_p to ollama (#3744)
### What problem does this PR solve?

Pass top_p to ollama. Close #1769

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-29 14:52:27 +08:00
Kevin Hu
27cd765d6f
Fix raptor issue (#3737)
### What problem does this PR solve?

#3732

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-29 11:55:41 +08:00
Zhichang Yu
80af3cc2d4
Don't log exception if object doesn't exist (#3724)
### What problem does this PR solve?

Don't log exception if object doesn't exist. Close #1483

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-28 19:37:01 +08:00
Kevin Hu
91f1814a87
Fix error response (#3719)
### What problem does this PR solve?



### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Jin Hai <haijin.chn@gmail.com>
2024-11-28 18:56:10 +08:00
Jin Hai
9fcad0500d
Add more web test cases (#3702)
### What problem does this PR solve?

Test cases about dataset

### Type of change

- [x] Other (please describe): test cases

---------

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-28 15:46:35 +08:00
Zhichang Yu
43e367f2ea
Detect shape error of embedding (#3710)
### What problem does this PR solve?

Detect shape error of embedding. Close #2997

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-28 14:10:22 +08:00
kuschzzp
e678819f70
Fix RGBA error (#3707)
### What problem does this PR solve?

**Passing cv_mdl.describe() is not an RGB converted image**

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-28 13:09:02 +08:00
Zhichang Yu
bc701d7b4c
Edit chunk shall update instead of insert it (#3709)
### What problem does this PR solve?

Edit chunk shall update instead of insert it. Close #3679 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-28 13:00:38 +08:00
Kevin Hu
57208d8e53
Fix batch size issue. (#3675)
### What problem does this PR solve?

#3657

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-27 18:06:43 +08:00
Zhichang Yu
2249d5d413
Always open text file for write with UTF-8 (#3688)
### What problem does this PR solve?

Always open text file for write with UTF-8. Close #932 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-27 16:24:16 +08:00
Zhichang Yu
90ffcb4ddb
Fix graphrag + infinity bugs (#3681)
### What problem does this PR solve?

Fix graphrag + infinity bugs

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-27 12:45:43 +08:00
liuhua
7f48acb3fd
Fix enable/disable bug (#3662)
### What problem does this PR solve?

Fix enable/disable bug   #3628

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-11-27 09:37:11 +08:00
liuhua
8b35776916
Fix a bug in VolcEngine (#3658)
### What problem does this PR solve?

Fix a bug in VolcEngine  #3553

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-27 09:30:49 +08:00
Zhichang Yu
7669fc8f52
Fix es get NotFoundError (#3659)
### What problem does this PR solve?

Fix es get NotFoundError

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-26 18:07:07 +08:00
Kevin Hu
0891a393d7
Let ThreadPool exit gracefully. (#3653)
### What problem does this PR solve?

#3646

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-26 16:31:07 +08:00
liuhua
5c59651bda
Fix the bug causing garbled text (#3640)
### What problem does this PR solve?

Fix the bug causing garbled text #3613

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-11-26 12:06:56 +08:00
Zhichang Yu
9348616659
Handle infinity empty response (#3627)
### What problem does this PR solve?

Handle infinity empty response. Close #3623
Show version in docker build log

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-25 14:09:42 +08:00
Zhichang Yu
08ead81dde
Bump infinity to v0.5.0-dev5 (#3520)
### What problem does this PR solve?

Bump infinity to v0.5.0-dev5

### Type of change

- [x] Refactoring
2024-11-25 11:53:58 +08:00