344 Commits

Author SHA1 Message Date
Kevin Hu
2d1fbefdb5
search between multiple indiices for team function (#3079)
### What problem does this PR solve?

#2834 
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-10-29 13:19:01 +08:00
Kevin Hu
7e0148c058
fix local variable ans (#3077)
### What problem does this PR solve?
#3064

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-29 10:42:45 +08:00
Kevin Hu
f86826b7a0
refactor error message of qwen (#3074)
### What problem does this PR solve?
#3055

### Type of change
- [x] Refactoring
2024-10-29 10:08:08 +08:00
Kevin Hu
9457d20ef1
make gemini robust (#3012)
### What problem does this PR solve?

#3003

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-25 10:50:44 +08:00
Kevin Hu
7f81fc8f9b
refactor auto keywords and auto question (#2990)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-10-23 17:00:56 +08:00
Kevin Hu
89d5b2414e
fix SILICONFLOW rerank error (#2980)
### What problem does this PR solve?

#2977

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-23 10:12:39 +08:00
Yinquan WANG
445dce4363
[Bug]: unnecessary auto-increment calculations in the tokens statistics of the chat model (#2969)
### What problem does this PR solve?

the details is shown in
https://github.com/infiniflow/ragflow/issues/2968

### Type of change

- [X] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-10-22 16:26:04 +08:00
Kevin Hu
1fce6caf80
make titles in markdown not be splited with following content (#2971)
### What problem does this PR solve?

#2970 
### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
2024-10-22 15:25:23 +08:00
Kevin Hu
226bdd6e99
add auto keywords and auto-question (#2965)
### What problem does this PR solve?

#2687

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-10-22 13:12:49 +08:00
Yinquan WANG
5aa9d7787e
[Bug]: When use OpenAI chat model , raise ERROR: 'CompletionUsage' object has no attribute 'get' #2948 (#2949)
[Bug]: When use OpenAI chat model , raise ERROR: 'CompletionUsage'
object has no attribute 'get' #2948

### What problem does this PR solve?

the detail of this PR is shown at
https://github.com/infiniflow/ragflow/issues/2948

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-10-22 11:40:05 +08:00
Kevin Hu
b2524eec49
fix sequence2txt error and usage total token issue (#2961)
### What problem does this PR solve?

#1363

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-22 11:38:37 +08:00
chongchuanbing
ac26d09a59
Feature/feat1017 (#2872)
### What problem does this PR solve?

1. fix: mid map show error in knowledge graph, juse because
```@antv/g6```version changed
2. feat: concurrent threads configuration support in graph extractor
3. fix: used tokens update failed for tenant
4. feat: timeout configuration support for llm
5. fix: regex error in graph extractor
6. feat: qwen rerank(```gte-rerank```) support
7. fix: timeout deal in knowledge graph index process. Now chat by
stream output, also, it is configuratable.
8. feat: ```qwen-long``` model configuration

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: chongchuanbing <chongchuanbing@gmail.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-10-21 12:11:08 +08:00
Ziyu Huang
e5f7733b31
Resolves #2905 openai compatible model provider add llama.cpp rerank support (#2906)
### What problem does this PR solve?
Resolve #2905 



due to the in-consistent of token size, I make it safe to limit 500 in
code, since there is no config param to control

my llama.cpp run set -ub to 1024:

${llama_path}/bin/llama-server --host 0.0.0.0 --port 9901 -ub 1024 -ngl
99 -m $gguf_file --reranking "$@"





### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Here is my test Ragflow use llama.cpp

```
lot update_slots: id  0 | task 458 | prompt done, n_past = 416, n_tokens = 416
slot      release: id  0 | task 458 | stop processing: n_past = 416, truncated = 0
slot launch_slot_: id  0 | task 459 | processing task
slot update_slots: id  0 | task 459 | tokenizing prompt, len = 2
slot update_slots: id  0 | task 459 | prompt tokenized, n_ctx_slot = 8192, n_keep = 0, n_prompt_tokens = 111
slot update_slots: id  0 | task 459 | kv cache rm [0, end)
slot update_slots: id  0 | task 459 | prompt processing progress, n_past = 111, n_tokens = 111, progress = 1.000000
slot update_slots: id  0 | task 459 | prompt done, n_past = 111, n_tokens = 111
slot      release: id  0 | task 459 | stop processing: n_past = 111, truncated = 0
srv  update_slots: all slots are idle
request: POST /rerank 172.23.0.4 200

```
2024-10-21 10:06:29 +08:00
Kevin Hu
b9fa00f341
add API for tenant function (#2866)
### What problem does this PR solve?

feat: API access key management
https://github.com/infiniflow/ragflow/issues/2846
feat: Render markdown file with remark-loader
https://github.com/infiniflow/ragflow/issues/2846

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-10-16 16:10:24 +08:00
0000sir
4991107822
Fix keys of Xinference deployed models, especially has the same model name with public hosted models. (#2832)
### What problem does this PR solve?

Fix keys of Xinference deployed models, especially has the same model
name with public hosted models.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: 0000sir <0000sir@gmail.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-10-16 10:21:08 +08:00
Kevin Hu
b540d41cdc
let presentation do raptor (#2838)
### What problem does this PR solve?

#2837

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-10-15 10:11:09 +08:00
Kevin Hu
b164116277
refine token similarity (#2824)
### What problem does this PR solve?


### Type of change

- [x] Performance Improvement
2024-10-14 13:33:18 +08:00
Kevin Hu
190eea7097
trival (#2808)
### What problem does this PR solve?



### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-11 15:33:38 +08:00
Kevin Hu
2d1c83da59
fix LIGHTEN issue (#2806)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-11 15:01:27 +08:00
JobSmithManipulation
3f065c75da
support chat model in huggingface (#2802)
### What problem does this PR solve?

#2794

### Type of change
- [x] New Feature (non-breaking change which adds functionality)
2024-10-11 14:45:48 +08:00
Kevin Hu
5e7c1fb23a
reduce rerank batch size (#2801)
### What problem does this PR solve?

### Type of change


- [x] Performance Improvement
2024-10-11 11:29:19 +08:00
JobSmithManipulation
18f80743eb
support api-version and change default-model in adding azure-openai and openai (#2799)
### What problem does this PR solve?
#2701 #2712 #2749

### Type of change
-[x] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-10-11 11:26:42 +08:00
Kevin Hu
29f022c91c
fix bedrock issue (#2776)
### What problem does this PR solve?

#2722

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-10 09:13:35 +08:00
lidp
20e63f8ec4
Fix docx images (#2756)
### What problem does this PR solve?

#2755 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-09 19:37:32 +08:00
Sky Blue
2df15742fc
fix xinference add rerank model bug (#2758)
### What problem does this PR solve?

Fix xinference add rerank model bug,
https://github.com/infiniflow/ragflow/issues/2294#issue-2510788135

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-09 19:37:11 +08:00
Kevin Hu
7f44cf543a
move import positions (#2753)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-10-09 10:34:58 +08:00
JobSmithManipulation
16472eb3ea
solve knowledgegraph issue when calling gemini model (#2738)
### What problem does this PR solve?
#2720

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-08 18:27:04 +08:00
Jake Stride
d92acdcf1d
Update azure_sas_conn.py - fixing container_url typo (#2740)
### What problem does this PR solve?

Fixes #2739

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-10-08 18:26:30 +08:00
Kevin Hu
04ff9cda7c
expand rerank range (#2746)
### What problem does this PR solve?


### Type of change

- [x] Performance Improvement
2024-10-08 16:34:33 +08:00
JobSmithManipulation
a3ab5ba9ac
support sequence2txt and tts model in Xinference (#2696)
### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-10-08 10:43:18 +08:00
Omar Leonardo Sanchez Granados
34761fa4ca
Fix/bedrock issues (#2718)
### What problem does this PR solve?

Adding a Bedrock API key for Claude Sonnet was broken. I find the issue
came up when trying to test the LLM configuration, the system is a
required parameter in boto3.

As well, there were problems in Bedrock implementation for embeddings
when trying to encode queries.

### Type of change

- [X] Bug Fix (non-breaking change which fixes an issue)
2024-10-05 16:44:50 +08:00
Kevin Hu
0a7654c747
fix error in exception (#2694)
### What problem does this PR solve?
#2670

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-09-30 17:54:27 +08:00
yqkcn
570ad420a8
remove unused import (#2679)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-09-30 16:59:39 +08:00
Kevin Hu
fc867cb959
rename get_txt to get_text (#2649)
### What problem does this PR solve?



### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-09-29 12:47:09 +08:00
yqkcn
aea553c3a8
Add get_txt function (#2639)
### What problem does this PR solve?

Add get_txt function to reduce duplicate code

### Type of change

- [x] Refactoring

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-09-29 10:29:56 +08:00
yqkcn
e82e8fde13
Fix logger error (#2643)
### What problem does this PR solve?

Fix logger error: AttributeError: 'Logger' object has no attribute
'basicConfig'

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-09-29 09:49:59 +08:00
yqkcn
a44ed9626a
handle nits in task_executor (#2637)
### What problem does this PR solve?

- fix typo
- fix string format
- format import

### Type of change

- [x] Refactoring
2024-09-29 09:49:45 +08:00
JobSmithManipulation
96f56a3c43
add huggingface model (#2624)
### What problem does this PR solve?

#2469

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-09-27 19:15:38 +08:00
Zhichang Yu
1b2f66fc11
Added doc on dev-slim (#2627)
Added doc on dev-slim

### Type of change

- [x] Documentation Update
- [x] Refactoring

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-09-27 19:15:27 +08:00
yqkcn
34abcf7704
style: fix typo and format code (#2618)
### What problem does this PR solve?

- Fix typo
- Remove unused import
- Format code

### Type of change

- [x] Other (please describe): typo and format
2024-09-27 13:17:25 +08:00
liuhua
e11a74eed5
Update Yichat base_url (#2620)
### What problem does this PR solve?

Update Yichat base_url

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-09-27 12:55:58 +08:00
liuhua
b68d349bd6
Fix: renrank_model and pdf_parser bugs | Update: session API (#2601)
### What problem does this PR solve?

Fix: renrank_model and pdf_parser bugs | Update: session API
#2575
#2559
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-09-26 16:05:25 +08:00
Kevin Hu
dda1367ab2
make it lighten (#2577)
### What problem does this PR solve?

#2295

### Type of change

- [x] Refactoring
2024-09-25 13:38:40 +08:00
lidp
08d5637770
Fix tokenizer bug (#2573)
### What problem does this PR solve?

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-09-25 10:30:49 +08:00
Kevin Hu
7bb28ca2bd
add lighten control (#2567)
### What problem does this PR solve?

#2295

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2024-09-24 19:22:01 +08:00
Kevin Hu
ecf441c830
refine using rerank model (#2553)
### What problem does this PR solve?

#2552

### Type of change

- [x] Performance Improvement
2024-09-24 12:38:18 +08:00
liuhua
d9c2a128a5
SparkTTS (#2535)
### What problem does this PR solve?

SparkTTS

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: liuhua <10215101452@stu.ecun.edu.cn>
2024-09-24 12:15:12 +08:00
Kevin Hu
4a6a2a0f1b
refine xinference (#2521)
### What problem does this PR solve?

#1588

### Type of change

- [x] Refactoring
2024-09-20 18:37:01 +08:00
Kevin Hu
78856703c4
make excel parsing configurable (#2517)
### What problem does this PR solve?

#2516

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-09-20 15:33:38 +08:00
Kevin Hu
099c37ba95
rm key set in xinference (#2511)
### What problem does this PR solve?

#2492

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-09-20 10:55:52 +08:00