263 Commits

Author SHA1 Message Date
黄腾
2da4e7aa46
add support for Tencent Cloud ASR (#2102)
### What problem does this PR solve?

add support for Tencent Cloud ASR

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-08-27 11:47:11 +08:00
黄腾
6b7c028578
add support for TTS model (#2095)
### What problem does this PR solve?

add support for TTS model
#1853

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-08-26 15:19:43 +08:00
Kevin Hu
e9202999cb
alter way of alerting info of empty task (#2094)
### What problem does this PR solve?

#2043

### Type of change

- [x] Refactoring
2024-08-26 13:45:26 +08:00
Jin Hai
a6d85c6c2f
Provide detailed error information. (#2093)
### What problem does this PR solve?

Most 'index failure' error is caused by ES.

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-08-26 13:36:00 +08:00
yungongzi
7539d142a9
VolcEngine SDK V3 adaptation (#2082)
1) Configuration interface update
2) Back-end adaptation API update
Note: The official no longer supports the Skylark1/2 series, and all
have been switched to the Doubao series


![image](https://github.com/user-attachments/assets/f6fd8782-0cdf-4c0b-ac8f-9eb130f667a5)

### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):

Co-authored-by: 海贼宅 <stu_xyx@163.com>
2024-08-26 13:34:29 +08:00
Kevin Hu
cc6a48b128
support monitoring task executor (#2069)
### What problem does this PR solve?
#1383

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-08-23 15:55:58 +08:00
Ran Tavory
19396998eb
Fix Bedrock system prompt (#2062)
### What problem does this PR solve?

Bugfix: usage of Bedrock models require the system prompt (for models
that support it) to be provided in the API in a different way, at least
that was my experience with it just today. This PR fixes it.


https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference.html

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-08-23 11:44:37 +08:00
黄腾
733219cc3f
add support for Baidu yiyan (#2049)
### What problem does this PR solve?

add support for Baidu yiyan

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-22 16:45:15 +08:00
黄腾
e4765ebe0c
add support for markdown file in one parse way (#2052)
### What problem does this PR solve?

#2021 add support for markdown file in one parse way

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-22 15:32:35 +08:00
Kevin Hu
642006c8e2
filter out + in es query (#2046)
### What problem does this PR solve?

#2028

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ]
2024-08-22 10:02:04 +08:00
Moonlit
59ba34e167
fix: Fix return type annotation in truncate function (#2044)
### What problem does this PR solve?

Fix return type annotation in truncate function

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-08-22 09:29:02 +08:00
Kevin Hu
0f95086813
add taskexecutor status check (#2038)
### What problem does this PR solve?


### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-08-21 17:48:00 +08:00
黄腾
be431449bd
add support for XunFei Spark (#2017)
### What problem does this PR solve?

#1853  add support for XunFei Spark

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-20 16:56:42 +08:00
黄腾
6f438e0a49
add support for Tencent Hunyuan (#2015)
### What problem does this PR solve?

#1853 

### Type of change


- [X] New Feature (non-breaking change which adds functionality)

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-20 15:27:13 +08:00
Kevin Hu
5efb3476f2
turn down es bulk size (#2013)
### What problem does this PR solve?


### Type of change

- [x] Performance Improvement
2024-08-20 09:59:17 +08:00
黄腾
79426fc41f
add support for Replicate (#1980)
### What problem does this PR solve?

#1853  add support for Replicate

### Type of change


- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-19 10:36:57 +08:00
黄腾
9729ca2aed
fix 01.ai url error (#1977)
### What problem does this PR solve?

#1976  fix 01.ai url error

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-16 16:35:40 +08:00
黄腾
b4ef50bdb5
fix OpenAI Embedding length error (#1972)
### What problem does this PR solve?
 
#1958   fix OpenAI Embedding length error

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-16 09:49:27 +08:00
黄腾
6acc46bc7b
fix add Bedrock llm error (#1952)
### What problem does this PR solve?

#1942  fix add Bedrock llm error

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-15 14:54:49 +08:00
黄腾
c59c1b603d
add support for 01.AI (#1951)
### What problem does this PR solve?

#1853  add support for 01.AI

### Type of change


- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-15 10:02:36 +08:00
Jin Hai
6b3a40be5c
Format file format from Windows/dos to Unix (#1949)
### What problem does this PR solve?

Related source file is in Windows/DOS format, they are format to Unix
format.

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-08-15 09:17:36 +08:00
Kevin Hu
78ed8fe9a5
add doc ids to chat (#1944)
### What problem does this PR solve?

### Type of change

- [x] Performance Improvement
2024-08-14 16:31:49 +08:00
Kevin Hu
da8802d010
refine error log while chunking (#1937)
### What problem does this PR solve?



### Type of change

- [x] Refactoring
2024-08-14 11:09:07 +08:00
Kevin Hu
d73a75506e
fix mind map bug (#1934)
### What problem does this PR solve?


### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-08-13 19:42:28 +08:00
黄腾
e013ac52af
add support for SILICONFLOW (#1926)
### What problem does this PR solve?

#1853 add support for SILICONFLOW

### Type of change


- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-13 16:09:10 +08:00
黄腾
68d1315079
add support for novita.ai (#1910)
### What problem does this PR solve?

#1853  add support for novita.ai

### Type of change


- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-12 17:26:26 +08:00
Kevin Hu
cafdee536f
add sql to naive parser (#1908)
### What problem does this PR solve?


### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [x] New Feature (non-breaking change which adds functionality)
2024-08-12 15:29:33 +08:00
黄腾
e9e39d57ce
add support for Upstage (#1902)
### What problem does this PR solve?

#1853  add support for Upstage

### Type of change


- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-12 11:06:25 +08:00
黄腾
94cb66ba80
add support for TogetherAI (#1890)
### What problem does this PR solve?

#1853 add support for TogetherAI

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-08-12 10:15:21 +08:00
黄腾
9a6dc89156
add support for PerfXCloud (#1883)
### What problem does this PR solve?

#1853  add support for PerfXCloud

### Type of change


- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-12 10:11:50 +08:00
H
fdd5b1b8cf
Fix token list , stats in api app.py (#1896)
### What problem does this PR solve?

#1842 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-08-09 19:03:01 +08:00
黄腾
411c645134
Enhance the robustness of the code (#1879)
### What problem does this PR solve?

Enhance the robustness of the code

### Type of change

- [x] Refactoring

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-09 10:18:08 +08:00
黄腾
aed1bbbcaa
add supprot for lepton (#1866)
### What problem does this PR solve?

add supprot for lepton
#1853

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-08 12:09:50 +08:00
Kung Quang
19ded65c66
Fix a "TypeError: expected string or buffer bug" in docx files extracted using Knowledge Graph.#1859 (#1865)
### What problem does this PR solve?

Fix a "TypeError: expected string or buffer bug" in docx files extracted
using Knowledge Graph. #1859
```
Traceback (most recent call last):
  File "//Users/XXX/ragflow/rag/svr/task_executor.py", line 149, in build
    cks = chunker.chunk(row["name"], binary=binary, from_page=row["from_page"],
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/XXX/ragflow/rag/app/knowledge_graph.py", line 18, in chunk
    chunks = build_knowlege_graph_chunks(tenant_id, sections, callback,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/XXX/ragflow/graphrag/index.py", line 87, in build_knowlege_graph_chunks
    tkn_cnt = num_tokens_from_string(chunks[i])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/XXX/github/ragflow/rag/utils/__init__.py", line 79, in num_tokens_from_string
    num_tokens = len(encoder.encode(string))
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/XXX/tiktoken/core.py", line 116, in encode
    if match := _special_token_regex(disallowed_special).search(text):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or buffer
```
This type is `Dict`
<img width="1689" alt="Pasted Graphic 3"
src="https://github.com/user-attachments/assets/e5ba5c45-df1d-4697-98c9-14365c839f20">
The correct type should be ` Str`
<img width="1725" alt="Pasted Graphic 2"
src="https://github.com/user-attachments/assets/e54d5e60-4ce4-4180-b394-24e485013534">

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2024-08-08 12:03:01 +08:00
黄腾
e34817c2a9
add support for cohere (#1849)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-07 18:40:51 +08:00
黄腾
60428c4ad2
fix LocalAI add bug (#1851)
### What problem does this PR solve?

#1848 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-07 18:10:42 +08:00
黄腾
ede733e130
add support for eml file parser (#1768)
### What problem does this PR solve?

add support for eml file parser
#1363

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-08-06 16:42:14 +08:00
黄腾
b67484e77d
add supprot for OpenAI-API-Compatible llm (#1787)
### What problem does this PR solve?

#1771  add supprot for OpenAI-API-Compatible 

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-06 16:20:21 +08:00
Wang
66e4113e0b
fix: align limitation with env (#1819)
### What problem does this PR solve?

try to process the large file will throw a File size exceeds error

### Type of change

- [x] Refactoring

Co-authored-by: Theta Wang (ncu) <chunshan.connect@gmail.com>
2024-08-06 16:04:51 +08:00
Kevin Hu
43199c45c3
refine loginfo about graprag progress (#1823)
### What problem does this PR solve?



### Type of change

- [x] Refactoring
2024-08-06 16:01:43 +08:00
Kevin Hu
fe797bcc66
be better chunks before graphrag (#1811)
### What problem does this PR solve?

#1594

### Type of change

- [x] Refactoring
2024-08-05 16:21:52 +08:00
Kevin Hu
2452c5624f
remove duplicated key in mind map (#1809)
### What problem does this PR solve?

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-08-05 15:57:33 +08:00
Kevin Hu
a5c03ccd4c
refine mindmap prompt (#1808)
### What problem does this PR solve?



### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-08-05 15:33:44 +08:00
H
d2213141e0
Fix graphrag callback (#1806)
### What problem does this PR solve?

#1800 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-08-05 14:44:54 +08:00
Kevin Hu
152072f900
Add graphrag (#1793)
### What problem does this PR solve?

#1594

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-08-02 18:51:14 +08:00
黄腾
4ba1ba973a
fix jina module not find bug (#1779)
### What problem does this PR solve?

fix jina module not find bug

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-08-01 19:52:56 +08:00
黄腾
ccbd4365be
refactor stepfun cv model (#1751)
### What problem does this PR solve?

refactor stepfun cv model

### Type of change

- [x] Refactoring

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-07-31 15:30:47 +08:00
jianyongli
9169643157
add step-1v-8k cv model (#1686)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: lijianyong <lijianyong@stepfun.com>
2024-07-30 16:57:27 +08:00
黄腾
5cff780ec4
lower openai version in requirements.txt (#1747)
### What problem does this PR solve?

lower openai version in requirements.txt

### Type of change

- [x] Refactoring

Co-authored-by: Zhedong Cen <cenzhedong2@126.com>
2024-07-30 16:55:59 +08:00
Wang Baoling
74ebc497c1
fix: ERROR: 'CompletionUsage' object has no attribute 'get' (#1736)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-07-30 15:12:16 +08:00