AI/ragflow
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/infiniflow/ragflow.git synced 2025-08-12 06:38:59 +08:00
Code Issues Packages Projects Releases Wiki Activity

boost paralal of graphrag (#1924)

Browse Source 
### What problem does this PR solve?


### Type of change

- [x] Performance Improvement
This commit is contained in:
Kevin Hu 2024-08-13 11:21:30 +08:00 committed by GitHub
parent b75115264d
commit da2d8b8267
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
graphrag/index.py
View File

@ -61,11 +61,11 @@ def build_knowlege_graph_chunks(tenant_id: str, chunks: List[str], callback, ent
assert left_token_count > 0, f"The LLM context length({llm_bdl.max_length}) is smaller than prompt({ext.prompt_token_count})"
BATCH_SIZE=1
BATCH_SIZE=4
texts, graphs = [], []
cnt = 0
threads = []
exe = ThreadPoolExecutor(max_workers=12)
exe = ThreadPoolExecutor(max_workers=50)
for i in range(len(chunks)):
tkn_cnt = num_tokens_from_string(chunks[i])
if cnt+tkn_cnt >= left_token_count and texts: