AI/ragflow
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/infiniflow/ragflow.git synced 2025-08-14 08:26:05 +08:00
Code Issues Packages Projects Releases Wiki Activity

let index be batchly. (#733)

Browse Source 
### What problem does this PR solve?

let index be batchly.

### Type of change


- [x] Refactoring
This commit is contained in:
KevinHuSh 2024-05-11 19:47:53 +08:00 committed by GitHub
parent 04a9e95161
commit ffe5737f7d
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 6 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
rag/svr/task_executor.py
View File

@ -285,7 +285,12 @@ def main():
init_kb(r) init_kb(r)
chunk_count = len(set([c["_id"] for c in cks])) chunk_count = len(set([c["_id"] for c in cks]))
st = timer() st = timer()
es_r = ELASTICSEARCH.bulk(cks, search.index_name(r["tenant_id"])) es_r = ""
for b in range(0, len(cks), 32):
es_r = ELASTICSEARCH.bulk(cks[b:b+32], search.index_name(r["tenant_id"]))
if b % 128 == 0:
callback(prog=0.8 + 0.1 * (b + 1) / len(cks), msg="")
cron_logger.info("Indexing elapsed({}): {}".format(r["name"], timer()-st)) cron_logger.info("Indexing elapsed({}): {}".format(r["name"], timer()-st))
if es_r: if es_r:
callback(-1, "Index failure!") callback(-1, "Index failure!")