AI/ragflow
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/infiniflow/ragflow.git synced 2025-08-13 22:55:58 +08:00
Code Issues Packages Projects Releases Wiki Activity

enable 3 char words to finegrind tokenize (#2210)

Browse Source 
### What problem does this PR solve?


### Type of change


- [x] Performance Improvement
This commit is contained in:
Kevin Hu 2024-09-03 13:37:32 +08:00 committed by GitHub
parent 21179a9be9
commit 6d232f1bdb
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
rag/nlp/query.py
View File

@ -83,7 +83,7 @@ class EsQueryer:
), tks ), tks
def need_fine_grained_tokenize(tk): def need_fine_grained_tokenize(tk):
if len(tk) < 4: if len(tk) < 3:
return False return False
if re.match(r"[0-9a-z\.\+#_\*-]+$", tk): if re.match(r"[0-9a-z\.\+#_\*-]+$", tk):
return False return False