AI/ragflow
1
0
Fork 0
mirror of https://git.mirrors.martin98.com/https://github.com/infiniflow/ragflow.git synced 2025-08-14 23:16:00 +08:00
Code Issues Packages Projects Releases Wiki Activity

Fix: fix special delimiter parsing issue (#5448)

Browse Source 
### What problem does this PR solve?

Fix special delimiter parsing issue #5382 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
This commit is contained in:
Yongteng Lei 2025-02-27 18:33:55 +08:00 committed by GitHub
parent 244cf49ba4
commit 83d0949498
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
deepdoc/parser/txt_parser.py
View File

@ -51,11 +51,13 @@ class RAGFlowTxtParser:
s = t
if s < len(delimiter):
dels.extend(list(delimiter[s:]))
dels = [re.escape(d) for d in delimiter if d]
dels = [re.escape(d) for d in dels if d]
dels = [d for d in dels if d]
dels = "|".join(dels)
secs = re.split(r"(%s)" % dels, txt)
for sec in secs:
if re.match(f"^{dels}$", sec):
continue
add_chunk(sec)
return [[c, ""] for c in cks]