41 Commits

Author SHA1 Message Date
Kevin Hu
dd0ebbea35
Light GraphRAG (#4585)
### What problem does this PR solve?

#4543

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2025-01-22 19:43:14 +08:00
Jin Hai
3894de895b
Update comments (#4569)
### What problem does this PR solve?

Add license statement.

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2025-01-21 20:52:28 +08:00
Kevin Hu
8fb18f37f6
Code refactor. (#4291)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-12-30 18:38:51 +08:00
Zhichang Yu
0d68a6cd1b
Fix errors detected by Ruff (#3918)
### What problem does this PR solve?

Fix errors detected by Ruff

### Type of change

- [x] Refactoring
2024-12-08 14:21:12 +08:00
Jin Hai
e079656473
Update progress info and start welcome info (#3768)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Refactoring

---------

Signed-off-by: jinhai <haijin.chn@gmail.com>
2024-11-30 18:48:06 +08:00
Michal Masrna
c4f2464935 fix: laws.py added missing import logging (#3501)
### What problem does this PR solve?

_Choosing Laws Chunk Method results in an error when parsing a document.
The error is caused by a missing import in the `laws.py` file._

```
Traceback (most recent call last):
  File "/ragflow/rag/svr/task_executor.py", line 445, in handle_task
    do_handle_task(task)
  File "/ragflow/rag/svr/task_executor.py", line 384, in do_handle_task
    cks = build(r)
          ^^^^^^^^
  File "/ragflow/rag/svr/task_executor.py", line 196, in build
    cks = chunker.chunk(row["name"], binary=binary, from_page=row["from_page"],
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ragflow/rag/app/laws.py", line 161, in chunk
    for txt, poss in pdf_parser(filename if not binary else binary,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ragflow/rag/app/laws.py", line 124, in __call__
    logging.debug("layouts:".format(
    ^^^^^^^
NameError: name 'logging' is not defined. Did you forget to import 'logging'

```

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [ ] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):

Co-authored-by: Michal Masrna <m.marna1@gmail.com>
2024-11-20 20:52:05 +08:00
Zhichang Yu
30f6421760
Use consistent log file names, introduced initLogger (#3403)
### What problem does this PR solve?

Use consistent log file names, introduced initLogger

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2024-11-14 17:13:48 +08:00
Zhichang Yu
a2a5631da4
Rework logging (#3358)
Unified all log files into one.

### What problem does this PR solve?

Unified all log files into one.

### Type of change

- [x] Refactoring
2024-11-12 17:35:13 +08:00
yqkcn
570ad420a8
remove unused import (#2679)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-09-30 16:59:39 +08:00
Kevin Hu
fc867cb959
rename get_txt to get_text (#2649)
### What problem does this PR solve?



### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-09-29 12:47:09 +08:00
yqkcn
aea553c3a8
Add get_txt function (#2639)
### What problem does this PR solve?

Add get_txt function to reduce duplicate code

### Type of change

- [x] Refactoring

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2024-09-29 10:29:56 +08:00
Jin Hai
6b3a40be5c
Format file format from Windows/dos to Unix (#1949)
### What problem does this PR solve?

Related source file is in Windows/DOS format, they are format to Unix
format.

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-08-15 09:17:36 +08:00
KevinHuSh
92e9320657
upgrade laws parser of docx (#1332)
### What problem does this PR solve?


### Type of change

- [x] Refactoring
2024-07-01 15:50:24 +08:00
Zhedong Cen
fc7cc1d36c
Optimize docx handle method in laws parser (#1302)
### What problem does this PR solve?

Optimize docx handle method in laws parser

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-06-28 17:42:59 +08:00
Zhedong Cen
8dd45459be
Add support for HTML file (#973)
### What problem does this PR solve?

Add support for HTML file

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-05-30 09:12:55 +08:00
KevinHuSh
7013d7f620
refine text decode (#657)
### What problem does this PR solve?
#651 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-07 12:25:47 +08:00
KevinHuSh
8c07992b6c
refine code (#595)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-28 19:13:33 +08:00
Jin Hai
f1c98aad6b
Update version info (#564)
### What problem does this PR solve?

_Briefly describe what this PR aims to solve. Include background context
that will help reviewers understand the purpose of the PR._

### Type of change

- [x] Documentation Update
- [x] Refactoring

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-04-26 20:07:26 +08:00
chrysanthemum-boy
72384b191d
Add .doc file parser. (#497)
### What problem does this PR solve?
Add `.doc` file parser, using tika.
```
pip install tika
```
```
from tika import parser
from io import BytesIO

def extract_text_from_doc_bytes(doc_bytes):
    file_like_object = BytesIO(doc_bytes)
    parsed = parser.from_buffer(file_like_object)
    return parsed["content"]
```
### Type of change

- [x] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: chrysanthemum-boy <fannc@qq.com>
2024-04-23 15:31:43 +08:00
KevinHuSh
0dfc8ddc0f
enlarge docker memory usage (#501)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-23 14:41:10 +08:00
KevinHuSh
a38e163035
remove doc from supported processing types (#488)
### What problem does this PR solve?
#474 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-04-22 15:46:09 +08:00
KevinHuSh
ed6081845a
Fit a lot of encodings for text file. (#458)
### What problem does this PR solve?

#384

### Type of change

- [x] Performance Improvement
2024-04-19 18:02:53 +08:00
KevinHuSh
f6c7204002
refine log format (#312)
### What problem does this PR solve?

Issue link:#264
### Type of change


- [x] Documentation Update
- [x] Refactoring
2024-04-11 10:13:43 +08:00
KevinHuSh
a0a480b708
continue add layout model for 'laws' (#292)
### What problem does this PR solve?

Issue link:#289

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-04-10 14:06:36 +08:00
KevinHuSh
243de6ac90
add a new model for 'Laws' (#290)
### What problem does this PR solve?

Issue link:#289
### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-04-10 11:59:00 +08:00
KevinHuSh
fd7fcb5baf
apply pep8 formalize (#155) 2024-03-27 11:33:46 +08:00
KevinHuSh
da21320b88
fix plainPdf bugs (#152) 2024-03-26 15:11:07 +08:00
KevinHuSh
f6aee7f230
add use layout or not option (#145)
* add use layout or not option

* trival
2024-03-22 19:21:09 +08:00
KevinHuSh
d7c362f237
adjust hierarchical_merge strategy (#100) 2024-03-06 09:09:16 +08:00
KevinHuSh
602038ac49
fix task cancling bug (#98) 2024-03-05 16:33:47 +08:00
KevinHuSh
8a57f2afd5
change callback strategy, add timezone to docker (#96) 2024-03-05 12:08:41 +08:00
KevinHuSh
685b4d8a95
fix table desc bugs, add positions to chunks (#91) 2024-03-04 14:42:26 +08:00
KevinHuSh
7fd1eca582
init README of deepdoc, add picture processer. (#71)
* init README of deepdoc, add picture processer.

* add resume parsing
2024-02-23 18:28:12 +08:00
KevinHuSh
cacd36c5e1
use onnx models, new deepdoc (#68) 2024-02-21 16:32:38 +08:00
KevinHuSh
a8294f2168 Refine resume parts and fix bugs in retrival using sql (#66) 2024-02-19 19:22:17 +08:00
KevinHuSh
407b2523b6 remove unused codes, seperate layout detection out as a new api. Add new rag methed 'table' (#55) 2024-02-05 18:08:17 +08:00
KevinHuSh
51482f3e2a Some document API refined. (#53)
Add naive chunking method to RAG
2024-02-02 19:21:37 +08:00
KevinHuSh
e6acaf6738 Add Q&A and Book, fix task running bugs (#50) 2024-02-01 18:53:56 +08:00
KevinHuSh
6224edcd1b Add task moduel, and pipline the task and every parser (#49) 2024-01-31 19:57:45 +08:00
KevinHuSh
96a1a44cb6 add paper & manual parser (#46) 2024-01-30 18:28:09 +08:00
KevinHuSh
072f9dd5bc Add app to rag module: presentaion & laws (#43) 2024-01-25 18:57:39 +08:00