52 Commits

Author SHA1 Message Date
Zhichang Yu
3411d0a2ce
Added cuda_is_available (#4725)
### What problem does this PR solve?

Added cuda_is_available

### Type of change

- [x] Refactoring
2025-02-05 18:01:23 +08:00
Zhichang Yu
e1526846da
Fixed GPU detection on CPU only environment (#4711)
### What problem does this PR solve?

Fixed GPU detection on CPU only environment. Close #4692

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-02-05 12:02:43 +08:00
Kevin Hu
1bff6b7333
Fix t_ocr.py for PNG image. (#4625)
### What problem does this PR solve?
#4586

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-24 11:47:27 +08:00
Zhichang Yu
4230402fbb
deepdoc use GPU if possible (#4618)
### What problem does this PR solve?

deepdoc use GPU if possible

### Type of change

- [x] Refactoring
2025-01-24 09:48:02 +08:00
Mathias Panzenböck
1a367664f1
Remove usage of eval() from postprocess.py (#4571)
Remove usage of `eval()` from postprocess.py

### What problem does this PR solve?

The use of `eval()` is a potential security risk. While the use of
`eval()` is guarded and thus not a security risk normally, `assert`s
aren't run if `-O` or `-OO` is passed to the interpreter, and as such
then the guard would not apply. In any case there is no reason to use
`eval()` here at all.

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Other (please describe):

Potential security fix if somehow the passed `modul_name` could be user
controlled.
2025-01-22 19:37:24 +08:00
Jin Hai
3894de895b
Update comments (#4569)
### What problem does this PR solve?

Add license statement.

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2025-01-21 20:52:28 +08:00
Mathias Panzenböck
75e1981e13
Remove use of eval() from recognizer.py (#4480)
`eval(op_type)` -> `getattr(operators, op_type)`

### What problem does this PR solve?

Using `eval()` can lead to code injections and is entirely unnecessary
here.

### Type of change

- [x] Other (please describe):

Best practice code improvement, preventing the possibility of code
injection.
2025-01-20 09:52:47 +08:00
Mathias Panzenböck
4f9f9405b8
Remove use of eval() from ocr.py (#4481)
`eval(op_name)` -> `getattr(operators, op_name)`

### What problem does this PR solve?

Using `eval()` can lead to code injections and is entirely unnecessary
here.

### Type of change

- [x] Other (please describe):

Best practice code improvement, preventing the possibility of code
injection.
2025-01-20 09:52:30 +08:00
Kevin Hu
c852a6dfbf
Accelerate titles' embeddings. (#4492)
### What problem does this PR solve?


### Type of change

- [x] Performance Improvement
2025-01-15 15:20:29 +08:00
Zhi-Qiang You
b7ce4e7e62
fix:t_recognizer TypeError: 'super' object is not callable (#4404)
### What problem does this PR solve?

[Bug]: layout recognizer failed for wrong boxes class type #4230
(https://github.com/infiniflow/ragflow/issues/4230)

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)

---------

Co-authored-by: youzhiqiang <zhiqiang.you@aminer.com>
Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
2025-01-08 10:59:35 +08:00
Kevin Hu
2e40c2a6f6
Fix t_recognizer issue. (#4387)
### What problem does this PR solve?

#4230

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-07 13:17:46 +08:00
Kevin Hu
983ec0666c
Fix param error. (#4355)
### What problem does this PR solve?

#4230

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-06 13:54:17 +08:00
Kevin Hu
59a78408be
Fix t_recognizer.py after model updating. (#4330)
### What problem does this PR solve?

#4230

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2025-01-02 17:00:11 +08:00
Kevin Hu
2cbe064080
Add Llama3.3 (#4174)
### What problem does this PR solve?

#4168

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-23 11:18:01 +08:00
Kevin Hu
ce1e855328
Upgrades Document Layout Analysis model. (#4054)
### What problem does this PR solve?

#4052

### Type of change

- [x] New Feature (non-breaking change which adds functionality)
2024-12-17 11:27:19 +08:00
Zhichang Yu
1254ecf445
Added static check at PR CI (#3921)
### What problem does this PR solve?

Added static check at PR CI

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
- [x] Refactoring
2024-12-08 21:23:51 +08:00
Zhichang Yu
0d68a6cd1b
Fix errors detected by Ruff (#3918)
### What problem does this PR solve?

Fix errors detected by Ruff

### Type of change

- [x] Refactoring
2024-12-08 14:21:12 +08:00
Zhichang Yu
bc701d7b4c
Edit chunk shall update instead of insert it (#3709)
### What problem does this PR solve?

Edit chunk shall update instead of insert it. Close #3679 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-28 13:00:38 +08:00
Zhichang Yu
2249d5d413
Always open text file for write with UTF-8 (#3688)
### What problem does this PR solve?

Always open text file for write with UTF-8. Close #932 

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-11-27 16:24:16 +08:00
Zhichang Yu
30f6421760
Use consistent log file names, introduced initLogger (#3403)
### What problem does this PR solve?

Use consistent log file names, introduced initLogger

### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2024-11-14 17:13:48 +08:00
Zhichang Yu
a2a5631da4
Rework logging (#3358)
Unified all log files into one.

### What problem does this PR solve?

Unified all log files into one.

### Type of change

- [x] Refactoring
2024-11-12 17:35:13 +08:00
Ikko Eltociear Ashimine
c552a02e7f
chore: update operators.py (#2724)
### What problem does this PR solve?
substract -> subtract
### Type of change

- [ ] Bug Fix (non-breaking change which fixes an issue)
- [ ] New Feature (non-breaking change which adds functionality)
- [ ] Documentation Update
- [x] Refactoring
- [ ] Performance Improvement
- [ ] Other (please describe):
2024-10-08 10:34:52 +08:00
Jin Hai
6b3a40be5c
Format file format from Windows/dos to Unix (#1949)
### What problem does this PR solve?

Related source file is in Windows/DOS format, they are format to Unix
format.

### Type of change

- [x] Refactoring

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-08-15 09:17:36 +08:00
H
c943517932
Fix pdfparser error (#1707)
### What problem does this PR solve?

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-07-25 18:54:36 +08:00
Kevin Hu
99adeabc85
remove dependency (#1536)
### What problem does this PR solve?

#702 
### Type of change
- [x] Refactoring
2024-07-16 16:30:17 +08:00
Jin Hai
cdea1d0a85
Update readme and add license (#1018)
### What problem does this PR solve?

- Update readme
- Add license

### Type of change

- [x] Documentation Update

---------

Signed-off-by: Jin Hai <haijin.chn@gmail.com>
2024-06-01 16:24:10 +08:00
KevinHuSh
46454362d7
fix raptor bugs (#928)
### What problem does this PR solve?

#922 
### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-27 11:01:20 +08:00
dashi6174
fe18627ebc
Fix some syntax errors, re not import (#904)
re not import

### What problem does this PR solve?

### Type of change

- [x] Bug Fix (non-breaking change which fixes an issue)
2024-05-23 19:38:13 +08:00
KevinHuSh
cab274f560
remove PyMuPDF (#618)
### What problem does this PR solve?
#613 

### Type of change


- [x] Other (please describe):
2024-04-30 12:38:09 +08:00
KevinHuSh
8c07992b6c
refine code (#595)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-28 19:13:33 +08:00
KevinHuSh
9d60a84958
refactor code (#583)
### What problem does this PR solve?

### Type of change

- [x] Refactoring
2024-04-28 13:19:54 +08:00
KevinHuSh
453c29170f
make sure the models will not be load twice (#422)
### What problem does this PR solve?

#381 
### Type of change

- [x] Refactoring
2024-04-18 09:37:23 +08:00
KevinHuSh
38e5737067
add base url for OpenAI (#166) 2024-03-28 19:15:16 +08:00
KevinHuSh
a5384446e3
let's load model from local (#163) 2024-03-28 16:10:47 +08:00
KevinHuSh
fd7fcb5baf
apply pep8 formalize (#155) 2024-03-27 11:33:46 +08:00
KevinHuSh
979b3a5b4b
support snapshot download from local (#153)
* support snapshot download from local

* let snapshot download from local
2024-03-27 09:53:42 +08:00
KevinHuSh
da21320b88
fix plainPdf bugs (#152) 2024-03-26 15:11:07 +08:00
KevinHuSh
71fe314955
refine page ranges (#147) 2024-03-25 13:11:57 +08:00
KevinHuSh
6c6b144de2
refine manual parser (#140) 2024-03-21 18:17:32 +08:00
KevinHuSh
9da671b951
refine manul parser (#131) 2024-03-19 12:26:04 +08:00
KevinHuSh
675a9f8d9a
add dockerfile for cuda envirement. Refine table search strategy, (#123) 2024-03-14 19:45:29 +08:00
KevinHuSh
bcb58b7e71
layout refine (#115) 2024-03-08 18:59:53 +08:00
KevinHuSh
8f86ab9f7f
refine pdf parser, add time zone to userinfo (#112) 2024-03-08 11:24:24 +08:00
KevinHuSh
2d7c9080f4
deal with stop reason being length problem (#109) 2024-03-07 16:12:01 +08:00
KevinHuSh
b89ac3c4be
chage tas execution logic (#103) 2024-03-06 19:16:31 +08:00
KevinHuSh
685b4d8a95
fix table desc bugs, add positions to chunks (#91) 2024-03-04 14:42:26 +08:00
KevinHuSh
8a726fb04b
solve task execution issues (#90) 2024-03-01 19:48:01 +08:00
KevinHuSh
0429107e80
fix user login issue (#85) 2024-02-29 14:03:07 +08:00
KevinHuSh
4568a4b2cb
refine admin initialization (#75) 2024-02-27 14:57:34 +08:00
KevinHuSh
d1c600d5d3
add ocr and recognizer demo, update README (#74) 2024-02-26 19:51:35 +08:00