90 Commits

Author SHA1 Message Date
Jyong
84ac004772
py lint ()
Signed-off-by: -LAN- <laipz8200@outlook.com>
Co-authored-by: -LAN- <laipz8200@outlook.com>
2024-12-26 00:16:35 +08:00
Jyong
9231fdbf4c
Feat/support parent child chunk () 2024-12-25 19:49:07 +08:00
yihong
56e15d09a9
feat: mypy for all type check () 2024-12-24 18:38:51 +08:00
Bowen Liang
6c8e208ef3
chore: bump minimum supported Python version to 3.11 () 2024-11-24 13:28:46 +08:00
wy96f
94c9cadbd8
fix image files not deleted on indexing_estimate ()
Co-authored-by: root <root@localhost.localdomain>
2024-11-21 13:03:16 +08:00
Zane
14f3d44c37
refactor: improve handling of leading punctuation removal () 2024-11-18 21:32:33 +08:00
Bowen Liang
51db59622c
chore(lint): cleanup repeated cause exception in logging.exception replaced by helpful message () 2024-11-15 15:41:40 +08:00
pinsily
5f21d13572
fix: handle KeyError when accessing rules in CleanProcessor.clean () 2024-11-05 14:47:15 +08:00
Jyong
82033af097
clean un-allowed special charters when doing indexing estimate () 2024-11-01 15:09:22 +08:00
Bowen Liang
240b66d737
chore: avoid implicit optional in type annotations of method () 2024-10-09 14:36:43 +08:00
Bowen Liang
a1104ab97e
chore: refurish python code by applying Pylint linter rules () 2024-09-13 22:42:08 +08:00
Bowen Liang
292220c596
chore: apply pep8-naming rules for naming convention () 2024-09-11 16:40:52 +08:00
Bowen Liang
2cf1187b32
chore(api/core): apply ruff reformatting () 2024-09-10 17:00:20 +08:00
Nam Vu
2d7954c7da
Fix variable typo () 2024-09-08 13:14:11 +08:00
KVOJJJin
14af87527f
Feat:remove estimation of embedding cost ()
Co-authored-by: jyong <718720800@qq.com>
2024-09-04 14:41:47 +08:00
Jyong
35431bce0d
fix dataset_id and index_node_id idx missed in document_segments tabl… () 2024-08-27 10:25:24 +08:00
Shota Totsuka
430e100142
refactor: Add @staticmethod decorator in api/core () 2024-08-26 19:45:03 +08:00
wellCh4n
b7778de224
fix: document error message can not be cleared () 2024-08-20 19:30:57 +08:00
-LAN-
a98284b1ef
refactor(api): Switch to dify_config ()
Signed-off-by: -LAN- <laipz8200@outlook.com>
2024-07-30 11:15:26 +08:00
sino
d27e3ab99d
chore: remove unresolved reference () 2024-07-09 23:04:44 +08:00
Bowen Liang
dcb72e0067
chore: apply flake8-comprehensions Ruff rules to improve collection comprehensions ()
Co-authored-by: -LAN- <laipz8200@outlook.com>
2024-06-27 11:21:31 +08:00
Jyong
ba5f8afaa8
Feat/firecrawl data source ()
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
Co-authored-by: chenhe <guchenhe@gmail.com>
Co-authored-by: takatost <takatost@gmail.com>
2024-06-15 02:46:02 +08:00
Bowen Liang
f976740b57
improve: mordernizing validation by migrating pydantic from 1.x to 2.x () 2024-06-14 01:05:37 +08:00
takatost
d1dbbc1e33
feat: backend model load balancing support () 2024-06-05 00:13:04 +08:00
Jyong
b6631cd878
modify rerank and splitter code directory () 2024-06-04 20:07:25 +08:00
Jyong
233c4150d1
support images and tables extract from docx () 2024-05-23 18:05:23 +08:00
Rain Chen
c255a20d7c
allow to config max segmentation tokens length for RAG document using environment variable () 2024-05-20 13:20:27 +08:00
Bowen Liang
04ad46dd31
chore: skip unnecessary key checks prior to accessing a dictionary () 2024-05-19 18:30:45 +08:00
LIU HONGWEI
c227f3d985
feat: Deprecate datetime.utcnow() in favor of datetime.now(timezone.utc).replace(tzinfo=None) for better timezone handling () () 2024-04-12 16:22:24 +08:00
Jyong
33ea689861
fix detached instance error in keyword index create thread and fix question classifier node out of index error () 2024-04-09 14:34:51 +08:00
Jyong
283979fc46
fix keyword index error when storage source is S3 () 2024-04-09 01:42:58 +08:00
takatost
7753ba2d37
FEAT: NEW WORKFLOW ENGINE ()
Co-authored-by: Joel <iamjoel007@gmail.com>
Co-authored-by: Yeuoly <admin@srmxy.cn>
Co-authored-by: JzoNg <jzongcode@gmail.com>
Co-authored-by: StyleZhang <jasonapring2015@outlook.com>
Co-authored-by: jyong <jyong@dify.ai>
Co-authored-by: nite-knite <nkCoding@gmail.com>
Co-authored-by: jyong <718720800@qq.com>
2024-04-08 18:51:46 +08:00
Jyong
b0b0cc045f
add mutil-thread document embedding ()
Co-authored-by: jyong <jyong@dify.ai>
2024-03-28 17:02:35 +08:00
Jyong
6454e1d644
chunk-overlap None check ()
Co-authored-by: jyong <jyong@dify.ai>
2024-03-11 15:36:56 +08:00
Jyong
31070ffbca
fix qa index processor tenant id is None error ()
Co-authored-by: jyong <jyong@dify.ai>
2024-03-06 16:46:08 +08:00
Charlie.Wei
fa7ba30ba3
Fix rebuild index&csv parsing ()
Co-authored-by: luowei <glpat-EjySCyNjWiLqAED-YmwM>
Co-authored-by: crazywoola <427733928@qq.com>
Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>
2024-03-06 11:33:32 +08:00
Jyong
5b953c1ef2
Fix some RAG bugs ()
Co-authored-by: jyong <jyong@dify.ai>
2024-02-27 11:39:05 +08:00
Jyong
0620fa3094
Feat/vdb migrate command ()
Co-authored-by: jyong <jyong@dify.ai>
2024-02-26 19:47:29 +08:00
Jyong
4be3087642
Fix/new RAG bugs ()
Co-authored-by: jyong <jyong@dify.ai>
2024-02-23 16:54:15 +08:00
Jyong
91ea6fe4ee
Fix/langchain document schema ()
Co-authored-by: jyong <jyong@dify.ai>
2024-02-23 14:16:44 +08:00
Jyong
6c4e6bf1d6
Feat/dify rag ()
Co-authored-by: jyong <jyong@dify.ai>
2024-02-22 23:31:57 +08:00
Jyong
97fe817186
Fix/upload limit ()
Co-authored-by: jyong <jyong@dify.ai>
Co-authored-by: StyleZhang <jasonapring2015@outlook.com>
2024-02-22 17:16:22 +08:00
Bowen Liang
063191889d
chore: apply ruff's pyupgrade linter rules to modernize Python code with targeted version () 2024-02-09 15:21:33 +08:00
crazywoola
243ca5b1e2
fix: typo in package path of core.splitter () 2024-02-07 15:34:02 +08:00
Bowen Liang
843280f82b
enhancement: introduce Ruff for Python linter for reordering and removing unused imports with automated pre-commit and sytle check () 2024-02-06 13:21:13 +08:00
takatost
9f637ead38
bump version to 0.5.3 () 2024-02-01 18:11:57 +08:00
KVOJJJin
89fcf4ea7c
Feat: chunk overlap supported ()
Co-authored-by: jyong <jyong@dify.ai>
2024-01-26 13:24:40 +08:00
takatost
6cf93379b3
fix: split chunks return empty strings () 2024-01-25 13:59:18 +08:00
Jyong
869690c485
fix notion estimate ()
Co-authored-by: jyong <jyong@dify.ai>
2024-01-19 13:27:12 +08:00
Jyong
cb7a608d75
ascii filter Unicode U+FFFE ()
Co-authored-by: jyong <jyong@dify.ai>
2024-01-15 16:52:18 +08:00