mirror of
https://git.mirrors.martin98.com/https://github.com/infiniflow/ragflow.git
synced 2025-08-11 08:09:03 +08:00
Introduced jemalloc (#5590)
### What problem does this PR solve? Introduced jemalloc. Python uses pymalloc (which is an reimplementation of gblibc malloc) to manage RES. It has pools for small objects to avoid returning memory to OS aggressively. My experience is: Replacing pymalloc with [jemalloc](https://github.com/jemalloc/jemalloc) can reduce RES and speedup task_executor.py. ### Type of change - [x] Performance Improvement
This commit is contained in:
parent
688cb8f19d
commit
afe9269534
@ -59,6 +59,7 @@ RUN --mount=type=cache,id=ragflow_apt,target=/var/cache/apt,sharing=locked \
|
||||
apt install -y default-jdk && \
|
||||
apt install -y libatk-bridge2.0-0 && \
|
||||
apt install -y libpython3-dev libgtk-4-1 libnss3 xdg-utils libgbm-dev && \
|
||||
apt install -y libjemalloc-dev && \
|
||||
apt install -y python3-pip pipx nginx unzip curl wget git vim less
|
||||
|
||||
RUN if [ "$NEED_MIRROR" == "1" ]; then \
|
||||
|
@ -15,8 +15,9 @@ CONSUMER_NO_BEG=$1
|
||||
CONSUMER_NO_END=$2
|
||||
|
||||
function task_exe(){
|
||||
JEMALLOC_PATH=$(pkg-config --variable=libdir jemalloc)/libjemalloc.so
|
||||
while [ 1 -eq 1 ]; do
|
||||
$PY rag/svr/task_executor.py $1;
|
||||
LD_PRELOAD=$JEMALLOC_PATH $PY rag/svr/task_executor.py $1;
|
||||
done
|
||||
}
|
||||
|
||||
|
@ -17,8 +17,9 @@ if [[ -z "$WS" || $WS -lt 1 ]]; then
|
||||
fi
|
||||
|
||||
function task_exe(){
|
||||
JEMALLOC_PATH=$(pkg-config --variable=libdir jemalloc)/libjemalloc.so
|
||||
while [ 1 -eq 1 ];do
|
||||
$PY rag/svr/task_executor.py $1;
|
||||
LD_PRELOAD=$JEMALLOC_PATH $PY rag/svr/task_executor.py $1;
|
||||
done
|
||||
}
|
||||
|
||||
|
Loading…
x
Reference in New Issue
Block a user