AI/ragflow
1
0
Fork 0
mirror of https://git-proxy.hk.martin98.com/https://github.com/infiniflow/ragflow.git synced 2025-04-10 15:40:35 +08:00
Code Issues Packages Projects Releases Wiki Activity

Dynamically create the service_conf.yaml file by replacing environment variables from .env (#3341)

Browse Source 
### What problem does this PR solve?

This pull request implements the feature mentioned in #3322. 

Instead of manually having to edit the `service_conf.yaml` file when
changes have been made to `.env` and mapping it into the docker
container at runtime, a template file is used and the values replaced by
the environment variables from the `.env` file when the container is
started.
 

### Type of change

- [X] New Feature (non-breaking change which adds functionality)

---------

Co-authored-by: Kevin Hu <kevinhu.sh@gmail.com>
Co-authored-by: Zhichang Yu <yuzhichang@gmail.com>
This commit is contained in:
Guido Schmutz 2024-11-12 15:56:53 +01:00 committed by GitHub
parent 7274420ecd
commit 0c95a3382b
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
12 changed files with 122 additions and 35 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
Dockerfile
View File

@ -129,6 +129,7 @@ ENV PATH="${VIRTUAL_ENV}/bin:${PATH}"
ENV PYTHONPATH=/ragflow/
COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template
COPY docker/entrypoint.sh ./entrypoint.sh
RUN chmod +x ./entrypoint.sh

1
Dockerfile.scratch.oc9
View File

@ -53,6 +53,7 @@ RUN conda run -n py11 python -m nltk.downloader wordnet
ENV PYTHONPATH=/ragflow/
ENV HF_ENDPOINT=https://hf-mirror.com
COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template
ADD docker/entrypoint.sh ./entrypoint.sh
RUN chmod +x ./entrypoint.sh

1
Dockerfile.slim
View File

@ -122,6 +122,7 @@ ENV PATH="${VIRTUAL_ENV}/bin:${PATH}"
ENV PYTHONPATH=/ragflow/
COPY docker/service_conf.yaml.template ./conf/service_conf.yaml.template
COPY docker/entrypoint.sh ./entrypoint.sh
RUN chmod +x ./entrypoint.sh

17
README.md
View File

@ -216,7 +216,7 @@ releases! 🌟
5. In your web browser, enter the IP address of your server and log in to RAGFlow.
> With the default settings, you only need to enter `http://IP_OF_YOUR_MACHINE` (**sans** port number) as the default
HTTP serving port `80` can be omitted when using the default configurations.
6. In [service_conf.yaml](./docker/service_conf.yaml), select the desired LLM factory in `user_default_llm` and update
6. In [service_conf.yaml.template](./docker/service_conf.yaml.template), select the desired LLM factory in `user_default_llm` and update
the `API_KEY` field with the corresponding API key.
> See [llm_api_key_setup](https://ragflow.io/docs/dev/llm_api_key_setup) for more information.
@ -229,16 +229,11 @@ When it comes to system configurations, you will need to manage the following fi
- [.env](./docker/.env): Keeps the fundamental setups for the system, such as `SVR_HTTP_PORT`, `MYSQL_PASSWORD`, and
`MINIO_PASSWORD`.
- [service_conf.yaml](./docker/service_conf.yaml): Configures the back-end services.
- [docker-compose.yml](./docker/docker-compose.yml): The system relies
on [docker-compose.yml](./docker/docker-compose.yml) to start up.
You must ensure that changes to the [.env](./docker/.env) file are in line with what are in the [service_conf.yaml](./docker/service_conf.yaml) file.
- [service_conf.yaml.template](./docker/service_conf.yaml.template): Configures the back-end services. The environment variables in this file will be automatically populated when the Docker container starts. Any environment variables set within the Docker container will be available for use, allowing you to customize service behavior based on the deployment environment.
- [docker-compose.yml](./docker/docker-compose.yml): The system relies on [docker-compose.yml](./docker/docker-compose.yml) to start up.
> The [./docker/README](./docker/README.md) file provides a detailed description of the environment settings and service
> configurations, and you are REQUIRED to ensure that all environment settings listed in
> the [./docker/README](./docker/README.md) file are aligned with the corresponding configurations in
> the [service_conf.yaml](./docker/service_conf.yaml) file.
> configurations which can be used as `${ENV_VARS}` in the [service_conf.yaml.template](./docker/service_conf.yaml.template) file.
To update the default HTTP serving port (80), go to [docker-compose.yml](./docker/docker-compose.yml) and change `80:80`
to `<YOUR_SERVING_PORT>:80`.
@ -293,11 +288,11 @@ docker build -f Dockerfile -t infiniflow/ragflow:dev .
docker compose -f docker/docker-compose-base.yml up -d
```
Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml** to `127.0.0.1`:
Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/.env** to `127.0.0.1`:
```
127.0.0.1 es01 infinity mysql minio redis
```
In **docker/service_conf.yaml**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
In **docker/service_conf.yaml.template**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
4. If you cannot access HuggingFace, set the `HF_ENDPOINT` environment variable to use a mirror site:

11
docker/.env
View File

@ -1,6 +1,9 @@
# The version of Elasticsearch.
STACK_VERSION=8.11.3
# The hostname where the Elasticsearch service is exposed
ES_HOST=es01
# The port used to expose the Elasticsearch service to the host machine,
# allowing EXTERNAL access to the service running inside the Docker container.
ES_PORT=1200
@ -27,10 +30,16 @@ INFINITY_PSQL_PORT=5432
# The password for MySQL.
# When updated, you must revise the `mysql.password` entry in service_conf.yaml.
MYSQL_PASSWORD=infini_rag_flow
# The hostname where the MySQL service is exposed
MYSQL_HOST=mysql
# The database of the MySQL service to use
MYSQL_DBNAME=rag_flow
# The port used to expose the MySQL service to the host machine,
# allowing EXTERNAL access to the MySQL database running inside the Docker container.
MYSQL_PORT=5455
# The hostname where the MySQL service is exposed
MINIO_HOST=minio
# The port used to expose the MinIO console interface to the host machine,
# allowing EXTERNAL access to the web-based console running inside the Docker container.
MINIO_CONSOLE_PORT=9001
@ -44,6 +53,8 @@ MINIO_USER=rag_flow
# When updated, you must revise the `minio.password` entry in service_conf.yaml accordingly.
MINIO_PASSWORD=infini_rag_flow
# The hostname where the Redis service is exposed
REDIS_HOST=redis
# The port used to expose the Redis service to the host machine,
# allowing EXTERNAL access to the Redis service running inside the Docker container.
REDIS_PORT=6379

16
docker/README.md
View File

@ -27,7 +27,7 @@ The [.env](./.env) file contains important environment variables for Docker.
- `ES_PORT`
The port used to expose the Elasticsearch service to the host machine, allowing **external** access to the service running inside the Docker container. Defaults to `1200`.
- `ELASTIC_PASSWORD`
The password for Elasticsearch. When updated, you must revise the `es.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
The password for Elasticsearch.
### Kibana
@ -46,7 +46,7 @@ The [.env](./.env) file contains important environment variables for Docker.
### MySQL
- `MYSQL_PASSWORD`
The password for MySQL. When updated, you must revise the `mysql.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
The password for MySQL.
- `MYSQL_PORT`
The port used to expose the MySQL service to the host machine, allowing **external** access to the MySQL database running inside the Docker container. Defaults to `5455`.
@ -57,16 +57,16 @@ The [.env](./.env) file contains important environment variables for Docker.
- `MINIO_PORT`
The port used to expose the MinIO API service to the host machine, allowing **external** access to the MinIO object storage service running inside the Docker container. Defaults to `9000`.
- `MINIO_USER`
The username for MinIO. When updated, you must revise the `minio.user` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
The username for MinIO.
- `MINIO_PASSWORD`
The password for MinIO. When updated, you must revise the `minio.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
The password for MinIO.
### Redis
- `REDIS_PORT`
The port used to expose the Redis service to the host machine, allowing **external** access to the Redis service running inside the Docker container. Defaults to `6379`.
- `REDIS_PASSWORD`
The password for Redis. When updated, you must revise the `redis.password` entry in [service_conf.yaml](./service_conf.yaml) accordingly.
The password for Redis.
### RAGFlow
@ -119,7 +119,7 @@ The [.env](./.env) file contains important environment variables for Docker.
## 🐋 Service configuration
[service_conf.yaml](./service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
[service_conf.yaml](./service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor. In a dockerized setup, this file is automatically created based on the [service_conf.yaml.template](./service_conf.yaml.template) file (replacing all environment variables by their values).
- `ragflow`
- `host`: The API server's IP address inside the Docker container. Defaults to `0.0.0.0`.
@ -139,11 +139,11 @@ The [.env](./.env) file contains important environment variables for Docker.
- `host`: The MinIO serving IP *and* port inside the Docker container. Defaults to `minio:9000`.
- `oauth`
The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.
The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.
- `github`: The GitHub authentication settings for your application. Visit the [Github Developer Settings page](https://github.com/settings/developers) to obtain your client_id and secret_key.
- `user_default_llm`
The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.
The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.
- `factory`: The LLM supplier. Available options:
- `"OpenAI"`
- `"DeepSeek"`

1
docker/docker-compose-gpu.yml
View File

@ -16,7 +16,6 @@ services:
- 80:80
- 443:443
volumes:
- ./service_conf.yaml:/ragflow/conf/service_conf.yaml
- ./ragflow-logs:/ragflow/logs
- ./nginx/ragflow.conf:/etc/nginx/conf.d/ragflow.conf
- ./nginx/proxy.conf:/etc/nginx/proxy.conf

7
docker/entrypoint.sh
View File

@ -1,5 +1,12 @@
#!/bin/bash
# replace env variables in the service_conf.yaml file
rm -rf /ragflow/conf/service_conf.yaml
while IFS= read -r line || [[ -n "$line" ]]; do
# Use eval to interpret the variable with default values
eval "echo \"$line\"" >> /ragflow/conf/service_conf.yaml
done < /ragflow/conf/service_conf.yaml.template
# unset http proxy which maybe set by docker daemon
export http_proxy=""; export https_proxy=""; export no_proxy=""; export HTTP_PROXY=""; export HTTPS_PROXY=""; export NO_PROXY=""

74
docker/service_conf.yaml.template Normal file
View File

@ -0,0 +1,74 @@
ragflow:
host: ${RAGFLOW_HOST:-0.0.0.0}
http_port: 9380
mysql:
name: '${MYSQL_DBNAME:-rag_flow}'
user: '${MYSQL_USER:-root}'
password: '${MYSQL_PASSWORD:-infini_rag_flow}'
host: '${MYSQL_HOST:-mysql}'
port: 3306
max_connections: 100
stale_timeout: 30
minio:
user: '${MINIO_USER:-rag_flow}'
password: '${MINIO_PASSWORD:-infini_rag_flow}'
host: '${MINIO_HOST:-minio}:9000'
es:
hosts: 'http://${ES_HOST:-es01}:9200'
username: '${ES_USER:-elastic}'
password: '${ES_PASSWORD:-infini_rag_flow}'
redis:
db: 1
password: '${REDIS_PASSWORD:-infini_rag_flow}'
host: '${REDIS_HOST:-redis}:6379'
# postgres:
# name: '${POSTGRES_DBNAME:-rag_flow}'
# user: '${POSTGRES_USER:-rag_flow}'
# password: '${POSTGRES_PASSWORD:-infini_rag_flow}'
# host: '${POSTGRES_HOST:-postgres}'
# port: 5432
# max_connections: 100
# stale_timeout: 30
# s3:
# endpoint: 'endpoint'
# access_key: 'access_key'
# secret_key: 'secret_key'
# region: 'region'
# azure:
# auth_type: 'sas'
# container_url: 'container_url'
# sas_token: 'sas_token'
# azure:
# auth_type: 'spn'
# account_url: 'account_url'
# client_id: 'client_id'
# secret: 'secret'
# tenant_id: 'tenant_id'
# container_name: 'container_name'
# user_default_llm:
# factory: 'Tongyi-Qianwen'
# api_key: 'sk-xxxxxxxxxxxxx'
# base_url: ''
# oauth:
# github:
# client_id: xxxxxxxxxxxxxxxxxxxxxxxxx
# secret_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
# url: https://github.com/login/oauth/access_token
# feishu:
# app_id: cli_xxxxxxxxxxxxxxxxxxx
# app_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
# app_access_token_url: https://open.feishu.cn/open-apis/auth/v3/app_access_token/internal
# user_access_token_url: https://open.feishu.cn/open-apis/authen/v1/oidc/access_token
# grant_type: 'authorization_code'
# authentication:
# client:
# switch: false
# http_app_key:
# http_secret_key:
# site:
# switch: false
# permission:
# switch: false
# component: false
# dataset: false

20
docs/configurations.md
View File

@ -12,11 +12,9 @@ Configurations for installing RAGFlow via Docker.
When it comes to system configurations, you will need to manage the following files:
- [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env): Contains important environment variables for Docker.
- [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml): Configures the back-end services. It specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
- [service_conf.yaml.template](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml.template): Configures the back-end services. It specifies the system-level configuration for RAGFlow and is used by its API server and task executor. Upon container startup, the `service_conf.yaml` file will be generated based on this template file. This process replaces any environment variables within the template, allowing for dynamic configuration tailored to the container's environment.
- [docker-compose.yml](https://github.com/infiniflow/ragflow/blob/main/docker/docker-compose.yml): The Docker Compose file for starting up the RAGFlow service.
You must ensure that changes to the [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file are in line with what are in the [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) file.
To update the default HTTP serving port (80), go to [docker-compose.yml](./docker/docker-compose.yml) and change `80:80`
to `<YOUR_SERVING_PORT>:80`.
@ -47,7 +45,7 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con
- `ES_PORT`
The port used to expose the Elasticsearch service to the host machine, allowing **external** access to the service running inside the Docker container. Defaults to `1200`.
- `ELASTIC_PASSWORD`
The password for Elasticsearch. When updated, you must revise the `es.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
The password for Elasticsearch.
### Kibana
@ -66,7 +64,7 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con
### MySQL
- `MYSQL_PASSWORD`
The password for MySQL. When updated, you must revise the `mysql.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
The password for MySQL.
- `MYSQL_PORT`
The port used to expose the MySQL service to the host machine, allowing **external** access to the MySQL database running inside the Docker container. Defaults to `5455`.
@ -77,16 +75,16 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con
- `MINIO_PORT`
The port used to expose the MinIO API service to the host machine, allowing **external** access to the MinIO object storage service running inside the Docker container. Defaults to `9000`.
- `MINIO_USER`
The username for MinIO. When updated, you must revise the `minio.user` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
The username for MinIO.
- `MINIO_PASSWORD`
The password for MinIO. When updated, you must revise the `minio.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
The password for MinIO. accordingly.
### Redis
- `REDIS_PORT`
The port used to expose the Redis service to the host machine, allowing **external** access to the Redis service running inside the Docker container. Defaults to `6379`.
- `REDIS_PASSWORD`
The password for Redis. When updated, you must revise the `redis.password` entry in [service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) accordingly.
The password for Redis.
### RAGFlow
@ -140,7 +138,7 @@ If you cannot download the RAGFlow Docker image, try the following mirrors.
## Service configuration
[service_conf.yaml](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml) specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
[service_conf.yaml.template](https://github.com/infiniflow/ragflow/blob/main/docker/service_conf.yaml.template) specifies the system-level configuration for RAGFlow and is used by its API server and task executor.
### `ragflow`
@ -164,13 +162,13 @@ If you cannot download the RAGFlow Docker image, try the following mirrors.
### `oauth`
The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.
The OAuth configuration for signing up or signing in to RAGFlow using a third-party account. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.
- `github`: The GitHub authentication settings for your application. Visit the [Github Developer Settings](https://github.com/settings/developers) page to obtain your client_id and secret_key.
### `user_default_llm`
The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml**.
The default LLM to use for a new RAGFlow user. It is disabled by default. To enable this feature, uncomment the corresponding lines in **service_conf.yaml.template**.
- `factory`: The LLM supplier. Available options:
- `"OpenAI"`

4
docs/guides/develop/launch_ragflow_from_source.md
View File

@ -64,13 +64,13 @@ docker compose -f docker/docker-compose-base.yml up -d
### Update `host` and `port` Settings for Third-party Services
1. Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml** to `127.0.0.1`:
1. Add the following line to `/etc/hosts` to resolve all hosts specified in **docker/service_conf.yaml.template** to `127.0.0.1`:
```
127.0.0.1 es01 infinity mysql minio redis
```
2. In **docker/service_conf.yaml**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
2. In **docker/service_conf.yaml.template**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**.
### Launch the RAGFlow Backend Service

4
docs/guides/llm_api_key_setup.md
View File

@ -19,7 +19,7 @@ If you find your online LLM is not on the list, don't feel disheartened. The lis
You have two options for configuring your model API key:
- Configure it in **service_conf.yaml** before starting RAGFlow.
- Configure it in **service_conf.yaml.template** before starting RAGFlow.
- Configure it on the **Model Providers** page after logging into RAGFlow.
### Configure model API key before starting up RAGFlow
@ -37,7 +37,7 @@ You have two options for configuring your model API key:
### Configure model API key after logging into RAGFlow
:::caution WARNING
After logging into RAGFlow, configuring your model API key through the **service_conf.yaml** file will no longer take effect.
After logging into RAGFlow, configuring your model API key through the **service_conf.yaml.template** file will no longer take effect.
:::
After logging into RAGFlow, you can *only* configure API Key on the **Model Providers** page: