
Enhance `LLMNode` with multimodal capability, introducing support for image outputs. This implementation extracts base64-encoded images from LLM responses, saves them to the storage service, and records the file metadata in the `ToolFile` table. In conversations, these images are rendered as markdown-based inline images. Additionally, the images are included in the LLMNode's output as file variables, enabling subsequent nodes in the workflow to utilize them. To integrate file outputs into workflows, adjustments to the frontend code are necessary. For multimodal output functionality, updates to related model configurations are required. Currently, this capability has been applied exclusively to Google's Gemini models. Close #15814. Signed-off-by: -LAN- <laipz8200@outlook.com> Co-authored-by: -LAN- <laipz8200@outlook.com>
Dify Backend API
Usage
Important
In the v1.3.0 release,
poetry
has been replaced withuv
as the package manager for Dify API backend service.
-
Start the docker-compose stack
The backend require some middleware, including PostgreSQL, Redis, and Weaviate, which can be started together using
docker-compose
.cd ../docker cp middleware.env.example middleware.env # change the profile to other vector database if you are not using weaviate docker compose -f docker-compose.middleware.yaml --profile weaviate -p dify up -d cd ../api
-
Copy
.env.example
to.env
cp .env.example .env
-
Generate a
SECRET_KEY
in the.env
file.bash for Linux
sed -i "/^SECRET_KEY=/c\SECRET_KEY=$(openssl rand -base64 42)" .env
bash for Mac
secret_key=$(openssl rand -base64 42) sed -i '' "/^SECRET_KEY=/c\\ SECRET_KEY=${secret_key}" .env
-
Create environment.
Dify API service uses UV to manage dependencies. First, you need to add the uv package manager, if you don't have it already.
pip install uv # Or on macOS brew install uv
-
Install dependencies
uv sync --dev
-
Run migrate
Before the first launch, migrate the database to the latest version.
uv run flask db upgrade
-
Start backend
uv run flask run --host 0.0.0.0 --port=5001 --debug
-
Start Dify web service.
-
Setup your application by visiting
http://localhost:3000
. -
If you need to handle and debug the async tasks (e.g. dataset importing and documents indexing), please start the worker service.
uv run celery -A app.celery worker -P gevent -c 1 --loglevel INFO -Q dataset,generation,mail,ops_trace,app_deletion
Testing
-
Install dependencies for both the backend and the test environment
uv sync --dev
-
Run the tests locally with mocked system environment variables in
tool.pytest_env
section inpyproject.toml
uv run -P api bash dev/pytest/pytest_all_tests.sh