diff --git a/README.md b/README.md index 79a5fcf3d..6793e65b4 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,64 @@ -English | [简体中文](./README_zh.md) +
+ +ragflow logo + +
-## System Environment Preparation +

+ English | + 简体中文 +

-### Install docker +

+ + Static Badge + + docker pull ragflow:v1.0 + + license + +

-If your machine doesn't have *Docker* installed, please refer to [Install Docker Engine](https://docs.docker.com/engine/install/) +[RAGFLOW](http://ragflow.io) is a knowledge management platform built on custom-build document understanding engine and LLM, +with reasoned and well-founded answers to your question. Clone this repository, you can deploy your own knowledge management +platform to empower your business with AI. + +
+ +
-### OS Setups -Firstly, you need to check the following command: +# Features +- **Custom-build document understanding engine.** Our deep learning engine is made according to the needs of analyzing and searching various type of documents in different domain. + - For documents from different domain for different purpose, the engine applys different analyzing and search strategy. + - Easily intervene and manipulate the data proccessing procedure when things goes beyond expectation. + - Multi-media document understanding is supported using OCR and multi-modal LLM. +- **State-of-the-art table structure and layout recognition.** Precisely extract and understand the document including table content. [README](./deepdoc/README.md) + - For PDF files, layout and table structures including row, column and span of them are recognized. + - Put the table accrossing the pages together. + - Reconstruct the table structure components into html table. +- **Querying database dumped data are supported.** After uploading tables from any database, you can search any data records just by asking. + - Instead of using SQL to query a database, every one cat get the wanted data just by asking using natrual language. + - The record number uploaded is not limited. + - Some extra description of column headers should be provided. +- **Reasoned and well-founded answers.** The cited document part in LLM's answer is provided and pointed out in the original document. + - The answers are based on retrieved result for which we apply vector-keyword hybrids search and rerank. + - The part of document cited in the answer is presented in the most expressive way. + - For PDF file, the cited parts in document can be located in the original PDF. + + +# Release Notification +**Star us on GitHub, and be notified for a new releases instantly!** +![star-us](https://github.com/langgenius/dify/assets/100913391/95f37259-7370-4456-a9f0-0bc01ef8642f) + +# Installation +## System Requirements +Be aware of the system minimum requirements before starting installation. +- CPU >= 2 cores +- RAM >= 8GB + +Then, you need to check the following command: ```bash 121:/ragflow# sysctl vm.max_map_count vm.max_map_count = 262144 @@ -24,7 +74,11 @@ Add or update the following line in the file: vm.max_map_count=262144 ``` -## Here we go! +## Install docker + +If your machine doesn't have *Docker* installed, please refer to [Install Docker Engine](https://docs.docker.com/engine/install/) + +## Quick Start > If you want to change the basic setups, like port, password .etc., please refer to [.env](./docker/.env) before starting the system. > If you change anything in [.env](./docker/.env), please check [service_conf.yaml](./docker/service_conf.yaml) which is a @@ -37,10 +91,13 @@ vm.max_map_count=262144 > [OpenAI](https://platform.openai.com/login?launch), [通义千问/QWen](https://dashscope.console.aliyun.com/model), > [智谱AI/ZhipuAI](https://open.bigmodel.cn/) ```bash -121:/ragflow# cd docker +121:/# git clone https://github.com/infiniflow/ragflow.git +121:/# cd ragflow/docker 121:/ragflow/docker# docker compose up -d ``` -If after about a half of minutes, use the following command to check the server status. If you can have the following outputs, +> The core image is about 15GB, please be patient for the first time + +After pulling all the images and running up, use the following command to check the server status. If you can have the following outputs, _**Hallelujah!**_ You have successfully launched the system. ```bash 121:/ragflow# docker logs -f ragflow-server @@ -58,10 +115,25 @@ _**Hallelujah!**_ You have successfully launched the system. INFO:werkzeug:Press CTRL+C to quit ``` -Open your browser, after entering the IP address of your server, if you see the flowing in your browser, _**Hallelujah**_ again! -> The default serving port is 80, if you want to change that, please refer to [ragflow.conf](./nginx/ragflow.conf), -> and change the *listen* value. - -
- -
\ No newline at end of file +Open your browser, enter the IP address of your server, _**Hallelujah**_ again! +> The default serving port is 80, if you want to change that, please refer to [docker-compose.yml](./docker-compose.yaml), +> and change the left part of *'80:80'*'. + +# Configuration +If you need to change the default setting of the system when you deploy it. There several ways to configure it. +Please refer to [README](./docker/README.md) and manually set the configuration. +After changing something, please run *docker-compose up -d* again. + +# RoadMap + +- [ ] File manager. +- [ ] Support URLs. Crawl web and extract the main content. + + +# Contributing + +For those who'd like to contribute code, see our [Contribution Guide](https://github.com/infiniflow/ragflow/blob/main/CONTRIBUTING.md). + +# License + +This repository is available under the [Ragflow Open Source License](LICENSE), which is essentially Apache 2.0 with a few additional restrictions. diff --git a/api/apps/llm_app.py b/api/apps/llm_app.py index 9f22a32eb..e0e213f4d 100644 --- a/api/apps/llm_app.py +++ b/api/apps/llm_app.py @@ -15,18 +15,12 @@ # from flask import request from flask_login import login_required, current_user - -from api.db.services import duplicate_name from api.db.services.llm_service import LLMFactoriesService, TenantLLMService, LLMService -from api.db.services.user_service import TenantService, UserTenantService from api.utils.api_utils import server_error_response, get_data_error_result, validate_request -from api.utils import get_uuid, get_format_time -from api.db import StatusEnum, UserTenantRole, LLMType -from api.db.services.knowledgebase_service import KnowledgebaseService -from api.db.db_models import Knowledgebase, TenantLLM -from api.settings import stat_logger, RetCode +from api.db import StatusEnum, LLMType +from api.db.db_models import TenantLLM from api.utils.api_utils import get_json_result -from rag.llm import EmbeddingModel, CvModel, ChatModel +from rag.llm import EmbeddingModel, ChatModel @manager.route('/factories', methods=['GET']) @@ -119,4 +113,5 @@ def list(): return get_json_result(data=res) except Exception as e: - return server_error_response(e) \ No newline at end of file + return server_error_response(e) + diff --git a/api/ragflow_server.py b/api/ragflow_server.py index a0d7a710f..981cc1bb9 100644 --- a/api/ragflow_server.py +++ b/api/ragflow_server.py @@ -40,7 +40,7 @@ if __name__ == '__main__': /_/ |_| \__,_/ \__, //_/ /_/ \____/ |__/|__/ /____/ - """) + """, flush=True) stat_logger.info( f'project base: {utils.file_utils.get_project_base_directory()}' ) diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 000000000..812f8b13b --- /dev/null +++ b/docker/README.md @@ -0,0 +1,80 @@ + +# Docker Environment Variable + +Look into [.env](./.env), there're some important variables. + +## MYSQL_PASSWORD +The mysql password could be changed by this variable. But you need to change *mysql.password* in [service_conf.yaml](./service_conf.yaml) at the same time. + + +## MYSQL_PORT +It refers to exported port number of mysql docker container, it's useful if you want to access the database outside the docker containers. + +## MINIO_USER +It refers to user name of [Mino](https://github.com/minio/minio). The modification should be synchronous updating at minio.user of [service_conf.yaml](./service_conf.yaml). + +## MINIO_PASSWORD +It refers to user password of [Mino](https://github.com/minio/minio). The modification should be synchronous updating at minio.password of [service_conf.yaml](./service_conf.yaml). + + +## SVR_HTTP_PORT +It refers to The API server serving port. + + +# Service Configuration +[service_conf.yaml](./service_conf.yaml) is used by the *API server* and *task executor*. It's the most important configuration of the system. + +## ragflow + +### host +The IP address used by the API server. + +### port +The serving port of API server. + +## mysql + +### name +The database name in mysql used by this system. + +### user +The database user name. + +### password +The database password. The modification should be synchronous updating at *MYSQL_PASSWORD* in [.env](./.env). + +### port +The serving port of mysql inside the container. The modification should be synchronous updating at [docker-compose.yml](./docker-compose.yml) + +### max_connections +The max database connection. + +### stale_timeout +The timeout duation in seconds. + +## minio + +### user +The username of minio. The modification should be synchronous updating at *MINIO_USER* in [.env](./.env). + +### password +The password of minio. The modification should be synchronous updating at *MINIO_PASSWORD* in [.env](./.env). + +### host +The serving IP and port inside the docker container. This is not updating until changing the minio part in [docker-compose.yml](./docker-compose.yml) + +## user_default_llm +Newly signed-up users use LLM configured by this part. Otherwise, user need to configure his own LLM in *setting*. + +### factory +The LLM suppliers. '通义千问', "OpenAI" and "智谱AI" are supported. + +### api_key +The corresponding API key of your assigned LLM vendor. + +## oauth +This is OAuth configuration which allows your system using the third-party account to sign-up and sign-in to the system. + +### github +Got to [Github](https://github.com/settings/developers), register new application, the *client_id* and *secret_key* will be given. +