signoz/pkg/query-service
Raj Kamal Singh 1b6b3c2fdf
Feat: query service: logs pipelines timestamp parsing processor (#4105)
* chore: relocate tests for trace and grok parsing processor

* chore: add test for timestamp parsing processor

* feat: update PipelineOperator model for time parser fields and get tests passing

* chore: add test cases for validating time parser fails silently on mismatched logs

* chore: add helper for generating regex for strptime layouts

* feat: time_parser ignore logs whose parseFrom value doesn't match strptime layout

* chore: escape regex special chars if any in the layout string before preparing regex

* chore: add operator.If on time_parser when using layout type epoch

* chore: finish up with operator.If on time_parser for  layout type

* chore: postable pipeline validation for time parser

* chore: some cleanup

* chore: some more cleanup

* chore: add validation for strptime layouts in postable pipelines

---------

Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
2023-11-29 18:55:01 +05:30
..
2023-09-17 10:40:45 +05:30
2023-11-16 15:11:38 +05:30
2021-01-03 18:15:44 +05:30
2021-09-02 13:18:47 +05:30

Query Service

Query service is the interface between frontend and databases. It is written in Golang. It will have modules for all supported databases. Query service is responsible to:

  • parse the request from Frontend
  • create relevant Clickhouse queries (and all other supported database queries)
  • parse response from databases and handle error if any
  • clickhouse response in the format accepted by Frontend

Complete the clickhouse setup locally.

https://github.com/SigNoz/signoz/blob/main/CONTRIBUTING.md#to-run-clickhouse-setup-recommended-for-local-development

  • Comment out the query-service and the frontend section in signoz/deploy/docker/clickhouse-setup/docker-compose.yaml
  • Change the alertmanager section in signoz/deploy/docker/clickhouse-setup/docker-compose.yaml as follows:
alertmanager:
    image: signoz/alertmanager:0.23.4
    volumes:
      - ./data/alertmanager:/data
    expose:
      - "9093"
    ports:
      - "8080:9093"
    # depends_on:
    #   query-service:
    #     condition: service_healthy
    restart: on-failure
    command:
      - --queryService.url=http://172.17.0.1:8085
      - --storage.path=/data
  • Run the following:
cd signoz/
If you are using x86_64 processors (All Intel/AMD processors) run sudo make run-x86
If you are on arm64 processors (Apple M1 Macs) run sudo make run-arm

Backend Configuration

  • Open ./constants/constants.go

    • Replace const RELATIONAL_DATASOURCE_PATH = "/var/lib/signoz/signoz.db"
      with const RELATIONAL_DATASOURCE_PATH = "./signoz.db".
  • Query Service needs below env variables to run:

    export ClickHouseUrl=tcp://localhost:9001
    export STORAGE=clickhouse
    export ALERTMANAGER_API_PREFIX=http://localhost:9093/api/

Build and Run locally

cd pkg/query-service
go build -o build/query-service main.go
ClickHouseUrl=tcp://localhost:9001 STORAGE=clickhouse build/query-service --prefer-delta=true 

Frontend Configuration for local query-service.

  • Set the following environment variables
export FRONTEND_API_ENDPOINT=http://localhost:8080
  • Run the following
cd signoz\frontend\
yarn install
yarn dev

Note:

If you use go version 1.18 for development and contributions, then please checkout the following issue. https://github.com/SigNoz/signoz/issues/1371

Docker Images

The docker images of query-service is available at https://hub.docker.com/r/signoz/query-service