52 Commits

Author SHA1 Message Date
Raj Kamal Singh
bab8c8274c
feat: aws integration UI facing api: services (#6803)
* feat: cloud service integrations: get model and repo interface started

* feat: cloud service integrations: flesh out more of cloud services model

* feat: cloud integrations: reorganize things a little

* feat: cloud integrations: get svc controller started

* feat: cloud integrations: add stubs for EC2 and RDS postgres services

* feat: cloud integrations: add validation for listing and getting available svcs and some cleanup

* feat: cloud integrations: refactor helpers in existing integrations code for reuse

* feat: cloud integrations: parsing of cloud service definitions

* feat: cloud integrations: impl for getCloudProviderService

* feat: cloud integrations: some reorganization

* feat: cloud integrations: some more cleanup

* feat: cloud integrations: add validation for listing available cloud provider services

* feat: cloud integrations: API endpoint for listing available cloud provider services

* feat: cloud integrations: add validation for getting details of a particular service

* feat: cloud integrations: API endpoint for getting details of a service

* feat: cloud integrations: add controller validation for configuring cloud services

* feat: cloud integrations: get serviceConfigRepo started

* feat: cloud integrations: service config in service list summaries when queried for cloud account id

* feat: cloud integrations: only a supported service for a connected cloud account can be configured

* feat: cloud integrations: add validation for configuring services via the API

* feat: cloud integrations: API for configuring services

* feat: cloud integrations: some cleanup

* feat: cloud integrations: fix broken test

---------

Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
2025-01-16 17:36:09 +05:30
Shivanshu Raj Shrivastava
2ead4fbb66
fix: Modifies messaging queue paylod (#6783)
* fix: use filterset

Signed-off-by: Shivanshu Raj Shrivastava <shivanshu1333@gmail.com>
2025-01-09 16:26:06 +05:30
Shivanshu Raj Shrivastava
505757b971
[feat] Add overview APIs for Celery and Kafka in messaging queues integration (#6756)
* feat: added queue overview api for generic messaging system
Signed-off-by: Shivanshu Raj Shrivastava <shivanshu1333@gmail.com>
2025-01-08 06:50:11 +00:00
Shivanshu Raj Shrivastava
60dc479a19
fix: add bucketing (#6669)
Co-authored-by: Nityananda Gohain <nityanandagohain@gmail.com>
2024-12-18 19:57:33 +05:30
Shivanshu Raj Shrivastava
83aa48c721
update service.instance.id (#6665)
* nit: update resource id and revert the flag
2024-12-18 19:06:22 +05:30
Shivanshu Raj Shrivastava
83f6dea2db
Add support for trace_v3 schema in messaging queues (#6663)
feat: support trace v3 queries
2024-12-18 17:04:01 +05:30
Shivanshu Raj Shrivastava
51baf7f8d3
feat: add byte rate to producer API (#6579)
* fix: fix partition check for topic throughput
2024-12-09 15:18:41 +00:00
Ekansh Gupta
d1503f1418
feat: fixProducerAPI (#6422)
chore: bugfix
2024-11-12 05:30:36 +00:00
Shivanshu Raj Shrivastava
b770fc2457
fix: typo (#6334) 2024-10-31 20:11:50 +05:30
Shivanshu Raj Shrivastava
fbe75cd057
fix: use query builder for metrics onboarding API (#6327) 2024-10-30 23:13:56 +05:30
Shivanshu Raj Shrivastava
a31c4b8339
Fix api query context (#6268)
* chore: fix naming api query context
2024-10-25 14:06:54 +05:30
Shivanshu Raj Shrivastava
204728ff60
chore: move constants to fix ci (#6213) 2024-10-18 01:15:54 +05:30
Shivanshu Raj Shrivastava
e51f4d986d
feat: kafka Scenario 1, 3, 4 all squashed (#6144)
* feat: kafka partition level observerability features
2024-10-17 19:44:42 +05:30
shivanshu
98cbdf570f
feat: api documentation and nits 2024-10-04 01:44:52 +05:30
shivanshu
d380894c35
feat: a bunch of advancements, query optimisation, new response format 2024-10-04 01:11:05 +05:30
shivanshu
ea0263cc73
feat: onboarding APIs 2024-10-04 01:11:05 +05:30
shivanshu
f38a1d9f1c
feat: add the queries 2024-10-04 01:11:05 +05:30
shivanshu
aabf364cc6
feat: add partition level granularity 2024-08-27 18:27:44 +05:30
shivanshu
8d655bf419
chore: use MinAllowedStepInterval 2024-08-27 14:04:00 +05:30
shivanshu
90cb8ba9a1
chore: modify producer output 2024-08-26 20:35:08 +05:30
shivanshu
f508ee7521
chore: query, response update 2024-08-26 20:28:22 +05:30
shivanshu
413caad0d8
chore: cleanup 2024-08-26 20:28:22 +05:30
shivanshu
666f601ecd
feat: change to builder queries 2024-08-26 20:28:22 +05:30
shivanshu
5cdcbef00c
feat: add network latency for kafka 2024-08-26 20:28:22 +05:30
shivanshu
3b94dab3ce
chore: small patch to fix consumer_group check 2024-08-26 15:36:18 +05:30
shivanshu
c957c0f757
chore: addressing review comments 2024-08-05 18:14:40 +05:30
shivanshu
3ff0aa4b4b
chore: consumer group filtering 2024-08-05 18:09:58 +05:30
shivanshu
063c9adba6
chore: pr-reviews 2024-08-05 18:09:58 +05:30
shivanshu
5c3ce146fa
chore: add queue type 2024-08-05 18:09:58 +05:30
shivanshu
481bb6e8b8
feat: add consumer and producer APIs 2024-08-05 18:09:58 +05:30
Raj Kamal Singh
59121bd932
chore: nginx integration: add note about adjusting regex if using custom log format (#5615) 2024-07-31 17:52:51 +05:30
Srikanth Chekuri
6ee9705599
chore: bump SigNoz/signoz-otel-collector and SigNoz/prometheus (#5294) 2024-06-20 18:33:45 +05:30
Srikanth Chekuri
2f949d2738
chore: address large number of staticcheck issues (#5176)
* chore: address large number of staticcheck issues

* chore: fix tests

* chore: fix more issue

* chore: fix options
2024-06-11 20:10:38 +05:30
Srikanth Chekuri
592073a564
chore: add AWS RDS and ElastiCache integration (#5071) 2024-06-02 14:42:05 +05:30
Raj Kamal Singh
5a778dcb18
Chore: integrations: populate updatedAt for integration dashboards (#5019)
* chore: add test for updatedAt value being populated in integration dashboards and get it passing

* chore: also populate createdAt, createBy and updateBy for instaled integration dashboards

* chore: update clickhouse integration config instructions
2024-05-16 21:44:46 +05:30
Raj Kamal Singh
3efd9801a1
Chore: restrict logs connection test for integrations to use log attributes for identifying logs (#4977)
* chore: change logs connection test spec to be based on an attrib value

* chore: disallow unknown fields while unmarshalling JSON for an integration

* chore: add description field to collected metric spec

* chore: update logs connection test for builtin integrations

* chore: update logic for calculating logs connection status
2024-05-15 14:36:52 +05:30
Raj Kamal Singh
e6e0a59f5f
Feat: integrations: clickhouse (#4879)
* chore: get built-in clickhouse integration started

* chore: update config pre-requisites for clickhouse integration

* chore: add details of metrics data collected for clickhouse integration

* chore: clickhouse integration: move list of data-collected to its own file

* chore: clickhouse integration: get overview dashboard started

* chore: start with logs collection instructions for clickhouse

* chore: regex parsing for clickhouse text logs

* chore: timestamp parsing for clickhouse logs

* chore: severity parsing for clickhouse logs

* chore: clickhouse logs parsing: move parsed message to body if available

* chore: update pre-reqs for collecting from system.query_log table

* feat: add instructions for collecting from system.query_log table

* feat: add logs attribs collected

* chore: some cleanup of clickhouse overview dashboard

* feat: finish up with clickhouse overview dashboard for clickhouse integration
2024-04-26 09:45:57 +05:30
Raj Kamal Singh
39e0ef68ca
chore: integration instructions: add typical log file locations on macOS (#4779) 2024-04-01 12:06:08 +05:30
Raj Kamal Singh
990fc83269
Feat/integrations v0 mongo and nginx (#4763)
* feat: flesh out pre-requisites for collecting mongodb logs and metrics

* chore: remove stale pipelines in bundled integrations

* chore: clean up 'collect metrics' step for mongodb

* feat: add instructions for collecting and parsing mongodb logs

* feat: add metrics and logs attributes to mongodb data collected list

* feat: nginx logs collection instructions and some other cleanup

* feat: add list of parsed log attributes to data collected list for nginx

* chore: do not run pipeline population integration test if no built-in integration has a pipeline
2024-03-28 19:57:07 +05:30
Raj Kamal Singh
0ac9f6f663
Feat: QS: redis integration v0: instructions for collecting and parsing logs (#4753)
* chore: minor cleanups to postgres integration instructions

* chore: update instructions for connecting redis integration

* feat: add instructions for collecting redis logs

* chore: flesh out prerequisites for connecting redis integration

* chore: add list of metrics collected for redis
2024-03-27 20:03:27 +05:30
Raj Kamal Singh
f24135f5b0
Feat: QS: postgres integration: instructions for collecting and parsing logs (#4738)
* chore: offer metrics config instructions for signoz cloud only

* chore: some more cleanups

* chore: get log collection instructions started

* feat: flesh out log collection otel config for postgres

* chore: some cleanup

* chore: some more cleanup

* chore: some more cleanup
2024-03-23 11:39:28 +05:30
Raj Kamal Singh
f4e94c0ad1
chore: update postgres config instructions to work for both signoz cloud and self-hosted (#4718) 2024-03-19 12:28:22 +05:30
Raj Kamal Singh
6f3183823f
Feat: postgres integration v0 (#4704)
* chore: update annotations for pre blocks in configuration instructions

* chore: update list of collected metrics for postgres integration

* chore: change non-string units to string in metrics collected list

* chore: some cleanups for postgres config instructions

* chore: some cleanup to metrics connection status resource labels

* chore: remove stub pipeline in postgres integration - no interesting log parsing to be done
2024-03-18 18:20:12 +05:30
Raj Kamal Singh
43f9830e8d
Feat: integrations v0 metrics connection status (#4715)
* chore: add test expectations for integration metrics connection status

* chore: reorg logs connection status calculation for parallelization

* chore: add interface for reader.GetMetricLastReceivedTsMillis

* chore: add plumbing for calculating integration metrics connection status

* chore: impl and test mocks for reader.GetMetricReceivedLatest

* chore: wrap things up and get test passing

* chore: some cleanup

* chore: some more cleanup

* chore: use prom metric names for integration connection test
2024-03-18 10:01:53 +05:30
Raj Kamal Singh
5a2d729ba9
Feat: qs autopopulate installed integration dashboards (#4680)
* chore: add test for dashboards for installed integrations

* feat: include dashboards for installed integrations in API response

* chore: add test expectation for getting installed integration dashboard by id

* feat: add support for retrieving installed integration dashboards by id

* chore: add dashboard id validation for integrations
2024-03-11 20:06:59 +05:30
Raj Kamal Singh
9ace374855
Feat: QS: Log Pipelines for installed integrations (#4674)
* chore: refactor: inject sqlx.DB into opamp.initDB instead of DB file name

* chore: reorganize test utils a little

* chore: add test validating pipelines for installed integrations show up in pipelines list

* chore: get basic integration pipelines testcase passing

* chore: reconcile experimental changes with latest state of develop

* chore: add integration test for reordering of pipelines

* chore: marker for integration pipelines using Id

* chore: hookup propagation of installed integration pipelines by opamp

* chore: add util for mapping slices

* chore: add support for reordering integration pipelines

* chore: exclude user saved integration pipelines if no longer installed

* chore: flesh out rest of intgeration pipelines scenarios

* chore: handle scenario when an integration is installed before any pipelines exist

* chore: notify agentConf of update after uninstalling an integration

* chore: some minor cleanup

* chore: some more cleanup

* chore: update ee server for changed controllers

* chore: some more cleanup

* chore: change builtin integration id prefix to avoid using colons that break yaml

* chore: update builtin integration id in test
2024-03-11 14:15:11 +05:30
Vikrant Gupta
a4d5774ae3
fix: update the md file for integrations config (#4677) 2024-03-11 11:48:36 +05:30
Raj Kamal Singh
a295bf2fb6
Feat: QS: structure for built in integrations (#4655)
* feat: get builtin integrations started with nginx

* feat: get started with embedding and parsing of builtin integrations

* chore: add icons for nginx and redis integrations

* chore: stash current state of work

* chore: remove all yaml annotations since moved to JSON assets for bundled integrations

* chore: add file uri hydration in integration spec

* chore: refactor file uri hydration logic

* chore: add support for referencing JSON files with file uri

* chore: bring in initial integration assets

* chore: hookup builtin integrations and get all tests passing

* chore: update icons for postgres and mongo and some cleanup

* chore: some more cleanup

---------

Co-authored-by: Raj Singh <raj@Rajs-MacBook-Pro.local>
2024-03-07 19:26:20 +05:30
Raj Kamal Singh
7136ecc2fe
chore: accept connection test lookback seconds as request param (#4650)
Also removes connectionStatus from integration details.
2024-03-06 11:06:04 +05:30
Raj Kamal Singh
ab5285dee6
Feat: qs api integration connection status (#4628)
* chore: add integration attribs for connection tests and status

* chore: add connection status to integration details response

* chore: update integration lifecycle test to check for connection status too

* feat: add GetIntegrationConnectionTests to integrations manager and controller

* chore: add tests for querying integration connection status

* feat: add http API support for integration connection status

* chore: some cleanups

* chore: use PostableRule for integration alerts

* chore: some more cleanup
2024-03-05 15:23:56 +05:30