* fix: improve user telemetry
- move GetEmailFromJwt to common function
- update AttachJwtToContext() to use standard way of attaching value to context
- update userEmail in every possible sendEvent call
- send groupId in sendEvent call for SaaS operator analytics
* chore: added DEFAULT_CLOUD_EMAIL const
* chore: add AttachJwtToContext to analytics middleware
* test: added AttachJwtToContext to logs pipelines
* chore: return warning logs too from collector simulator
* chore: also return collector logs in preview API response to help with debugging
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* chore: experiment with using a tmp file for getting collector logs in simulator
* chore: collector simulator: cleaned up tmp file based collector logs capture
* chore: add test validating regex proc doesn't error for logs that dont match
* chore: return collector error logs from pipeline preview API
* chore: add test validating regex processor doesn't log errors for mismatched logs
* chore: add if condition for generated regex processors
* chore: add test case validating json parser ignore non json logs
* chore: add if condition for operator generated for json parser
* chore: add test case validating move processor ignores logs with missing field
* chore: add if condition for operator generated for move
* chore: add test case validating copy processor ignores logs with missing field
* chore: add if condition for operator generated for copy
* chore: add test case validating remove processor ignores logs with missing field
* chore: add if condition for operator generated for remove
* chore: log pipelines: ensuring json parser ignores log if json field is missing
* chore: log pipelines: ensure regex parser ignores log if field is missing
* chore: add test for ensuring pipeline previews work for trace parser processors
* chore: updates to trace parser validation in postable pipelines
* chore: extract auth.randomHex into utils.RandomHex for reuse
* chore: get trace parser preview test passing
* chore: start with JSON serialized trace parser in test to cover deserialization
* chore: address PR feedback
* chore: add test for previewing pipeline with grok parser
* chore: import grok parser in logparsing preview to ensure its a registered logtransform op
* chore: add dependencies for grok parsing operator
Switched all file operations from ioutil package to io and os packages due to the deprecation of ioutil in Go 1.16. This change contributes to the maintainability of the codebase, ensuring it's up-to-date with the current standard library. Additionally, implemented usage of filepath.Join for platform-independent file paths.
* chore: add integration test for log pipelines being recommended to agents on connection
* chore: agentConf.Manager as AgentConfigProvider
* feat: logparsingpipelines as an AgentFeature
* chore: some cleanup
* feat: update agentConf init in ee query-service
* chore: more cleanup and move opamp/logspipeline -> logparsingpipeline/collector_config.go
* chore: some more cleanup
* chore: set agent.RemoteConfig after broadcasting conf to all agents
* chore: add test scenario for post remote conf application
* feat: add interface for opamp.AgentConfigProvider
* feat: add iface and plumbing for generating recommended conf in opamp/agent
* feat: get opamp server config provider tests started
* chore: add test scenario for agent connection without a config recommendation
* chore: add test scenario for agent connection with a config recommendation
* chore: add test for validating config deployment status gets reported
* chore: add test for rolling out latest config recommendations when config changes
* chore: wrap up opamp server lifecycle tests
* chore: some tests cleanup
* chore: get all tests passing
* chore: update opamp server init logic in ee query service
* chore: some cleanup
* chore: some final cleanup
* feat: add logsparsingpipeline controller api for pipeline previews
* feat: add http handler and route for generating pipeline previews
* feat: use a response model for pipeline previews response
* chore: rename model.GetLogsResponse -> model.SignozLog for use in both requests and responses
* feat: add test for simulating log pipelines processing
* feat: get pipeline preview tests passing
* chore: cleanup
* feat: query builder formula is updated
* feat: formula is updated for having and limit
* feat: orderBy is updated
* feat: formula is added
* chore: add query-service support for formula limit and order by
* feat: enable more filters is displayed when all data source is metrics
* chore: feedback is updated
* chore: feedback is updated
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
Co-authored-by: Rajat Dabade <rajat@signoz.io>
* chore: use v3.Filterset as pipeline filters in logparsing pipelines integration tests
* chore: get logparsing integration tests passing with filterset based pipeline
* chore: get all other breaking tests passing
* chore: move models.logparsingpipeline to logparsingpipeline.model
* chore: implement Valuer and Scanner interfaces for v3.FilterSet
* feat: limit filter is enabled in the metrics filters
* chore: orderBy is enabled for metrics data source
* chore: fix table type limit and order by
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* chore: add integration test for log parsing pipelines validation
* chore: add helpers for creating unauthorized, unavailable & not found api errors
* chore: return *model.APIError from logpipeline and agentConf functions
* chore: some cleanup
* chore: some more cleanup
* chore: one more round of cleanups
* fix: avoid pipeline builder panic if first op is disabled
* chore: use pipeline builder test to trigger getOperator panic
---------
Co-authored-by: Nityananda Gohain <nityanandagohain@gmail.com>
* fix: exists, nexists support for top level logs columns
* fix: dont check value for exists and nexists
* fix: exists, nexists updated for materialized columns
* fix: remove unnecesary variable
* fix: exists check updated in all places
* feat: get and update log fields updated to support new materialized columns and index
* feat: get attribute keys logic updated
* feat: logs qb changes to support new materialized columns
* feat: don't allow updating static top level fields
* feat: old logs parser updated
* feat: upgrade clickhouse
* fix: columnname format updated for materialized columns
* feat: get fields updated
* feat: minor changes
* fix: default values for distributed columns
* feat: common function for for clickhouse formatted column name
* chore: remove interesting selected fields
* fix: update clickhouse version in other place
---------
Co-authored-by: Prashant Shahi <prashant@signoz.io>
