* chore: add test validating pipeline alias collisions dont lead to bad config recommendations
* chore: emit error log on detecting duplicate processors in generated config
* chore: ensure collector config processor names for pipelines are unique
* chore: minor cleanups
* chore: add explicit happy case test for regex parsing processor
* chore: add explicit happy case test for grok parsing processor
* chore: add explicit happy case test for JSON parsing processor
* chore: log pipelines: move trace parser test to processors_test.go
* chore: add explicit happy case test for ADD processor
* chore: add explicit happy case test for remove processor
* chore: add explicit happy case test for copy processor
* chore: add explicit happy case test for move processor
* fix: fix broken grok parser test: change test int value to int64
---------
Co-authored-by: Nityananda Gohain <nityanandagohain@gmail.com>
* chore: refactor: inject sqlx.DB into opamp.initDB instead of DB file name
* chore: reorganize test utils a little
* chore: add test validating pipelines for installed integrations show up in pipelines list
* chore: get basic integration pipelines testcase passing
* chore: reconcile experimental changes with latest state of develop
* chore: add integration test for reordering of pipelines
* chore: marker for integration pipelines using Id
* chore: hookup propagation of installed integration pipelines by opamp
* chore: add util for mapping slices
* chore: add support for reordering integration pipelines
* chore: exclude user saved integration pipelines if no longer installed
* chore: flesh out rest of intgeration pipelines scenarios
* chore: handle scenario when an integration is installed before any pipelines exist
* chore: notify agentConf of update after uninstalling an integration
* chore: some minor cleanup
* chore: some more cleanup
* chore: update ee server for changed controllers
* chore: some more cleanup
* chore: change builtin integration id prefix to avoid using colons that break yaml
* chore: update builtin integration id in test
* chore: panic if agent config recommendation can't be generated
* chore: add case with coalesce op in field nil check generation tests
* fix: ignore expr ast member nodes that contain coalesce op in them
* chore: add test validating contains and ncontains in pipeline filter are case insensitive
* chore: qb2expr: translate contains and ncontains to case insensitive comparison
* chore: minor cleanup
* chore: add test validating that using paths like attributes["http.method"] works
* chore: refactor nil checks on processor fields generated for pipelines
* chore: get nil checks working on paths like attributes["http.method"]
* chore: use parsed AST for generating nil checks for add operator value expressions
* chore: some cleanup
* chore: some more cleanup
* chore: some more cleanup
* chore: some more cleanup
---------
Co-authored-by: Nityananda Gohain <nityanandagohain@gmail.com>
* chore: add test validating pipeline config containing dollar works
* chore: collector simulator: use fileprovider with expandconverter like actual collector
* chore: get tests passing
* chore: take a configGenerator func in simulation and unify config generation logic
* fix: escape $ in config generated for log pipelines
* chore: update test helper for making logs
* chore: add happy case test for severity parser
* feat: get severity parsing processor working and add more happy path tests
* chore: add test for validating severity parser doesn't spam collector logs
* chore: add if condition to generated severity_parser operators
* chore: add postablePipeline validation for severity parser
* chore: minor cleanup in tests
* chore: allow trace and fatal in severity mappings
* chore: relocate tests for trace and grok parsing processor
* chore: add test for timestamp parsing processor
* feat: update PipelineOperator model for time parser fields and get tests passing
* chore: add test cases for validating time parser fails silently on mismatched logs
* chore: add helper for generating regex for strptime layouts
* feat: time_parser ignore logs whose parseFrom value doesn't match strptime layout
* chore: escape regex special chars if any in the layout string before preparing regex
* chore: add operator.If on time_parser when using layout type epoch
* chore: finish up with operator.If on time_parser for layout type
* chore: postable pipeline validation for time parser
* chore: some cleanup
* chore: some more cleanup
* chore: add validation for strptime layouts in postable pipelines
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* chore: logs pipelines: add test for validating workaround for working with dots in keys
* fix: temp workaround for supporting pipeline filters using names with dots converted to underscore
* chore: add test validating collector doesn't spam logs for string ops on missing attributes
* fix: pipelines filter: check if attrib is not nil before running string operator
* chore: do a nil check for all but == and != ops
* fix: improve user telemetry
- move GetEmailFromJwt to common function
- update AttachJwtToContext() to use standard way of attaching value to context
- update userEmail in every possible sendEvent call
- send groupId in sendEvent call for SaaS operator analytics
* chore: added DEFAULT_CLOUD_EMAIL const
* chore: add AttachJwtToContext to analytics middleware
* test: added AttachJwtToContext to logs pipelines
* chore: return warning logs too from collector simulator
* chore: also return collector logs in preview API response to help with debugging
---------
Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
* chore: experiment with using a tmp file for getting collector logs in simulator
* chore: collector simulator: cleaned up tmp file based collector logs capture
* chore: add test validating regex proc doesn't error for logs that dont match
* chore: return collector error logs from pipeline preview API
* chore: add test validating regex processor doesn't log errors for mismatched logs
* chore: add if condition for generated regex processors
* chore: add test case validating json parser ignore non json logs
* chore: add if condition for operator generated for json parser
* chore: add test case validating move processor ignores logs with missing field
* chore: add if condition for operator generated for move
* chore: add test case validating copy processor ignores logs with missing field
* chore: add if condition for operator generated for copy
* chore: add test case validating remove processor ignores logs with missing field
* chore: add if condition for operator generated for remove
* chore: log pipelines: ensuring json parser ignores log if json field is missing
* chore: log pipelines: ensure regex parser ignores log if field is missing
* chore: add test for ensuring pipeline previews work for trace parser processors
* chore: updates to trace parser validation in postable pipelines
* chore: extract auth.randomHex into utils.RandomHex for reuse
* chore: get trace parser preview test passing
* chore: start with JSON serialized trace parser in test to cover deserialization
* chore: address PR feedback
* chore: add test for previewing pipeline with grok parser
* chore: import grok parser in logparsing preview to ensure its a registered logtransform op
* chore: add dependencies for grok parsing operator
* chore: add integration test for log pipelines being recommended to agents on connection
* chore: agentConf.Manager as AgentConfigProvider
* feat: logparsingpipelines as an AgentFeature
* chore: some cleanup
* feat: update agentConf init in ee query-service
* chore: more cleanup and move opamp/logspipeline -> logparsingpipeline/collector_config.go
* chore: some more cleanup
* chore: set agent.RemoteConfig after broadcasting conf to all agents
* chore: add test scenario for post remote conf application
* feat: add logsparsingpipeline controller api for pipeline previews
* feat: add http handler and route for generating pipeline previews
* feat: use a response model for pipeline previews response
* chore: rename model.GetLogsResponse -> model.SignozLog for use in both requests and responses
* feat: add test for simulating log pipelines processing
* feat: get pipeline preview tests passing
* chore: cleanup
* chore: use v3.Filterset as pipeline filters in logparsing pipelines integration tests
* chore: get logparsing integration tests passing with filterset based pipeline
* chore: get all other breaking tests passing
* chore: move models.logparsingpipeline to logparsingpipeline.model
* chore: implement Valuer and Scanner interfaces for v3.FilterSet
* chore: add integration test for log parsing pipelines validation
* chore: add helpers for creating unauthorized, unavailable & not found api errors
* chore: return *model.APIError from logpipeline and agentConf functions
* chore: some cleanup
* chore: some more cleanup
* chore: one more round of cleanups
* fix: avoid pipeline builder panic if first op is disabled
* chore: use pipeline builder test to trigger getOperator panic
---------
Co-authored-by: Nityananda Gohain <nityanandagohain@gmail.com>
* chore: integrate pipelines API
* fix: limit support integrated in pipelines
* fix: interface to string
* fix: json parser and allow deleting all pipelines
* fix: output modified if operators are disabled
* fix: validation updated for operators
* fix: expression check added
* fix: regex expression check added
* fix: remove operator validation updated
* fix: tests updated for pipeline builder
* fix: fix error messages in http handler
* fix: dont return payload if there is an error
* fix: extracting userId from context moved to auth package
* fix: api errors moved to http handler
* fix: get version logic updated
* fix: deployment result message updated
* fix: pipeline builder edgecase fixed and tests updated
* fix: get failing postablePipeline tests to pass
---------
Co-authored-by: Vishal Sharma <makeavish786@gmail.com>
Co-authored-by: Raj <rkssisodiya@gmail.com>
