15 Commits

Author SHA1 Message Date
Raj Kamal Singh
161a69fbe9
chore: remove workaround for supporting pipeline filters using attribs with . replaced with _ (#5405) 2024-07-02 17:14:08 +05:30
Raj Kamal Singh
86ff865842
Fix: logs pipelines: ignore coalesce op when generating nil check for add value expressions (#4305)
* chore: panic if agent config recommendation can't be generated

* chore: add case with coalesce op in field nil check generation tests

* fix: ignore expr ast member nodes that contain coalesce op in them
2023-12-29 18:21:01 +05:30
Raj Kamal Singh
7fed80b145
Fix: log pipelines contains and ncontains filters should be case insensitive (#4299)
* chore: add test validating contains and ncontains in pipeline filter are case insensitive

* chore: qb2expr: translate contains and ncontains to case insensitive comparison

* chore: minor cleanup
2023-12-28 19:44:17 +05:30
Raj Kamal Singh
bcd6ac47f7
Fix: Logs: Pipelines: add nil check for grok parser parseFrom field in generated collector config (#4286)
* chore: add test validating grok parser doesn't spam logs if parse from is missing

* chore: add nil check for grok parser parseFrom
2023-12-28 11:03:31 +05:30
Raj Kamal Singh
ec27916fa5
Fix: QS: Log Pipelines: generate correct nil checks for operators referencing fields like attributes["http.status.code"] (#4284)
* chore: add test validating that using paths like attributes["http.method"] works

* chore: refactor nil checks on processor fields generated for pipelines

* chore: get nil checks working on paths like attributes["http.method"]

* chore: use parsed AST for generating nil checks for add operator value expressions

* chore: some cleanup

* chore: some more cleanup

* chore: some more cleanup

* chore: some more cleanup

---------

Co-authored-by: Nityananda Gohain <nityanandagohain@gmail.com>
2023-12-28 10:31:36 +05:30
Raj Kamal Singh
581bd07b35
Fix: query service: escape dollars in collector config generated for pipelines. (#4242)
* chore: add test validating pipeline config containing dollar works

* chore: collector simulator: use fileprovider with expandconverter like actual collector

* chore: get tests passing

* chore: take a configGenerator func in simulation and unify config generation logic

* fix: escape $ in config generated for log pipelines
2023-12-19 21:24:53 +05:30
Raj Kamal Singh
1b6b3c2fdf
Feat: query service: logs pipelines timestamp parsing processor (#4105)
* chore: relocate tests for trace and grok parsing processor

* chore: add test for timestamp parsing processor

* feat: update PipelineOperator model for time parser fields and get tests passing

* chore: add test cases for validating time parser fails silently on mismatched logs

* chore: add helper for generating regex for strptime layouts

* feat: time_parser ignore logs whose parseFrom value doesn't match strptime layout

* chore: escape regex special chars if any in the layout string before preparing regex

* chore: add operator.If on time_parser when using layout type epoch

* chore: finish up with operator.If on time_parser for  layout type

* chore: postable pipeline validation for time parser

* chore: some cleanup

* chore: some more cleanup

* chore: add validation for strptime layouts in postable pipelines

---------

Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
2023-11-29 18:55:01 +05:30
Raj Kamal Singh
988ede7776
Fix/pipelines temp work around for supporting dots in resource keys (#4064)
* chore: logs pipelines: add test for validating workaround for working with dots in keys

* fix: temp workaround for supporting pipeline filters using names with dots converted to underscore
2023-11-26 12:57:23 +05:30
Raj Kamal Singh
f5b1d4146f
Fix: pipelines: string ops on missing attribs in pipeline filters should not spam warnings (#4049)
* chore: add test validating collector doesn't spam logs for string ops on missing attributes

* fix: pipelines filter: check if attrib is not nil before running string operator

* chore: do a nil check for all but == and != ops
2023-11-24 17:00:39 +05:30
Raj Kamal Singh
3e65543b5f
Fix: resource filters should work in logs pipelines (#3889)
* chore: add test validating resource based filters work in logs pipelines

* fix: get resource filters working in logs pipelines
2023-11-03 18:42:03 +05:30
Raj Kamal Singh
050b866173
chore: return warning logs too from collector simulator (#3888)
* chore: return warning logs too from collector simulator

* chore: also return collector logs in preview API response to help with debugging

---------

Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
2023-11-03 17:41:09 +05:30
Raj Kamal Singh
626da7533e
Fix: log pipelines: generated operators should have appropriate if condition to avoid spamming collector logs (#3870)
* chore: experiment with using a tmp file for getting collector logs in simulator

* chore: collector simulator: cleaned up tmp file based collector logs capture

* chore: add test validating regex proc doesn't error for logs that dont match

* chore: return collector error logs from pipeline preview API

* chore: add test validating regex processor doesn't log errors for mismatched logs

* chore: add if condition for generated regex processors

* chore: add test case validating json parser ignore non json logs

* chore: add if condition for operator generated for json parser

* chore: add test case validating move processor ignores logs with missing field

* chore: add if condition for operator generated for move

* chore: add test case validating copy processor ignores logs with missing field

* chore: add if condition for operator generated for copy

* chore: add test case validating remove processor ignores logs with missing field

* chore: add if condition for operator generated for remove

* chore: log pipelines: ensuring json parser ignores log if json field is missing

* chore: log pipelines: ensure regex parser ignores log if field is missing
2023-11-01 22:12:35 +05:30
Raj Kamal Singh
3db8a25eb9
Add support in query service for querybuilder filterset based log pipelines (#3560)
* chore: use v3.Filterset as pipeline filters in logparsing pipelines integration tests

* chore: get logparsing integration tests passing with filterset based pipeline

* chore: get all other breaking tests passing

* chore: move models.logparsingpipeline to logparsingpipeline.model

* chore: implement Valuer and Scanner interfaces for v3.FilterSet
2023-09-20 12:39:34 +05:30
Raj Kamal Singh
f450d71a25
fix: avoid pipeline builder panic when first op is disabled (#3500)
* fix: avoid pipeline builder panic if first op is disabled

* chore: use pipeline builder test to trigger getOperator panic

---------

Co-authored-by: Nityananda Gohain <nityanandagohain@gmail.com>
2023-09-07 17:47:09 +05:30
Nityananda Gohain
1a3e46cecd
feat: integrate pipelines API (#2457)
* chore: integrate pipelines API

* fix: limit support integrated in pipelines

* fix: interface to string

* fix: json parser and allow deleting all pipelines

* fix: output modified if operators are disabled

* fix: validation updated for operators

* fix: expression check added

* fix: regex expression check added

* fix: remove operator validation updated

* fix: tests updated for pipeline builder

* fix: fix error messages in http handler

* fix: dont return payload if there is an error

* fix: extracting userId from context moved to auth package

* fix: api errors moved to http handler

* fix: get version logic updated

* fix: deployment result message updated

* fix: pipeline builder edgecase fixed and tests updated

* fix: get failing postablePipeline tests to pass

---------

Co-authored-by: Vishal Sharma <makeavish786@gmail.com>
Co-authored-by: Raj <rkssisodiya@gmail.com>
2023-07-31 21:34:42 +05:30