5 Commits

Author SHA1 Message Date
Raj Kamal Singh
aad44a1037
Feat: QS: logs pipelines severity parsing processor (#4132)
* chore: update test helper for making logs

* chore: add happy case test for severity parser

* feat: get severity parsing processor working and add more happy path tests

* chore: add test for validating severity parser doesn't spam collector logs

* chore: add if condition to generated severity_parser operators

* chore: add postablePipeline validation for severity parser

* chore: minor cleanup in tests

* chore: allow trace and fatal in severity mappings
2023-12-01 17:22:22 +05:30
Raj Kamal Singh
1b6b3c2fdf
Feat: query service: logs pipelines timestamp parsing processor (#4105)
* chore: relocate tests for trace and grok parsing processor

* chore: add test for timestamp parsing processor

* feat: update PipelineOperator model for time parser fields and get tests passing

* chore: add test cases for validating time parser fails silently on mismatched logs

* chore: add helper for generating regex for strptime layouts

* feat: time_parser ignore logs whose parseFrom value doesn't match strptime layout

* chore: escape regex special chars if any in the layout string before preparing regex

* chore: add operator.If on time_parser when using layout type epoch

* chore: finish up with operator.If on time_parser for  layout type

* chore: postable pipeline validation for time parser

* chore: some cleanup

* chore: some more cleanup

* chore: add validation for strptime layouts in postable pipelines

---------

Co-authored-by: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
2023-11-29 18:55:01 +05:30
Raj Kamal Singh
626da7533e
Fix: log pipelines: generated operators should have appropriate if condition to avoid spamming collector logs (#3870)
* chore: experiment with using a tmp file for getting collector logs in simulator

* chore: collector simulator: cleaned up tmp file based collector logs capture

* chore: add test validating regex proc doesn't error for logs that dont match

* chore: return collector error logs from pipeline preview API

* chore: add test validating regex processor doesn't log errors for mismatched logs

* chore: add if condition for generated regex processors

* chore: add test case validating json parser ignore non json logs

* chore: add if condition for operator generated for json parser

* chore: add test case validating move processor ignores logs with missing field

* chore: add if condition for operator generated for move

* chore: add test case validating copy processor ignores logs with missing field

* chore: add if condition for operator generated for copy

* chore: add test case validating remove processor ignores logs with missing field

* chore: add if condition for operator generated for remove

* chore: log pipelines: ensuring json parser ignores log if json field is missing

* chore: log pipelines: ensure regex parser ignores log if field is missing
2023-11-01 22:12:35 +05:30
Raj Kamal Singh
79aef73767
Fix: Query Service: get trace parser working in log parsing pipelines (#3820)
* chore: add test for ensuring pipeline previews work for trace parser processors

* chore: updates to trace parser validation in postable pipelines

* chore: extract auth.randomHex into utils.RandomHex for reuse

* chore: get trace parser preview test passing

* chore: start with JSON serialized trace parser in test to cover deserialization

* chore: address PR feedback
2023-10-29 16:58:31 +05:30
Raj Kamal Singh
3db8a25eb9
Add support in query service for querybuilder filterset based log pipelines (#3560)
* chore: use v3.Filterset as pipeline filters in logparsing pipelines integration tests

* chore: get logparsing integration tests passing with filterset based pipeline

* chore: get all other breaking tests passing

* chore: move models.logparsingpipeline to logparsingpipeline.model

* chore: implement Valuer and Scanner interfaces for v3.FilterSet
2023-09-20 12:39:34 +05:30