Prometheus probe

Prometheus probe allows users to run Prometheus queries and match the resulting output against specific conditions. The intent behind this probe is to allow users to define metrics-based SLOs in a declarative way and determine the experiment verdict based on its success. The probe runs the query on a Prometheus server defined by the endpoint, and checks whether the output satisfies the specified criteria. A PromQL query needs to be provided, whose outcome is then used for the probe validation.

YAML only feature

In case of complex queries that span multiple lines, the queryPath attribute can be used to provide the link to a file consisting of the query. This file can be made available in the experiment pod via a ConfigMap resource, with the ConfigMap being passed in the ChaosEngine or the ChaosExperiment CR. Also, query and queryPath attributes are mutually exclusive. Refer to the probe schema here.

Defining the probe

You can define the probes at .spec.experiments[].spec.probe path inside the chaos engine.

kind: Workflow
apiVersion: argoproj.io/v1alpha1
spec:
  templates:
    - inputs:
        artifacts:
          - raw:
              data: |
                apiVersion: litmuschaos.io/v1alpha1
                kind: ChaosEngine
                spec:
                  experiments:
                    - spec:
                        probe:
                          ####################################
                          Probes are defined here
                          ####################################

Schema

Listed below is the probe schema for the Prometheus probe, with properties shared across all the probes and properties unique to the Prometheus probe.

Field	Description	Type	Range	Notes
name	Flag to hold the name of the probe	Mandatory	N/A type: string	The name holds the name of the probe. It can be set based on the usecase
type	Flag to hold the type of the probe	Mandatory	httpProbe, k8sProbe, cmdProbe, promProbe	The type supports four types of probes. It can one of the httpProbe, k8sProbe, cmdProbe, promProbe
mode	Flag to hold the mode of the probe	Mandatory	SOT, EOT, Edge, Continuous, OnChaos	The mode supports five modes of probes. It can one of the SOT, EOT, Edge, Continuous, OnChaos
endpoint	Flag to hold the prometheus endpoints for the promProbe	Mandatory	N/A type: string	The endpoint contains the prometheus endpoints
query	Flag to hold the promql query for the promProbe	Mandatory	N/A type: string	The query contains the promql query to extract out the desired prometheus metrics via running it on the given prometheus endpoint
queryPath	Flag to hold the path of the promql query for the promProbe	Optional	N/A type: string	The queryPath field is used in case of complex queries that spans multiple lines, the queryPath attribute can be used to provide the path to a file consisting of the same. This file can be made available to the experiment pod via a ConfigMap resource, with the ConfigMap name being defined in the ChaosEngine OR the ChaosExperiment CR.

Comparator

Field	Description	Type	Range	Notes
type	Flag to hold type of the data used for comparison	Optional	float	The type contains type of data, which should be compared as part of comparison operation. Prometheus probe only compares with float data.
criteria	Flag to hold criteria for the comparison	Mandatory	It supports >=, <=, ==, >, <, !=, oneOf, between for int & float type. And equal, notEqual, contains, matches, notMatches, oneOf for string type.	The criteria contains criteria of the comparison, which should be fulfill as part of comparison operation.
value	Flag to hold value for the comparison	Mandatory	N/A type: string	The value contains value of the comparison, which should follow the given criteria as part of comparison operation.

Run properties

Field	Description	Type	Range	Notes
probeTimeout	Flag to hold the timeout of the probe	Mandatory	N/A type: integer	The probeTimeout represents the time limit for the probe to execute the specified check and return the expected data
retry	Flag to hold the retry count of the probe	Mandatory	N/A type: integer	The retry contains the number of times a check is re-run upon failure in the first attempt before declaring the probe status as failed.
interval	Flag to hold the interval of the probe	Mandatory	N/A type: integer	The interval contains the interval for which probes waits between subsequent retries
probePollingInterval	Flag to hold the polling interval for the probes (applicable for all modes)	Optional	N/A type: integer	The probePollingInterval contains the time interval for which continuous probe should be sleep after each iteration
initialDelaySeconds	Flag to hold the initial delay interval for the probes	Optional	N/A type: integer	The initialDelaySeconds represents the initial waiting time interval for the probes.
stopOnFailure	Flags to hold the stop or continue the experiment on probe failure	Optional	N/A type: boolean	The stopOnFailure can be set to true/false to stop or continue the experiment execution after probe fails

Definition

probe:
  - name: "check-probe-success"
    type: "promProbe"
    promProbe/inputs:
      endpoint: "<prometheus-endpoint>"
      query: "<promql-query>"
      comparator:
        criteria: "==" #supports >=,<=,>,<,==,!= comparison
        value: "<value-for-criteria-match>"
    mode: "Edge"
    runProperties:
      probeTimeout: 5
      interval: 5
      retry: 1