Feature Description

The remote-write feature is enhanced to support new Prometheus parameters for CNDP Grafana Cloud integration. These parameters enable the user to fine tune their setup.

The enhanced CEE dashboard has new panels to display the remote write metrics.

This enhancement also supports new alerts fluent-proxy-output-retries-failed and prometheus-remote-write-behind. See the UCC SMI Operations Guide > Alerts Reference for more information on alerts.

  • Remote Timeout:

    The remote-timeout-seconds command sets the timeout for requests to the remote write endpoint, in seconds. Default: 30 seconds.

    The following is a sample configuration:

    prometheus remote-write target demo
       remote-timeout-seconds 60
       exit
  • Queue Configuration:

    The queue-config command configures the queue used to write to remote storage.

    The following is a sample configuration:

    prometheus remote-write target demo
       ...
       queue-config capacity 500
       queue-config max-shards 100
       queue-config min-shards 2
       queue-config max-samples-per-send 300
       queue-config batch-send-deadline-seconds 10
       exit

    NOTES:

    • queue-config capacity : Specify the number of samples to buffer per shard. Default: 2500.

      It is recommended to have adequate capacity in each shard to buffer several requests. The adequate capacity can maintain the throughput while processing occasional slow remote requests.

    • queue-config max-shards : Specify the maximum number of shards. Default: 200.

    • queue-config min-shards : Specify the minimum number of shards. Default: 1.

    • queue-config max-samples-per-send : Specify the maximum number of samples per send. Default: 500.

    • queue-config batch-send-deadline-seconds : Specify the maximum time in seconds that a sample will wait in buffer. Default: 5 seconds.

  • Relabel Configuration:

    The relabel-configs command defines a list of relabel configurations before the metrics are written to remote storage. The relabeling feature in Prometheus rewrites the label set of a target dynamically.

    The following is a sample configuration:

    prometheus remote-write target demo
     ...
     relabel-configs test1
        target-label test1_label
        regex        (.+);(.+)
        replacement  ${1}@${2}
        action       replace
        source-labels container
        source-labels pod
        exit
     exit

NOTES:

  • target-label : Specify the label to which the resulting value is written in a replace action.

  • regex : Specify the regular expression against which the extracted value is matched.

    Default = (.*)

  • replacement : Specify the replacement value against which a regex replace is performed if the regular expression matches.

    Default = $1

  • action : Specify the replace, keep, or drop action to perform based on regex matching.

    Default = replace

  • source-labels : Specify the source label to select values from existing labels.

  • Multiple relabeling steps can be configured per scrape configuration. The steps are applied to the label set of each target in order of appearance in the configuration file.

  • Note that Prometheus will drop any label with empty value, hence use the labels with caution.