Feature Description
The remote-write feature is enhanced to support new Prometheus parameters for CNDP Grafana Cloud integration. These parameters enable the user to fine tune their setup.
The enhanced CEE dashboard has new panels to display the remote write metrics.
This enhancement also supports new alerts fluent-proxy-output-retries-failed
and prometheus-remote-write-behind
. See the UCC SMI Operations Guide > Alerts Reference for more information on alerts.
-
Remote Timeout:
The remote-timeout-seconds command sets the timeout for requests to the remote write endpoint, in seconds. Default: 30 seconds.
The following is a sample configuration:
prometheus remote-write target demo remote-timeout-seconds 60 exit
-
Queue Configuration:
The queue-config command configures the queue used to write to remote storage.
The following is a sample configuration:
prometheus remote-write target demo ... queue-config capacity 500 queue-config max-shards 100 queue-config min-shards 2 queue-config max-samples-per-send 300 queue-config batch-send-deadline-seconds 10 exit
NOTES:
-
queue-config capacity : Specify the number of samples to buffer per shard. Default: 2500.
It is recommended to have adequate capacity in each shard to buffer several requests. The adequate capacity can maintain the throughput while processing occasional slow remote requests.
-
queue-config max-shards : Specify the maximum number of shards. Default: 200.
-
queue-config min-shards : Specify the minimum number of shards. Default: 1.
-
queue-config max-samples-per-send : Specify the maximum number of samples per send. Default: 500.
-
queue-config batch-send-deadline-seconds : Specify the maximum time in seconds that a sample will wait in buffer. Default: 5 seconds.
-
-
Relabel Configuration:
The relabel-configs command defines a list of relabel configurations before the metrics are written to remote storage. The relabeling feature in Prometheus rewrites the label set of a target dynamically.
The following is a sample configuration:
prometheus remote-write target demo ... relabel-configs test1 target-label test1_label regex (.+);(.+) replacement ${1}@${2} action replace source-labels container source-labels pod exit exit
NOTES:
-
target-label : Specify the label to which the resulting value is written in a replace action.
-
regex : Specify the regular expression against which the extracted value is matched.
Default = (.*)
-
replacement : Specify the replacement value against which a regex replace is performed if the regular expression matches.
Default = $1
-
action : Specify the replace, keep, or drop action to perform based on regex matching.
Default = replace
-
source-labels : Specify the source label to select values from existing labels.
-
Multiple relabeling steps can be configured per scrape configuration. The steps are applied to the label set of each target in order of appearance in the configuration file.
-
Note that Prometheus will drop any label with empty value, hence use the labels with caution.