Remote Write Configuration
Configuring Remote Write to Push Prometheus Metrics
To push the Prometheus metrics to Grafana Cloud using remote-write, use the following sample configuration:
prometheus remote-write target demo
url https://prometheus-us-central1.grafana.net/api/prom/push
basic-auth username 725569
basic-auth password $8$ntCDRl2FkMDlm8mj9FohYwTuy/jo+7Cka0msfP2qW3Y=
proxy-url http://proxy-wsa.esl.cisco.com:80
exit
NOTES:
-
url —Specify the target URL of Grafana Cloud.
-
basic-auth username —Specify the username in Confd.
-
basic-auth password —Specify the password in Confd. The password is encrypted in Confd and passed to the metrics helm chart.
-
proxy-url —Specify the optional proxy URL to access Grafana Cloud in Confd.
Configuring Prometheus Parameters
To configure the Prometheus parameters to Grafana Cloud using remote-write, use the following sample configuration:
-
Remote Timeout—The remote-timeout-seconds command sets the timeout for requests to the remote write endpoint, in seconds. Default: 30 seconds.
The following is a sample configuration:
prometheus remote-write target demo remote-timeout-seconds 60 exit
-
Queue Configuration—The queue-config command configures the queue used to write to remote storage.
The following is a sample configuration:
prometheus remote-write target demo ... queue-config capacity 500 queue-config max-shards 100 queue-config min-shards 2 queue-config max-samples-per-send 300 queue-config batch-send-deadline-seconds 10 exit
NOTES:
-
queue-config capacity : Specify the number of samples to buffer per shard. Default: 2500.
It is recommended to have adequate capacity in each shard to buffer several requests. The adequate capacity can maintain the throughput while processing occasional slow remote requests.
-
queue-config max-shards : Specify the maximum number of shards. Default: 200.
-
queue-config min-shards : Specify the minimum number of shards. Default: 1.
-
queue-config max-samples-per-send : Specify the maximum number of samples per send. Default: 500.
-
queue-config batch-send-deadline-seconds : Specify the maximum time in seconds that a sample will wait in buffer. Default: 5 seconds.
-
-
Relabel Configuration—The relabel-configs command defines a list of relabel configurations before the metrics are written to remote storage. The relabeling feature in Prometheus rewrites the label set of a target dynamically.
The following is a sample configuration:
prometheus remote-write target demo ... relabel-configs test1 target-label test1_label regex (.+);(.+) replacement ${1}@${2} action replace source-labels container source-labels pod exit exit
NOTES:
-
target-label : Specify the label to which the resulting value is written in a replace action.
-
regex : Specify the regular expression against which the extracted value is matched.
Default = (.*)
-
replacement : Specify the replacement value against which a regex replace is performed if the regular expression matches.
Default = $1
-
action : Specify the replace, keep, or drop action to perform based on regex matching.
Default = replace
-
source-labels : Specify the source label to select values from existing labels.
-
Multiple relabeling steps can be configured per scrape configuration. The steps are applied to the label set of each target in order of appearance in the configuration file.
-
Note that Prometheus will drop any label with empty value, hence use the labels with caution.