Remote Write Configuration

Configuring Remote Write to Push Prometheus Metrics

To push the Prometheus metrics to Grafana Cloud using remote-write, use the following sample configuration:

prometheus remote-write target demo
   url https://prometheus-us-central1.grafana.net/api/prom/push
   basic-auth username 725569
   basic-auth password $8$ntCDRl2FkMDlm8mj9FohYwTuy/jo+7Cka0msfP2qW3Y=
   proxy-url http://proxy-wsa.esl.cisco.com:80
   exit

NOTES:

  • url —Specify the target URL of Grafana Cloud.

  • basic-auth username —Specify the username in Confd.

  • basic-auth password —Specify the password in Confd. The password is encrypted in Confd and passed to the metrics helm chart.

  • proxy-url —Specify the optional proxy URL to access Grafana Cloud in Confd.

Configuring Prometheus Parameters

To configure the Prometheus parameters to Grafana Cloud using remote-write, use the following sample configuration:

  • Remote Timeout—The remote-timeout-seconds command sets the timeout for requests to the remote write endpoint, in seconds. Default: 30 seconds.

    The following is a sample configuration:

    prometheus remote-write target demo
       remote-timeout-seconds 60
       exit
  • Queue Configuration—The queue-config command configures the queue used to write to remote storage.

    The following is a sample configuration:

    prometheus remote-write target demo
       ...
       queue-config capacity 500
       queue-config max-shards 100
       queue-config min-shards 2
       queue-config max-samples-per-send 300
       queue-config batch-send-deadline-seconds 10
       exit

    NOTES:

    • queue-config capacity : Specify the number of samples to buffer per shard. Default: 2500.

      It is recommended to have adequate capacity in each shard to buffer several requests. The adequate capacity can maintain the throughput while processing occasional slow remote requests.

    • queue-config max-shards : Specify the maximum number of shards. Default: 200.

    • queue-config min-shards : Specify the minimum number of shards. Default: 1.

    • queue-config max-samples-per-send : Specify the maximum number of samples per send. Default: 500.

    • queue-config batch-send-deadline-seconds : Specify the maximum time in seconds that a sample will wait in buffer. Default: 5 seconds.

  • Relabel Configuration—The relabel-configs command defines a list of relabel configurations before the metrics are written to remote storage. The relabeling feature in Prometheus rewrites the label set of a target dynamically.

    The following is a sample configuration:

    prometheus remote-write target demo
     ...
     relabel-configs test1
        target-label test1_label
        regex        (.+);(.+)
        replacement  ${1}@${2}
        action       replace
        source-labels container
        source-labels pod
        exit
     exit

NOTES:

  • target-label : Specify the label to which the resulting value is written in a replace action.

  • regex : Specify the regular expression against which the extracted value is matched.

    Default = (.*)

  • replacement : Specify the replacement value against which a regex replace is performed if the regular expression matches.

    Default = $1

  • action : Specify the replace, keep, or drop action to perform based on regex matching.

    Default = replace

  • source-labels : Specify the source label to select values from existing labels.

  • Multiple relabeling steps can be configured per scrape configuration. The steps are applied to the label set of each target in order of appearance in the configuration file.

  • Note that Prometheus will drop any label with empty value, hence use the labels with caution.