When a job is configured like the following:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (.+)
This is hard to figure out what it is saying. it turns out that this job basically
create a new url based on the formula of
${__scheme__}://${__address__}/${__metrics_path__}
which in many cases __scheme__, __address__, __metrics_path__ are all use
the default value, it makes things even more confusing.
for example scheme is normally http if it is not specified
__metrics_path__ normally defaults to /metrics
__address__ normally defaults to the object IP address.
So for pod type, we are looking at http://${POD_IP}:8080/metrics by default
So, it is up to the person who configure this job to make up the part
of the url by using various actions. For example, in the above configuration
target_label: __metrics_path__ actually uses the pod annotation's
prometheus.io/path value if the pod has such annotation. if a pod does not
have such annotation, then the value of __metrics_path__ obviously will
be an empty string, which most likely wont produce a valid url for prometheus to
retrieve any metrics.
For target_label __scheme__ in the above example, the action is to replace,
so the scheme will be basically whatever the annotation's prometheus.io/scheme
indicates.
where __address__ will be made up by two parts which was made up by the
regular expression using __address__ and pod annotation prometheus.io/port
if that pod indeed has that annotation. The default __address__ is the pod IP
address if nothing get changed.
