how does kubernetes_sd_configs actually work?

 When a job is configured like the following:

- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (.+)

This is hard to figure out what it is saying. it turns out that this job basically

create a new url based on the formula of 

   ${__scheme__}://${__address__}/${__metrics_path__}

which in many cases __scheme__, __address__, __metrics_path__ are all use

the default value, it makes things even more confusing.

for example scheme is normally http if it is not specified

__metrics_path__ normally defaults to /metrics

__address__ normally defaults to the object IP address.

So for pod type, we are looking at http://${POD_IP}:8080/metrics by default

So, it is up to the person who configure this job to make up the part

of the url by using various actions. For example, in the above configuration

target_label: __metrics_path__ actually uses the pod annotation's

prometheus.io/path value if the pod has such annotation. if a pod does not

have such annotation, then the value of __metrics_path__ obviously will

be an empty string, which most likely wont produce a valid url for prometheus to 

retrieve any metrics.

For target_label __scheme__ in the above example, the action is to replace,

so the scheme will be basically whatever the annotation's prometheus.io/scheme

indicates.

where __address__ will be made up by two parts which was made up by the

regular expression using __address__ and pod annotation prometheus.io/port

if that pod indeed has that annotation. The default __address__ is the pod IP

address if nothing get changed.