Istio provides deployment annotations that direct Prometheus to scrape application pods for metrics. You might use this to scrape application specific metrics such as the reward metric used in the A/B/n tutorial.
The annotations are:
prometheus.io/scrape: "true"
prometheus.io/path: /metrics
prometheus.io/port: "9080"
Unfortunately, in some instances of Istio, the Prometheus server installed with Istio expects communication with the pod to be implemented using mTLS. We have observed this in versioms prior to Istio 1.7.0. To avoid this, you can reconfigure Prometheus as follows:
kubectl --namespace istio-system edit configmap/prometheus
Find the scrape_configs entry with job_name: 'kubernetes-pods.
Comment out the entry with a source_label of __meta_kubernetes_pod_annotation_sidecar_istio_io_status if one exists.
In this example, the last three lines have been commented out:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs: # If first two labels are present, pod should be scraped by the istio-secure job.
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
#- source_labels: [__meta_kubernetes_pod_annotation_sidecar_istio_io_status]
# action: drop
# regex: (.+)
Then restart the prometheus pod if any changes were made:
kubectl --namespace istio-system delete pod $(kubectl --namespace istio-system get pod --selector='app=prometheus' -o jsonpath='{.items[0].metadata.name}')
You should only have to do this once.