Skip to content

Use Blackbox-Exporter

Add a new target

ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/instance: blackbox-exporter
    caascad.com/prometheus-monitor: caascad
  name: blackbox-exporter-OVERRIDEME # Exemple blackbox-exporter-Grafana
spec:
  endpoints:
  - interval: 60s
    params:
      module:
      - OVERRIDEME # see the available modules below : Example http_2xx
      target:
      - OVERRIDEME # target to check: Example https://grafana.ZONE_NAME.caascad.com
    path: /probe
    port: http
    relabelings:
    - sourceLabels:
      - __param_target
      - __param_module
      separator: /
      targetLabel: target
    - sourceLabels:
      - __param_target
      - __param_module
      separator: /
      targetLabel: instance
    - sourceLabels:
      - __param_module
      targetLabel: module
    metricRelabelings:
     - regex: 'pod'
       action: labeldrop
    scheme: http
    scrapeTimeout: 60s
  jobLabel: blackbox-exporter
  namespaceSelector:
    matchNames:
    - caascad-monitoring
  selector:
    matchLabels:
      app.kubernetes.io/instance: blackbox-exporter
      app.kubernetes.io/name: prometheus-blackbox-exporter

Caution

The .spec.endpoints[].params.target[] is an array. However you can set only one URL there. Any other URL will be ignored.

Caution

In .spec.endpoints[] the parameter .scrapeTimeout cannot be bigger than the .interval parameter.

Available modules

A module is a list of checks that will be performed on a target.

Modules already activated in Blackbox-Exporter:

  • icmp_ckeck

    Allows to test the ping on a service.

  • smtp_check

    Allows to test an smtp server by making a list of commands and checking the response. 

    - expect: "^220 ([^ ]+) ESMTP (.+)$"
- send: "EHLO ProberCheck\r"
- expect: "^250-STARTTLS"
- send: "STARTTLS\r"
- expect: "^220"
- starttls: true
- send: "EHLO ProberCheck\r"
- expect: "^250-AUTH"
- send: "QUIT\r"

  • http_2xx

    Send an http get request on the service and verify that the return code is 200.

  • http_2xx_ssl

    Send an http get request on the service and verify that the return code is 200 and ssl is present.

  • http_2xx_skip_tls_verify

    Send an http get request on the service and verify that the return code is 200 skipping the TLS certificate verification.

  • ssh_banner

    Check SSH service with the SSH banner.

  • tcp_connect

    Use the tcp protocol to check a specific port.

  • dns_udp

    Use udp protocol and checks the A field.

    The resolved_name is karma.<cluster_name>.caascad.com.

  • dns_tcp

    Use tcp protocol.

    The resolved_name is karma.<cluster_name>.caascad.com.

  • dns_soa

    Checks the SOA field.

    The resolved_name is karma.<cluster_name>.caascad.com.

Note

If you need to use other modules, contact Caascad support.

The full configuration is available here: 

kubectl get cm -n caascad-monitoring -o yaml caascad-blackbox-exporter

See the different possibilities with Blackbox-Exporter: Configuration

Deploy the ServiceMonitor

Just deploy your ServiceMonitor with kubectl (as usual).

Prometheus-operator runs an automatic discovery of ServiceMonitors and PodMonitors and will detect it soon.

Then it will get the informations it needs from the ServiceMonitor or PodMonitor.

It will get more information from the Service or Pod you selected with your monitor definition.

And it will regenerate the configuration of Prometheus. All of this is automatic.

Add custom DNS entries

In blackbox-exporter, it is possible to add custom DNS entries by filling HostAliases field.

Note

If you need to use this feature, please contact Caascad support.

Troubleshooting

Verify Prometheus Target

You can use a port-forward on Prometheus: 

kubectl -n caascad-monitoring port-forward svc/caascad-prometheus 9090:9090 &

Go to Status/Targets in order to verify if the endpoint target is up (search servicemonitor name), and see the error message if the target is down.

Ensure you check on the Grafana

Go to https://grafana.ZONE_NAME.caascad.com/

Select the Blackbox-Exporter dashboard, choose datasource Thanos-app, and verify that your target is present and up.

The probe fails (probe_success=0)

Do a port-forward on Blackbox-Exporter : 

kubectl -n caascad-monitoring port-forward \
    svc/caascad-blackbox-exporter \
    9115:9115

Do a http request specifying the following parameters:

  • target
  • module
  • debug

Example with target=https://karma.ocb-demo.caascad.com and module=http_2xx : http://localhost:9115/probe?target=https://karma.ocb-demo.caascad.com&module=http_2xx&debug=true

This request will return debug information for that probe.