Logs¶

Promtail is the source of logs. The Promtail agent (daemonset) is installed in each K8s customer environment (in caascad-logging namespace).

It allows to :

On Flexible Engine cloud provider, by default we collect :

with Kubernetes discovery, the logs of all the pods deployed that log to standard and error output.
with static config, the logs of Kubelet and Kube-proxy

On AWS cloud provider, by default we collect :

with Kubernetes discovery, the logs of all the pods deployed that log to standard and error output.
with static config, the logs of Kubelet (Kube-proxy unlike on Flexible Engine is recovered with pods logs)

Promtail sends logs to Loki deployed in the customer administration environment.

We specify here some important labels, it is not an exhaustive list.

For all logs :

For logs from Kubernetes discovery :

pods labels (Example if pod has label app.kubernetes.io/instance: blackbox-exporter, log label will be app_kubernetes_io_instance=blackbox-exporter)
container name (label container)
namespace (label namespace)
job (label job=namespace/service)

For logs from Kubelet :

For logs from Kube-proxy (only for Flexible Engine) :

Promtail can be configured with several stages which can allow to do different actions on the logs.

Features already available in Caascad :

Drop logs

We can deactivate the collection of logs according to their labels, their content.
Custom metrics from logs

We can create Prometheus metric from logs (from content or/and labels).
Create label from logs content

From the content of logs, Promtail can parse it and add a label on the logs sent to Loki.
Set timestamp

The timestamp action can change the timestamp of a log line before it is sent to Loki.

When a timestamp stage is not present, the timestamp of a log line in Loki defaults to the time when the log entry is scraped.

When a timestamp stage is present, the timestamp of a log line in Loki is timestamp log line.

Note

All features are documented here: promtail/stages

There is a single Loki instance deployed in customer administration environment.

All Promtails in customer environments send logs to this Loki.

Retention¶

Retention is enabled in Loki. Logs exceeding retention are deleted.

Note

When an administration environment is provisioned, customer specific retention period is applied for logs retention.

If no cutomer specific retention is specified, default retention period will be applied: 30 days.

At any time this retention can be modified through change request made to Caascad customer support.