Yet another Fluentd deployment for Kubernetes

Date June 27th, 2018 Author Vitaly Agapov

Accept the pain, but don't accept that you deserved it.

Brandon Sanderson «Oathbringer»

fluentd-k8s-esIt is always mandatory to make logs from the services available for developers and other involved persons. And this access is expected to be not too complicated, without kubectl magic and dashboards (when we are talking about Kubernetes). It is quite easy to start exporting all logs from the Kubernetes cluster to Elasticsearch. The most straightforward way is:


git clone
cd fluentd-kubernetes-daemonset
sed -i "s/elasticsearch-logging/MY-ES-HOST/" fluentd-daemonset-elasticsearch-rbac.yaml
kubectl apply -f fluentd-daemonset-elasticsearch-rbac.yaml

This works as expected but the volumes of data being exported could be larger than we want. First, the entries contain useless (for most of us) fields like container_id, pod_id, namespace_id etc. Second, there are tons of access logs from apiservers.


In order to change the fluentd behaviour we need to modify the config file. The first modification includes the new filter with record_transformer plugin. It will get rid of useless fields and save large amount of disk space for keeping the Elastic indexes.

    <filter kubernetes.**>
      @type record_transformer
      # Remove unwanted metadata keys
      for_remove ${record["docker"].delete("container_id");record["kubernetes"].delete("annotations"); record["kubernetes"]["labels"].delete("pod-template-hash"); record["kubernetes"].delete("master_url"); record["kubernetes"].delete("pod_id"); record["kubernetes"].delete("namespace_id");}
      remove_keys for_remove

The second simple modification is to add the excluding rule to the source to get rid of kube-apiserver logs:

      @type tail
      @id in_tail_container_logs
      path /var/log/containers/*.log
      exclude_path ["/var/log/containers/kube-apiserver*"]
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      format json
      time_format %Y-%m-%dT%H:%M:%S.%NZ

And the last step is creating the ConfigMap and modifying the DaemonSet to use this ConfigMap as volume mount.

Update: Also it worth adding a new env variable to the DaemonSet container spec: FLUENT_UID=0. Otherwise an error like unexpected error error_class=Errno::EACCES error=<Errno::EACCES: Permission denied @ rb_sysopen – /var/log/fluentd-containers.log.pos> can occur.

Tags: , , ,
Category: Kubernetes | No comments »


Leave a comment

 Comment Form