Date June 27th, 2018 Author Vitaly Agapov

It is always mandatory to make logs from the services available for developers and other involved persons. And this access is expected to be not too complicated, without kubectl magic and dashboards (when we are talking about Kubernetes). It is quite easy to start exporting all logs from the Kubernetes cluster to Elasticsearch. The most straightforward way is:

git clone https://github.com/fluent/fluentd-kubernetes-daemonset
cd fluentd-kubernetes-daemonset
sed -i "s/elasticsearch-logging/MY-ES-HOST/" fluentd-daemonset-elasticsearch-rbac.yaml
kubectl apply -f fluentd-daemonset-elasticsearch-rbac.yaml

This works as expected but the volumes of data being exported could be larger than we want. First, the entries contain useless (for most of us) fields like container_id, pod_id, namespace_id etc. Second, there are tons of access logs from apiservers.

In order to change the fluentd behaviour we need to modify the config file. The first modification includes the new filter with record_transformer plugin. It will get rid of useless fields and save large amount of disk space for keeping the Elastic indexes.

    <filter kubernetes.**>
      @type record_transformer
      enable_ruby
      # Remove unwanted metadata keys
      <record>
      for_remove ${record["docker"].delete("container_id");record["kubernetes"].delete("annotations"); record["kubernetes"]["labels"].delete("pod-template-hash"); record["kubernetes"].delete("master_url"); record["kubernetes"].delete("pod_id"); record["kubernetes"].delete("namespace_id");}
      </record>
      remove_keys for_remove
    </filter>

The second simple modification is to add the excluding rule to the source to get rid of kube-apiserver logs:

    <source>
      @type tail
      @id in_tail_container_logs
      path /var/log/containers/*.log
      exclude_path ["/var/log/containers/kube-apiserver*"]
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      format json
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </source>

And the last step is creating the ConfigMap and modifying the DaemonSet to use this ConfigMap as volume mount.

Update: Also it worth adding a new env variable to the DaemonSet container spec: FLUENT_UID=0. Otherwise an error like unexpected error error_class=Errno::EACCES error=<Errno::EACCES: Permission denied @ rb_sysopen – /var/log/fluentd-containers.log.pos> can occur.

Tags: Docker, Elasticsearch, Fluentd, Kubernetes
Category: Kubernetes | No comments »

Comments

Leave a comment