How to Run Logentries in a Kubernetes Environment

Logs are great. They are easy when you’re running on one or two boxes. However, they get harder when you’re running in a modest distributed environment and even more so if you’ve embraced micro-services and containers. I’ve compiled some notes and thoughts on getting Logentries running in a Kubernetes environment.

At ReactiveOps, we use Kubernetes for our clients. Centralized logs are critical for them to understand their environments and for us to assist in troubleshooting. Sometimes, a client will have existing logging tools or services they would like us to use. This is the case with my current client, who utilizes Logentries.

Getting it running in Kubernetes was not very difficult, but some things were learned. Since Kubernetes is all about containers, it makes sense that a log collector for Logentries should also run as a container. The folks at Logentries clearly feel the same way and have made the logentries/docker-logentries image available on docker hub. The instructions discuss how to run in a container, but it is limited to straight docker environment. Kubernetes is based on docker, but you do run things differently.

Getting Logentries Up and Running

I run the Logentries collector as a DaemonSet in its own ServiceAccount under the kube-system namespace. The reason for running it as DaemonSet is to ensure we run one of these containers on every node. A dedicated ServiceAccount isn’t necessary, but helps to isolate things. You can also use a different namespace if you’d rather not add to kube-system, but it fits pretty well since this is for logging everything.

Logentries allows you to send logs to different locations named Log Sets. In order to get logs to show up in Log Sets, you will need to create the Log Set and then an access token. Since this is a secret, we’ll store it as such. Here is an example secret.yml to create the logentries_token in Kubernetes:

---
apiVersion: v1
data:
  logentries_token: <your token | base64 -w0>
kind: Secret
metadata:
  name: logentries
  namespace: kube-system

Apply this with kubectl apply -f secret.yml. Next we will create the the ServiceAccount. The serviceaccount.yml looks like this:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: logentries
  namespace: kube-system

Apply this with kubectl apply -f serviceaccount.yml. Now we are ready for the creation of the actual work horses. The daemonset.yml looks like this:

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: logentries
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: logentries
  template:
    metadata:
      name: logentries
      labels:
        app: logentries
    spec:
      serviceAccount: logentries
      containers:
      - name: logentries
        image: logentries/docker-logentries
        imagePullPolicy: Always
        securityContext:
          privileged: true
        args:
          - "-t $LOGENTRIES_TOKEN"
          - "-j"
          # - "--no-stats"
          # - "--no-logs"
          # - "--no-dockerEvents"
          # - "-a <a special string>"
          # - "--stats-interval <STATSINTERVAL>
          # - "--matchByName REGEXP"
          # - "--matchByImage REGEXP"
          # - "--skipByName REGEXP"
          # - "--skipByImage REGEXP"
        volumeMounts:
        - name: socket
          mountPath: /var/run/docker.sock
        env:
        - name: LOGENTRIES_TOKEN
          valueFrom:
            secretKeyRef:
              name: logentries
              key: logentries_token
      volumes:
        - name: socket
          hostPath:
            path: /var/run/docker.sock

Which you apply with kubectl apply -f daemonset.yml.

The above should work if you have a standard type setup, similar to the one you get from kops. If you’ve done custom installation work, you may need to adjust some things, like the location of the socket etc.

The above is the default configuration which will send container logs, container statistics, and docker events to Logentries. Should you wish to alter the logging that happens via the flags, I’ve added them in commented fashion. Details on those flags can be found on docker hub: logentries/docker-logentries/.

Observations

In general, it works as advertised. Once the DaemonSet is running, the logs will flow. Memory utilization of each container or Pod is under 100MB and CPU consumption has been modest thus far.

The logs produced by the containers are in JSON form and the -j switch makes sure that things pass through properly. The JSON that arrives appears clean and fully functional.

The first thing I noticed was that it can be tricky to find the logs for a single Deployment. For example, depending on how your images or containers are named, you will likely need to use regular expressions. The simple search will not find substrings. If we have a name of pod-staging_nginx-worker searching for nginx, it will not return results, but name=/.*nginx.*/ will.

Now that we have logs for only the pod or container we’re interested in, there is the issue of all the stats that are also part of the found results. To reduce the results to only logs from the container itself, you can add another clause: AND line. Every line logged will be stored in a JSON attribute named line, thus forcing the search to also find only those items gets us what we need: name=/.*nginx.*/ AND line.

Another issue still in search of an answer is the use case where you’d like a Log Set for application X and application Y. With a classic Logentries collector, which accesses the log files directly, you can configure which Log Set the logs from a given file will go. In a Kubernetes environment, where the logs are harvested via a docker API, that is not possible. Unfortunately, Logentries does not offer a way to slice and tag once the logs are received from docker. One possible option would be to run a Logentries container alongside each pod and utilize --matchByName or --matchByImage and --no-stats and --no-dockerEvents. Clearly, that will lead to a large number logging containers. I will be investigating this a bit more, but the default is likely the only reasonable approach for now.

In conclusion, Logentries is perfectly usable and easy to set up with Kubernetes.

What's Next?

DEVOPS CONSULTING - Leverage our decades of large-scale DevOps expertise to migrate to the cloud, automate your infrastructure and take your SaaS and web apps to the next level.

DEVOPS-AS-A-SERVICE - Partner with experts that can maintain your DevOps platform and be responsible for day-to-day operational issues, allowing you to develop and ship your product without the need for internal DevOps hires.

Our automation builds faster, our senior engineers architect better, and our repeatable process cost less than an internal hire.

Learn more about DevOps-as-a-Service

Share with your Colleagues:

    

Subscribe to Our Monthly Newsletter

Recent Posts

Most Popular

Share with your Colleagues: