With the release of Mesos 0.27 a whole new framework for logging was introduced. The previous logger would capture std{out,err} indefinitely, which could cause the slave host to run out of disk space.

In this post I’m going to show you hopefully how to setup the new LogrotateContainerLogger.


Before we get started, I just want to say thanks to Joseph Wu. I know you’re paid to work on this, but without your contributions, none of this would exist.


Step 1) Get logrotate installed on your slaves.

My guess is that because it’s 2016, and you’re running mesos, this requirement is already satisfied by default, but maybe it isn’t, and you should go solve that.

Step 2) Find your liblogrotate_container_logger.so, and create a modules.json

The new logging framework takes advantage of the modules feature introduced in 0.21. Basically this enables you to load and configure custom shared objects via a json file that is loaded when mesos boots.

The first step in setting up this new logger is finding the shared object that contains the logger. Assuming you’re using the mesosphere packages for Centos 7, it’s here: /usr/lib/liblogrotate_container_logger.so, otherwise use your package manager to list out the files your mesos package installs to find it. Additionally, you may consider referencing a symbolic link to the explicitly versioned shared object so you get upgardes for free without changing this file.

Quick rant, if you RTFM, you’ll see that this json file supports the ideas of ‘name’ or ‘file’ attributes when specifying how the module loader should search for the library you’d like to load. ‘name’ in this case relies on LD_LIBRARY_PATH (or DYLD_LIBRARY_PATH if you’re on OS X) to be set to work. In my installation, this variable isn’t set by default, I also hate magic, this is magic, so this is why I use ‘file’, and why I had you go find the shared object.

Next step, create the json file, and put it somewhere like here: /etc/mesos-slave-modules.json. DO NOT put it in /etc/mesos-slave/, more on that later.

{
  "libraries": [{
    "file": "/usr/lib/liblogrotate_container_logger.so",
    "modules": [{
      "name": "org_apache_mesos_LogrotateContainerLogger"
    }]
  }]
}

Step 3) Setup your mesos-slave to start with some new command line arguments.

But first, more high praise:


Dear Jason Dusek and to all those who have positively contributed to mesos-init-wrapper,

Thank you. I spend a lot of time each year reading and re-writing init scripts and wrappers. You have set the bar high. This is a high-quality piece of awesome sauce, and should I ever have the pleasure of meeting any of you, I would love to buy you an adult beverage and/or food item.

Thank you for being decent and good.


So for this step you have a few options. Personally, my installation uses the mesos-init-wrapper which allows you to build out a hierarchy inside of /etc/mesos-slave to configure the command line arguments which get used when starting mesos. Additionally, you can of course use environment variables.

Method 1

1) Create /etc/mesos-slave/modules with the content:

file:///etc/mesos-slave-modules.json

2) Create /etc/mesos-slave/container_logger with the content:

org_apache_mesos_LogrotateContainerLogger

Method 2

Add the following environment variabbles to /etc/default/mesos-slave (or wherever your environment variables are stored):

MESOS_MODULES=file:///etc/mesos-slave-modules.json
MESOS_CONTAINER_LOGGER=org_apache_mesos_LogrotateContainerLogger

Step 4) Restart your mesos slave.

You should now see that when your tasks start in mesos they have a few logrotate.conf files, and the logs are being rotated.

Mesos Logrotate