So I quit my job. That’s fun. Anyways, I’ve been working on documenting things and I came around to putting all the things I’ve been learning about Jetty on to paper. Needless to say, it got colorful - here’s the after math - I hope you enjoy.


1) 9.2.x used to not work on Java 7 - now it is rumored it works on java 7 - we should try building a blueprint with the latest version of jetty on java 7 to see if it works.

2) The command start-log-file is um… misleading - here’s why:

First in their init script:

https://github.com/eclipse/jetty.project/blob/master/jetty-distribution/src/main/resources/bin/jetty.sh#L438

It’s used WITHOUT a – Then in usage:

https://github.com/eclipse/jetty.project/blob/master/jetty-start/src/main/resources/org/eclipse/jetty/start/usage.txt#L47

It’s mentioned with a – Then there’s this:

https://github.com/eclipse/jetty.project/blob/master/jetty-start/src/main/java/org/eclipse/jetty/start/StartLog.java#L116

https://github.com/eclipse/jetty.project/blob/master/jetty-start/src/main/java/org/eclipse/jetty/start/StartLog.java#L126

And it’s changed between versions of 9.1.x and 9.2.x so pretty much you need to read the code if you want to get this working correctly.

3) Why does this matter – well let me tell you. You see it would be really nice if all of the jetty logs actually made it to syslog - so they could get forwarded out to magical places like suro.

The first step is getting Jetty to reroute all your slf4jlogs and log4j logs to log4j2 (don’t worry we’ll get back StartLog).

We can start by using jetty-logging.xml plugin (Because it’s like amazing.)

http://www.eclipse.org/jetty/documentation/current/configuring-logging.html

Follow this:

http://www.eclipse.org/jetty/documentation/current/example-logging-log4j.html

Note: you need to make sure you enable the logging module; but you need to make sure the logging module comes AFTER the server module in the config file OTHERWISE the server module, who is responsible for thread pool configurations, will have its configurations rejected, and then you’ll be crying and asking why you took this job, and why this servlet container has to be so janky. (http://dev.eclipse.org/mhonarc/lists/jetty-dev/msg01523.html – and when the author claims he “fixed” this - he meant he fixed the default ordering in the config file, not the actual issue.)

Also this shit: http://stackoverflow.com/questions/20472245/jetty-9-ignoring-my-config-xml

So where was I?

Oh right - so now that you have jetty-logging.xml installed in your start.ini you’re going to need to configure it to use log4j2, before you do that though, you’re going to need to get yourself some amazing jars like these ones, only the latest ones:

Get these guys on your classpath:

lib/log4j2/2.0-beta4/log4j-api-2.0-beta4.jar
lib/log4j2/2.0-beta4/log4j-core-2.0-beta4.jar
lib/log4j2/2.0-beta4/log4j-slf4j-impl-2.0-beta4.jar
lib/slf4j/1.7.5/slf4j-api-1.7.5.jar
lib/slf4j/1.7.5/slf4j-ext-1.7.5.jar

But for fuck sake, don’t put them in paths like that; put them in here like that other jetty doc told you to do: The libraries are put in the ${jetty.base}/lib/logging/ directory.

Now you need to configure jetty to actually use those jars. That is documented shitily in a couple of places, OR we could be an adult and read the code!!!

https://github.com/eclipse/jetty.project/blob/master/jetty-util/src/main/java/org/eclipse/jetty/util/log/Log.java#L46

I’m pretty sure you can set this property in here, like this: https://wiki.eclipse.org/Jetty/Feature/Jetty_Logging#Changing_log_level_in_resources.2Fjetty-logging.properties

Last up is setting up a simple log4j.properties file in here: ${jetty.base}/resources/

So - how does this work?

Well a fuck ton of java code is already implemented using the slf4j api - we configure jetty to use this API - and it’s great! Then we’re injecting our log4j2 implementation of that api into the classpath. And now, if something is logging with slf4j or log4j - they will end up on our log4j logs.

By now - you should have a jetty dumping logs via log4j2 - and that’s awesome because this shit is crazy:

http://www.grobmeier.de/log4j-2-performance-close-to-insane-20072013.html#.VSQkPSjXNoA

You should read this counter opinion - the comments are interesting:

http://www.xorlev.com/blog/2013/08/11/overengineering-log4j2-s-asyncappender/

also it links to this: https://speakerdeck.com/xorlev/java-logging-ninth-circle-of-hell-a-gentle-introduction-to-java-logging

Also - this man is a god: Remko Popma
Here is his stack overflow account: http://stackoverflow.com/users/1446916/remko-popma
And his twitter: https://twitter.com/remkopopma
Someday I hope I can work with him.

Anyways.

So

you magically got this shit working - yay!

You know what you still don’t have?

The first chunk of that jetty log

Why?

Because these AMAZING human beings wrote their own logging service for their application when it starts!!!

Isn’t that amazing!???!?!?!?!?!

Here it is:

https://github.com/eclipse/jetty.project/blob/master/jetty-start/src/main/java/org/eclipse/jetty/start/StartLog.java#L40

Oh what’s that? Hardcoded to print to stderr and stdout - no problem dude!

Oh what’s that? StartLog.log,.warn,.info are only used like 50 times in the jar - it’s cool brah - I got you.

So what you need to do - because seriously fyl - is modify their init script. And that’s ok - because you’ve probably been in this ops game for awhile, and this isn’t your first rodeo because no one knows how to make an init script worth a damn anymore - i mean that is LITERALLY why we have upstart now. :boom:

Anyways (and our fun hasn’t even begun!)
You want to use a little bash tool called logger (http://unixhelp.ed.ac.uk/CGI/man-cgi?logger+1) And what you’re going to do is tell jetty to be in –debug mode because at this point why not get all the data.

And you’re going edit things like this:

https://github.com/eclipse/jetty.project/blob/master/jetty-distribution/src/main/resources/bin/jetty.sh#L438

Using this knowledge: http://urbanautomaton.com/blog/2014/09/09/redirecting-bash-script-output-to-syslog/

To remove start-log-file and replace it with logger instead.

Now that you got that working, you’re probably ready to add to your log4j.properties some syslog magic - these links should help:

https://logging.apache.org/log4j/2.x/manual/appenders.html#SyslogAppender

Use RFC 5424 for the love of everything nice :|

https://tools.ietf.org/html/rfc5424
http://stackoverflow.com/questions/26856447/log4j2-using-syslog-appender-and-rfc5424
http://stackoverflow.com/questions/938907/does-log4j-syslogappender-support-mdc-and-ndc
http://logback.qos.ch/manual/mdc.html
TYL: https://www.iana.org/assignments/enterprise-numbers/enterprise-numbers (we’re 19365)

And now you’re probably done, and or drink whiskey….

LIMITS - a.k.a. it’s 2015 - holy hell why am i having to type things up about this……..
A.k.a. strap in kids - we gonna become big girls and boys today while I fuck you up with some truth.

BUT WAIT THERE’S MORE!
You didn’t think this would be so much fun did you?
So… we decided it would be a good idea to switch to ubuntu. I still stand by that being a good idea - and yeah… cute thing though, all choices have consequences.

When you’re on ubuntu you have access to start-stop-daemon.
Sooo when we do things like this:
https://github.com/eclipse/jetty.project/blob/master/jetty-distribution/src/main/resources/bin/jetty.sh#L488
We’re going to use start-stop-daemon.

See jetty runs as the user jetty, but this init script is run by root on boot - sooooo how are we going to change user?

Well it would seem start-stop-daemon takes care of that for us with the -c command.

I wonder how that works?

http://manpages.ubuntu.com/manpages/trusty/man8/start-stop-daemon.8.html

We see this lovely command comes to us from the depths of hell known as dpkg…
(yes i’m this awesome - i miss me too)
http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/dpkg/trusty-proposed/view/head:/utils/start-stop-daemon.c#L1570
Wow - isn’t this so cool! It uses setuid: http://linux.die.net/man/2/setuid
Fascinating.
You know what system call couldn’t give one fuck about system limits?
You guessed it - setuid.
Setuid is god - and does not care about anything you might have configured in PAM!

(Pause - today you get to learn about PAM too.)

Which yeah.
So you probably don’t know anything about PAM or limits. so lets teach you something about that while we’re at it.

Read this: http://askubuntu.com/questions/215831/why-limit-the-number-of-open-files-and-running-processes-in-linux

Cool so how do we control that shit?

Read this shit: http://manpages.ubuntu.com/manpages/lucid/man8/pam_limits.8.html

Cool so we’re going to set limits in /etc/security/limits.conf and /etc/security/limits.d/

Then when we do shit on our machine, we’re hopefully using commands managed by PAM and hopefully in those configurations our pam_limits.so module is included.

Those commands might be like sudo or su

And those are configured via files like these: /etc/pam.d/sudo or /etc/pam.d/su

And if we would inspect those files we might find a line like this:

session    required   pam_limits.so

BUT IN UBUNTU YOU WILL FIND A LINE LIKE THIS:

# session    required   pam_limits.so

And then you go home.
and you get drunk.
and you ask yourself - self
why did i get into computing?
why did i want to learn about software?
why do i even get up in the morning?

So
All this to say.
Everything.
Everything.
E-V-E-R-Y-T-H-I-N-G
is terrible and sad.

so how do we overcome this?

Just set the limits of things using ulimit in your init script like everyone else does. and boy that is terrible, and you should feel bad. so bad, that you want to become a contributor to ubuntu and solve this atrocity against humanity.

See in the centos world - we didn’t have start-stop-daemon - we had good ‘ol fashion & and nohup - and therefore needed to use sudo and su - and their pam configuration files of course used the pam_limits module because why the FUCK would you ever - ever - ever comment that out - you dumb dumb dumb people.

blah.
I’m done - off to lunch.