The “binary logs” thing is one of the main complaints I hear against systemd. While you can easily install syslogd to regain plaintext logs, this post actually explains in depth why the binary log format is actually quite reasonable and well engineered. It changed my mind on this point.
Given how many times over the years I had journald completely hose itself and freeze apps running on production systems [1] , I don’t find his arguments exceptionally compelling. Far more problems with journald/journalctl than I ever did with various syslog implementations. Yes you can still install syslog, but journald still gets the logs first, and then forwards/duplicates the data to syslog.
Maybe journald is better now? Been a couple of years since I had to deal with it on high volume log systems. At the time we ended up using a program wrapper (something similar to logexec) that sent the logs directly to syslog, and avoided systemd/journald log handling entirely.
[1]: app outputting some log data, journald stops accepting app output, app stdout buffer fills, app freezes blocking on write to stdout
I see. Well nothing beats real world experience, so thank you very much for sharing that!
For me it’s quite the opposite, I never had any issues with journald, neither in production nor in development environments.
Seconded, I actually quite like that I can see all my logs the same way without setting up stuff on my side. With syslog I’d have to tell every program where to log and the systemd combo just takes away that manual burden.
“works for me”
I had this experience too, but that was because journald was hanging due to my disks being slow as molasses (I had deeper problems). I’m honestly not sure whether to blame journald for that.
META: can we start adding year marks to these articles? This is non-news, the post is 4 years old.
I’ve heard the “binary logs are evil!!!” mantra chanted against systemd so many times that it wasn’t funny anymore. It’s a terrible argument. With so many big players putting their logs into databases, the popularity of the ELK stack, it is pretty clear that storing logs in non-plaintext format works. Way back in 2015, I wrote two blog posts about the topic.
The gist of it is that binary logs can be awesome, if put to good use. That the journald is not the best tool is another matter, but journald being buggy doesn’t mean binary logs are bad. It just means that the journald is possibly not the most carefully engineered thing out there. There are many things to criticize about systemd and the journal, and they both have their fair share of issues, but binary storage of logs is not one of them.
Okay, so can we just assume all complaints about “binary logs” are just about these binary logs and get on with things?
The journald/systemd people don’t act like they have any clue what’s going on in the real world: people can’t use the tools they used to, and these tools evidently suck; Plain text sucked less, so what’s the plan to get anything better?
I don’t think that’s entirely reasonable. It’s converting a complaint about principle (“don’t do binary logs”) into a complaint about practice, and that makes a big difference. If journald is a bad implementation of an ok idea, that requires very different steps to fix than if it’s a fundamentally bad idea.
What you’re describing makes sense for people on the systemd project to say (“woah, people hate our binary logs, maybe we should work on them”[0]), but not for the rest of us trying to understand things.
[0] I fear they’re not saying that, as they seem somewhat impervious to feedback
I feel like @geocar is against binary logs as a source format, but not as an intermediate or analytics format. Even if your application uses structured logging, it can still be stored in a text file, for example as JSON, at the source. It can be converted to a binary log later in the chain, for example on a centralized logging server, using ELK, SQL, MongoDB, Splunk or whatever. The benefit is that you keep a lot of flexibility at the source (in terms of supporting multiple formats depending on the source application) and are still able to go back to the plain text log if you encounter a problem.
I’m not even against binary logs “as a source format.”
Firstly: I recognise that “complaints about binary logs” is directed at journald and isn’t the same thing about complaints about logs in some non-text format.
I think getting systemd in deep forced sysadmins to retool on top of journald and that hurt a lot for so very little gain (if there was any gain at all- and for most workflows I suspect there wasn’t). This has almost certainly put people off of binary logs, and has almost certainly got people complaining about binary logs.
To that end: I don’t think those feelings around binary logs are misplaced.
Some humility is [going to be] required when trying to win people over with binary logs, but appropriating the term “binary logs” to include tools the sysadmin chooses is like pulling the rug out from under somebody, and that’s not helping.
Thank you very much for clarifying. I agree that forcing sysadmin “to retool on top of journald” hurts.
No, it’s recognising that when enough people are complaining about “the wrong thing”, telling them it’s the wrong thing doesn’t help them. It just causes them to dig in.
What’s the right thing?
I think that’s the point of the bug…
As soon as the complaints start to be about journald and not “binary logs”, and the distinction is made explicit, yeah, we can. It’s been four years, so I’m not going to hold my breath.
For a lot of use cases, they do not suck. For many, they are a vast improvement over text logs.
Stop logging unstructured text to
syslogorstdout, and either log to files or to a database directly. Pretty much what you’ve been (or should have been) doing the past few decades, because bothsyslogandstdoutare terrible interfaces for logs.People complain about things that hurt, and between Windows and journald it should not be a surprise that “binary logs” is getting the flak. journald has a lot of outreach work to do if they want to fix it.
And yet when programmers make mistakes implementing them, the sysadmin are left cleaning up after them.
Text logs have the massive material advantage that the sysadmin can do something with them. Binary logs need tools to do things, and the journald implementation has a lot of work to do.
Most of the “big players” use a transparent structuring layer rather than making binary logs their golden source of knowledge. This allows people to get a lot of the advantages of binary logs with few disadvantages (and given how cheap disk is, the price is basically zero).
These are directions to developers, not to sysadmins. Sysadmins are the ones complaining.
Are we really to interpret this as refuse to install any software that doesn’t follow this rule?
I’m willing to whack some perl together to get the text log data queryable for my business, but you give me a binary turd I need tools and documentation and advice.
What do you mean by a “transparent structuring layer”?
Something to structure the plain text logs into some tagged format (like JSON or protocol buffers).
Splunk e.g. lets users create a bunch of regular expressions to create these tags.
Got it now. Thanks for clarifying!
For some values of “can do”, yes. Most traditional text logs are terrible to work with (see my linked blog posts, not going to repeat them here, again). Besides, as long as your journal files aren’t corrupt (which happens less and less often these days, I’m told), you can just use
journalctlto dump the entire thing, and grep in the logs, just like you grep in text files. Or filter them first, or dump in JSON and usejq, and so on. Plenty of options there.Clearly our experience differs. Most syslog-ng PE customers (and customers of related products) made binary logs (either PE’s LogStore, or an SQL database) their golden source of knowledge. A lot of startups - and bigger businesses - outsourced their logging to services like loggly, which are a black box like binary logs.
These are directions to sysadmins too. The majority of daemons support logging to files, or use a logging framework where you can set them up to log directly to a central collector, or to a database directly. For a huge list of applications, bypassing syslog has been there since day one. Apache, Nginx, pretty much any Java application can all do this, just to name a few things. There are some notable exceptions such as postfix which will always use syslog, but there are ways around that too.
You can bypass the journal with most applications, some support that easily, some require a bit more work, but it has been doable by sysadmins all these years. I know, because I’ve done it without modifying any code.
With the journal, you have journalctl, which is quite well documented.
Do you believe that syslog-ng has even significant market share of users responsible for logging? Even excluding SMB/VSMB?
I would be surprised to find that most people that use loggly don’t keep any local syslog files.
What exactly are you arguing here?
And?
Right, and the goal is to get people using journald right?
If journald doesn’t want to be used, what it’s reason for existing?
Yes.
Most I’ve seen only keep local logs because they’re too lazy to clean them up, and just leave them to the default logrotate. In the past… six or so years, all loggly (& similar) users I worked with, never looked at their text logs, if they had any to begin with.
For systemd developers, perhaps. I’m not one of them. I don’t mind the journal, because it’s been working fine for my needs. The goal is to show that you can bypass it, if you don’t trust it. That you can get to a state where your logs are processed and stored efficiently, in a way that is easy to work with - easier than plain text files. Without using the journal. But with it, it may be slightly easier to get there, because you can skip the whole getting around it dance for those applications that insist on using
syslogorstdoutfor logging.I think you’re completely wrong.
There are a lot of Debian/RHEL/Ubuntu/*BSD (let alone Windows) machines out there, and they’re definitely not using syslog-ng by default…
Debian publishes install information: syslog-ng verus rsyslogd. It’s no contest.
A big bank I’m working with has zero: all rsyslogd or Windows.
Also, the world is moving to journald…
So, why exactly do you believe this?
Okay, but why do you think this contradicts what I say?
You’re talking about people who have built a custom (text based!) logging system, streaming via the syslog protocol. The golden source was text files.
Ah well, this is a very different topic than what I’m replying to.
I can obviously bypass it by not using it.
I was simply trying to explain why people who complain about binary logging aren’t ignorant/crackpots, and are complaining about something important to them.
I think I know better how many syslog-ng PE customers there are out there (FTR, I work at BalaBit, who make syslog-ng). It has a significant market share. Significant enough to be profitable (and growing), in an already crowded market.
…and we have big banks who run syslog-ng PE exclusively, and plenty of other customers, big and small.
…and syslog-ng plays nicely with it, as does rsyslog. They nicely extend each other.
I think we’re misunderstanding each other… What I consider the golden source may be very different from what you consider. For me, the golden source is what people use when they work with the logs. It may or may not be the original source of it.
I don’t care much about the original source (unless it is also what people query), because that’s just a technical detail. I don’t care much how logs get from one point to another (though I prefer protocols that can represent structured data better than the syslog protocol). I care about how logs are stored, and how they are queried. Everything else is there to serve this end goal.
Thus, if an application writes its logs to a text file, which I then process and ship to a data warehouse, I consider that to be binary logs, because that’s how it will ultimately end up as. Since this warehouse is the interface, the original source can be safely discarded, once it shipped. As such, I can’t consider those the golden source.
If we restricted “binary logs” to stuff that originated as binary from the application, then we should not consider the Journal to use binary logs either, because most of its sources (stdout and syslog) are text-based. If the Journal uses binary logs, then anything that stores logs as binary data should be treated the same. Therefore, everything that ends up in a database, ultimately makes use of binary logs. Even if their original form, or the transports they arrived there, were text.
(Transport and storage are two very different things, by the way.)
I never said they are. All I said is that storing logs in binary is not inherently evil, linked to blog posts where I explain pretty much the same thing, and give examples for how binary storage of logs can improve one’s life. (Ok, I also asserted that syslog and stdout are terrible interfaces for logs, and I maintain that. This has nothing to do with text vs binary though - it is about free-form text being awful to work with; see the linked blog posts for a few examples why.)
Or we just have different definitions of significant.
Look, I have an advertising business that makes enough money to be profitable, and is growing, but I’m not going to say I have a “significant” market share of the digital advertising business.
But whatever.
And I didn’t disagree with that.
If you try and re-read my comments knowing that, maybe it’ll be more clear what I’m actually pointing to.
At this point, we’re just talking past each other, and there’s no point in that.
Thanks for linking to the blog posts, they were most informative.
I remember being really unhappy about journals being binary for a long time.
Eventually though, I stumbled upon the fact that you can actually call something like
strings $thejournalfile | grep MESSAGEon a journal and get the whole log in a more recognizable format. With just a bit more effort, you can actually recover the full log to plaintext in no time.It would be nice if there were a convenience script like this provided with systemd, but that realization (i.e., in a pinch when the system has all but died, I can still go look at the logs because
strings) pretty much ended any qualm I had with the journal.I recognize that systemd is (still, somehow) pretty divisive, but ime, it has made system administration dramatically simpler and more pleasurable. I’ll use it till someone shows me something better.