1. 43
Source URL considered unavailable as of 2 years ago, cached text available

All story content copyright of its respective owner.

I have been doing a lot of writing recently. Some of my writing has been with my sister, with whom I write murder mysteries using the nom-de-plume Maddi Davidson. Recently, we’ve been working on short stories, developing a lot of fun new ideas for dispatching people (literarily speaking, though I think about practical applications occasionally when someone tailgates me).

Writing mysteries is a lot more fun than the other type of writing I’ve been doing. Recently, I have seen a large-ish uptick in customers reverse engineering our code to attempt to find security vulnerabilities in it. This is why I’ve been writing a lot of letters to customers that start with “hi, howzit, aloha” but end with “please comply with your license agreement and stop reverse engineering our code, already.”

I can understand that in a world where it seems almost every day someone else had a data breach and lost umpteen gazillion records to unnamed intruders who may have been working at the behest of a hostile nation-state, people want to go the extra mile to secure their systems. That said, you would think that before gearing up to run that extra mile, customers would already have ensured they’ve identified their critical systems, encrypted sensitive data, applied all relevant patches, be on a supported product release, use tools to ensure configurations are locked down – in short, the usual security hygiene – before they attempt to find zero day vulnerabilities in the products they are using. And in fact, there are a lot of data breaches that would be prevented by doing all that stuff, as unsexy as it is, instead of hyperventilating that the Big Bad Advanced Persistent Threat using a zero-day is out to get me! Whether you are running your own IT show or a cloud provider is running it for you, there are a host of good security practices that are well worth doing.

Even if you want to have reasonable certainty that suppliers take reasonable care in how they build their products – and there is so much more to assurance than running a scanning tool - there are a lot of things a customer can do like, gosh, actually talking to suppliers about their assurance programs or checking certifications for products for which there are Good Housekeeping seals for (or “good code” seals) like Common Criteria certifications or FIPS-140 certifications. Most vendors – at least, most of the large-ish ones I know – have fairly robust assurance programs now (we know this because we all compare notes at conferences). That’s all well and good, is appropriate customer due diligence and stops well short of “hey, I think I will do the vendor’s job for him/her/it and look for problems in source code myself,” even though:

A customer can’t analyze the code to see whether there is a control that prevents the attack the scanning tool is screaming about (which is most likely a false positive)

A customer can’t produce a patch for the problem – only the vendor can do that

A customer is almost certainly violating the license agreement by using a tool that does static analysis (which operates against source code)

I should state at the outset that in some cases I think the customers doing reverse engineering are not always aware of what is happening because the actual work is being done by a consultant, who runs a tool that reverse engineers the code, gets a big fat printout, drops it on the customer, who then sends it to us. Now, I should note that we don’t just accept scan reports as “proof that there is a there, there,” in part because whether you are talking static or dynamic analysis, a scan report is not proof of an actual vulnerability. Often, they are not much more than a pile of steaming … FUD. (That is what I planned on saying all along: FUD.) This is why we require customers to log a service request for each alleged issue (not just hand us a report) and provide a proof of concept (which some tools can generate).

If we determine as part of our analysis that scan results could only have come from reverse engineering (in at least one case, because the report said, cleverly enough, “static analysis of Oracle XXXXXX”), we send a letter to the sinning customer, and a different letter to the sinning consultant-acting-on-customer’s behalf – reminding them of the terms of the Oracle license agreement that preclude reverse engineering, So Please Stop It Already. (In legalese, of course. The Oracle license agreement has a provision such as: "Customer may not reverse engineer, disassemble, decompile, or otherwise attempt to derive the source code of the Programs..." which we quote in our missive to the customer.) Oh, and we require customers/consultants to destroy the results of such reverse engineering and confirm they have done so.

Why am I bringing this up? The main reason is that, when I see a spike in X, I try to get ahead of it. I don’t want more rounds of “you broke the license agreement,” “no, we didn’t,” yes, you did,” “no, we didn’t.” I’d rather spend my time, and my team’s time, working on helping development improve our code than argue with people about where the license agreement lines are.

Now is a good time to reiterate that I’m not beating people up over this merely because of the license agreement. More like, “I do not need you to analyze the code since we already do that, it’s our job to do that, we are pretty good at it, we can – unlike a third party or a tool – actually analyze the code to determine what’s happening and at any rate most of these tools have a close to 100% false positive rate so please do not waste our time on reporting little green men in our code.” I am not running away from our responsibilities to customers, merely trying to avoid a painful, annoying, and mutually-time wasting exercise.

For this reason, I want to explain what Oracle’s purpose is in enforcing our license agreement (as it pertains to reverse engineering) and, in a reasonably precise yet hand-wavy way, explain “where the line is you can’t cross or you will get a strongly-worded letter from us.” Caveat: I am not a lawyer, even if I can use words like stare decisis in random conversations. (Except with my dog, because he only understands Hawaiian, not Latin.) Ergo, when in doubt, refer to your Oracle license agreement, which trumps anything I say herein!

With that in mind, a few FAQ-ish explanations:

Q. What is reverse engineering?

A. Generally, our code is shipped in compiled (executable) form (yes, I know that some code is interpreted). Customers get code that runs, not the code “as written.” That is for multiple reasons such as users generally only need to run code, not understand how it all gets put together, and the fact that our source code is highly valuable intellectual property (which is why we have a lot of restrictions on who accesses it and protections around it). The Oracle license agreement limits what you can do with the as-shipped code and that limitation includes the fact that you aren’t allowed to de-compile, dis-assemble, de-obfuscate or otherwise try to get source code back from executable code. There are a few caveats around that prohibition but there isn’t an “out” for “unless you are looking for security vulnerabilities in which case, no problem-o, mon!”

If you are trying to get the code in a different form from the way we shipped it to you – as in, the way we wrote it before we did something to it to get it in the form you are executing, you are probably reverse engineering. Don’t. Just – don’t.

Q. What is Oracle’s policy in regards to the submission of security vulnerabilities (found by tools or not)?

A. We require customers to open a service request (one per vulnerability) and provide a test case to verify that the alleged vulnerability is exploitable. The purpose of this policy is to try to weed out the very large number of inaccurate findings by security tools (false positives).

Q. Why are you going after consultants the customer hired? The consultant didn’t sign the license agreement!

A. The customer signed the Oracle license agreement, and the consultant hired by the customer is thus bound by the customer’s signed license agreement. Otherwise everyone would hire a consultant to say (legal terms follow) “Nanny, nanny boo boo, big bad consultant can do X even if the customer can’t!”

Q. What does Oracle do if there is an actual security vulnerability?

A. I almost hate to answer this question because I want to reiterate that customers Should Not and Must Not reverse engineer our code. However, if there is an actual security vulnerability, we will fix it. We may not like how it was found but we aren’t going to ignore a real problem – that would be a disservice to our customers. We will, however, fix it to protect all our customers, meaning everybody will get the fix at the same time. However, we will not give a customer reporting such an issue (that they found through reverse engineering) a special (one-off) patch for the problem. We will also not provide credit in any advisories we might issue. You can’t really expect us to say “thank you for breaking the license agreement.”

Q. But the tools that decompile products are getting better and easier to use, so reverse engineering will be OK in the future, right?

A. Ah, no. The point of our prohibition against reverse engineering is intellectual property protection, not “how can we cleverly prevent customers from finding security vulnerabilities – bwahahahaha – so we never have to fix them – bwahahahaha.” Customers are welcome to use tools that operate on executable code but that do not reverse engineer code. To that point, customers using a third party tool or service offering would be well-served by asking questions of the tool (or tool service) provider as to a) how their tool works and b) whether they perform reverse engineering to “do what they do.” An ounce of discussion is worth a pound of “no we didn’t,” “yes you did,” “didn’t,” “did” arguments. *

Q. “But I hired a really cool code consultant/third party code scanner/whatever. Why won’t mean old Oracle accept my scan results and analyze all 400 pages of the scan report?”

A. Hoo-boy. I think I have repeated this so much it should be a song chorus in a really annoying hip hop piece but here goes: Oracle runs static analysis tools ourselves (heck, we make them), many of these goldurn tools are ridiculously inaccurate (sometimes the false positive rate is 100% or close to it), running a tool is nothing, the ability to analyze results is everything, and so on and so forth. We put the burden on customers or their consultants to prove there is a There, There because otherwise, we waste a boatload of time analyzing – nothing** – when we

could be spending those resources, say, fixing actual security vulnerabilities.

Q. But one of the issues I found was an actual security vulnerability so that justifies reverse engineering, right?

A. Sigh. At the risk of being repetitive, no, it doesn’t, just like you can’t break into a house because someone left a window or door unlocked. I’d like to tell you that we run every tool ever developed against every line of code we ever wrote, but that’s not true. We do require development teams (on premises, cloud and internal development organizations) to use security vulnerability-finding tools, we’ve had a significant uptick in tools usage over the last few years (our metrics show this) and we do track tools usage as part of Oracle Software Security Assurance program. We beat up – I mean, “require” – development teams to use tools because it is very much in our interests (and customers’ interests) to find and fix problems earlier rather than later.

That said, no tool finds everything. No two tools find everything. We don’t claim to find everything. That fact still doesn’t justify a customer reverse engineering our code to attempt to find vulnerabilities, especially when the key to whether a suspected vulnerability is an actual vulnerability is the capability to analyze the actual source code, which – frankly – hardly any third party will be able to do, another reason not to accept random scan reports that resulted from reverse engineering at face value, as if we needed one.

Q. Hey, I’ve got an idea, why not do a bug bounty? Pay third parties to find this stuff!

A. Bug bounties are the new boy band (nicely alliterative, no?) Many companies are screaming, fainting, and throwing underwear at security researchers**** to find problems in their code and insisting that This Is The Way, Walk In It: if you are not doing bug bounties, your code isn’t secure. Ah, well, we find 87% of security vulnerabilities ourselves, security researchers find about 3% and the rest are found by customers. (Small digression: I was busting my buttons today when I found out that a well-known security researcher in a particular area of technology reported a bunch of alleged security issues to us except – we had already found all of them and we were already working on or had fixes. Woo hoo!)

I am not dissing bug bounties, just noting that on a strictly economic basis, why would I throw a lot of money at 3% of the problem (and without learning lessons from what you find, it really is “whack a code mole”) when I could spend that money on better prevention like, oh, hiring another employee to do ethical hacking, who could develop a really good tool we use to automate finding certain types of issues, and so on. This is one of those “full immersion baptism” or “sprinkle water over the forehead” issues – we will allow for different religious traditions and do it OUR way – and others can do it THEIR way. Pax vobiscum.

Q. If you don’t let customers reverse engineer code, they won’t buy anything else from you.

A. I actually heard this from a customer. It was ironic because in order for them to buy more products from us (or use a cloud service offering), they’d have to sign – a license agreement! With the same terms that the customer had already admitted violating. “Honey, if you won’t let me cheat on you again, our marriage is through.” “Ah, er, you already violated the ‘forsaking all others’ part of the marriage vow so I think the marriage is already over.”

The better discussion to have with a customer —and I always offer this — is for us to explain what we do to build assurance into our products, including how we use vulnerability finding tools. I want customers to have confidence in our products and services, not just drop a letter on them.

Q. Surely the bad guys and some nations do reverse engineer Oracle’s code and don’t care about your licensing agreement, so why would you try to restrict the behavior of customers with good motives?

A. Oracle’s license agreement exists to protect our intellectual property. “Good motives” – and given the errata of third party attempts to scan code the quotation marks are quite apropos – are not an acceptable excuse for violating an agreement willingly entered into. Any more than “but everybody else is cheating on his or her spouse” is an acceptable excuse for violating “forsaking all others” if you said it in front of witnesses.

At this point, I think I am beating a dead – or should I say, decompiled – horse. We ask that customers not reverse engineer our code to find suspected security issues: we have source code, we run tools against the source code (as well as against executable code), it’s actually our job to do that, we don’t need or want a customer or random third party to reverse engineer our code to find security vulnerabilities. And last, but really first, the Oracle license agreement prohibits it. Please don’t go there.

* I suspect at least part of the anger of customers in these back-and-forth discussions is because the customer had already paid a security consultant to do the work. They are angry with us for having been sold a bill of goods by their consultant (where the consultant broke the license agreement).

** The only analogy I can come up with is – my bookshelf. Someone convinced that I had a prurient interest in pornography could look at the titles on my bookshelf, conclude they are salacious, and demand an explanation from me as to why I have a collection of steamy books. For example (these are all real titles on my shelf):

Thunder Below! (“whoo boy, must be hot stuff!”)

Naked Economics (“nude Keynesians!”)***

Inferno (“even hotter stuff!”)

At Dawn We Slept (“you must be exhausted from your, ah, nighttime activities…”)

My response is that I don’t have to explain my book tastes or respond to baseless FUD. (If anybody is interested, the actual book subjects are, in order, 1) the exploits of WWII submarine skipper and Congressional Medal of Honor recipient CAPT Eugene Fluckey, USN 2) a book on economics 3) a book about the European theater in WWII and 4) the definitive work concerning the attack on Pearl Harbor. )

*** Absolutely not, I loathe Keynes. There are more extant dodos than actual Keynesian multipliers. Although “dodos” and “true believers in Keynesian multipliers” are interchangeable terms as far as I am concerned.

**** I might be exaggerating here. But maybe not.


  2. 22

    Q. But one of the issues I found was an actual security vulnerability so that justifies reverse engineering, right?

    A. Sigh. At the risk of being repetitive, no, it doesn’t, just like you can’t break into a house because someone left a window or door unlocked. I’d like to tell you that we run every tool ever developed against every line of code we ever wrote, but that’s not true. We do require development teams (on premises, cloud and internal development organizations) to use security vulnerability-finding tools, we’ve had a significant uptick in tools usage over the last few years (our metrics show this) and we do track tools usage as part of Oracle Software Security Assurance program. We beat up – I mean, “require” – development teams to use tools because it is very much in our interests (and customers’ interests) to find and fix problems earlier rather than later.

    I think this response sums up what I find wrong with those post. I sure as hell can try to break into my own house. I’m the one on the line if a vulnerability is found, not Oracle.

    1. 13

      It all comes down to “who owns what”, I think.

      Reverse engineering is legal and protected here in Italy (can’t remember if it is an italian law or an european law.. or both?) because if you buy something that runs on your PC, you own it, therefore you can mess with it. If you break their EULA they can refuse to give you support and that’s about it.

      In the USA they made this convenient trick where even if you buy it and you run it, you don’t own the software, you own a “license to use it”, therefore all the conditions that they force you by making you agree to their EULA are valid.

      1. 1

        I don’t mean to derail the discussion to some other topic, but I’m just curious if you know how the law in Italy works for software that you run on your own computer but is time-limited and pings to a server periodically to determine if you’re still allowed to run it?

      2. 6

        Well, if Oracle really wants to take responsibility for all of my losses and damages if there’s a security breach… Who am I to argue?

        I actually wonder whether it would be possible to make that stick.

      3. 17

        The blog post has since been deleted. /u/gaggra on Reddit helpfully copied it in case you wanted to read it.

          1. 3

            Nicely, Lobsters saves a cached version of pages.

            1. 2

              It’s a shame my comment about how unprofessional the post was didn’t make it into the archived version. Actually, I posted it to the moderation log about the time it got taken down…

            2. 17

              The off-topic postscript dig at Keynsian economics is really the cherry on top of this article. I’m struggling to think of a better note of bay area entitlement to end on. “What we really need is Uber for welfare” or “If Google had asked people what they wanted, they’d have said faster cars, not self-driving ones” or something.

              1. 12

                Historically, copyright law has been pretty protective of reverse engineering, and of course it’s a fundamental human right regardless of what the law says — but, just as certain companies will try to get you to sign employment agreements with all sorts of illegal terms in them and then enforce them, certain companies will try to enforce illegal or dubiously-legal copyright license agreements. This blog post demonstrates that Oracle is a bad actor. (Not that we didn’t have evidence of that before.)

                Given that, can we put together a plan to eliminate our dependency on Oracle? They are clearly not a trustworthy steward for things like VirtualBox and the JVM. MariaDB is a good alternative to MySQL, and Postgres is a good alternative to Oracle’s server software, but there’s no alternative to VirtualBox and the JVM that even comes close. What if the next version of the JVM has no OpenJDK version?

                The purpose of copyright law in the US is to “promote the progress of science and the useful arts.” Forbidding people who have purchased a work of literature, such as Oracle’s server software, from reading that work of literature, runs directly contrary to the justification for its existence — quite aside form the fact that reading is not one of the enumerated exclusive rights statutorily granted to the author.

                1. 8

                  What if the next version of the JVM has no OpenJDK version

                  Honestly, this seems like a problem that will resolve itself when it becomes a problem. ZFS is another project that appears to have forked after oracle stopped playing nice. And libre office. The projects that haven’t gone “full community” are the ones they haven’t abandoned yet. When they do…

                  Or as vanilla ice would say, if there’s a problem, yo, I’ll solve it.

                  1. 6

                    The reason this is different from libreoffice (and far more worrying) is that the patent grant for the JDK only applies to forks that pass the “compliance test suite”, and Oracle refuses to run the suite on any unfriendly forks. They could arbitrarily decide to refuse it for the next version of OpenJDK too.

                    So for those poor sods who live in countries that allow software patents, the situation is decidedly unstable.

                2. 8

                  I can say with some authority that this, “We must protect how we make money at all costs” is a very common feeling in the company. That said, I feel like there are a lot of forces at work that are moving towards a more open / open source friendly Oracle.

                  1. 11

                    Given that Oracle is currently litigating against the legal principle that allows free software to be a compatible replacement for proprietary software, specifically in order to kill Android, it is difficult for me to imagine a less open-source-friendly Oracle.

                    1. 4

                      legal principle that allows free software to be a compatible replacement for proprietary software

                      I feel like that’s a bit of a stretch.. might be better to say “free software that implements proprietary APIs”. As one will still be able to completely replace proprietary product A with open source product B.

                      Also as Vanilla Ice @tedu has sad, open source will simply move away from those proprietary implementations.

                      1. 10

                        There is currently no such thing as a proprietary api, only a proprietary implementation of an api. Oracle is trying to make proprietary apis a thing, separate entirely from implementations, at which point proprietary libraries can never be replaced by a compatible open source/libre solution. This principle also applies to eg protocols and formats, so if Oracle prevails, FOSS software will not be able to interoperate at all with proprietary software. This means that you can’t move piecemeal to FOSS, but have to move entire ecosystems at a time, which is bad.

                        1. 2

                          Sure, it’s bad.. but not impossible.

                          1. 4

                            If they are successful, it will be impossible to move piecemeal, and very hard to move the entire world at once. Also, most open source code in the world that currently exists will be rendered illegal.

                            So, basically the least open source friendly a company can possibly be without literally hiring people to assassinate those who produce or use open source.

                            1. 1

                              Also, most open source code in the world that currently exists will be rendered illegal.

                              I am sorry, but that is just incorrect.

                              1. 5

                                You are mistaken. Fortunately, there isn’t any real chance that Oracle’s decision will be taken as precedent.

                                1. 5

                                  Most open source code that currently exists in the world uses an api or a format or a protocol that was not invented by a FOSS group - if only because they (eg) link against libc, which uses min and max, which Oracle is trying to claim ownership of. All works that are copyrightable are automatically copyrighted - no application or notice is required. If all of these things are subject to copyright, that is copyright violation right there unless they are able to get a license for it. from the ‘owners’ of every api, protocol and format they interact with. Thus, illegal.

                                  edit: To clarify re:Oracle, they are trying to claim copyright over any api that has min and max functions, which would include libc. They are not, to the best of my knowledge, directly going after libc - this would just be a side effect of the way they have pursued their claim to the Java standard library.

                                  1. 0


                      2. 8

                        I will believe it when they un-re-close the source to Solaris. Or, even, admit that the OpenSolaris project ever existed, let alone the illumos project today.

                        It has been years, and it seems perilously unlikely that Oracle will seek to be anything other than a money hoover; a militant drag on the field of software engineering.

                      3. 12

                        What a freaking blowhard.

                        1. -4

                          this should be the top comment

                        2. 16

                          Oracle’s arrogance is unlimited. Being able to reverse-engineer software you are running on your own computer is a basic human right. Many big companies already realized that, for instance Microsoft in regard to Skype. Instead of keeping the old, complex, “unbreakable” protocol, they ditched it in favor of a newer version of the MSN-protocol, which is currently being reverse-engineered in this very moment.

                          I don’t know which part of the human psyche this blogpost is trying to address, but it definitely won’t make me stay away from investigating proprietary software running on my computer. It’s maybe a bit too harsh, but when companies spit out articles like these, there’s no doubt in my mind they are either hiding horrible software quality behind their blobs or outrightly included NSA-backdoors in their products.

                          1. 20

                            The writer also completely missed the point of bug bounties.

                            They’re not a replacement for a security team, they’re there so when that one guy gets the 0day, he might want to submit it for money rather than selling it on the black market or making it available for free.

                            Working at a company that has to deal with Oracle (mandatory and paid) support, this article stinks for me much more than it probably does for you. They’re quite bad and.. actually do some of the things that article says they don’t do (we have received one-off patches that haven’t been yet included in any official patch in 10+ years)

                            1. 7

                              My condolences for having to deal with this crappy company on a daily basis. As a software developer for suckless, we have to deal with problems introduced in Ubuntu which breaks our software for some people. Finding the issues, mostly unsolvable, is a pig pain. I may not have the big overview of Oracle, but when it comes to Ubuntu, and how it literally destroys a healthy ecosystem, I got a lot of stories to tell…

                              1. 3

                                I’ve been looking at suckless a bit, and used some of your software (ii). Wanna elaborate a bit in what Ubuntu does to destroy the ecosystem? I’ve been using Ubuntu quite a bit due to hardware support (GPU drivers, out of the box WiFi, etc), so I think I understand in general what you mean, but it would be interesting to hear some stories.

                                1. 2

                                  In many aspects, Ubuntu is tailored for a single path. Just the simple fact that they “steal” the Debian package base, apply a set of patches and call it Ubuntu is a steal for me, because they don’t really bring anything back to the Debian community. And try explaining to a novice why he can’t use Debian .debs for Ubuntu and vice versa. It’s just broken, and in many cases, it even breaks software that works fine on most other distros. But Ubuntu is not the only distro to blame. RedHat is another example. It depends on personal taste in any case, however, if people have trouble using suckless software and it’s the distribution’s fault, it’s arrogant of them to expect us to fix it. In any case, there are more points that can be discussed here (Ubuntu Software Center, Kernel hiccup, …). Feel free to join us on IRC.

                                  1. 2

                                    Can you provide some detail about the changes they make that cause problems?

                              2. 3

                                Most bug bounties don’t appear to pay nearly as much as the vuln is worth to exploiters. Well, some pay more, but generally for bugs with no market value. I’m not certain oracle would pay more than anyone else even with such a program.

                                1. 2

                                  I love the whole ‘only 3% of our bugs come from security researchers bit’. Maybe that’s because since they have no incentive to report bugs to you, so they turn to the black market.

                                2. 12

                                  Basic human right? You’re sure you’re not stretching a bit?

                                  1. 17

                                    I’m not talking about reverse engineering specifically, but being allowed to experiment with and take a closer look at things you own. Now, ownership is a difficult concept when it comes to software, but I think that running software on your computer makes you eligible for investigations of this nature. If I sell you a washing machine, you may not be the design or trademark holder, but I can’t stop you from taking the machine apart, fix it, tune it, whatever. Every attempt by companies to change that is just insane.

                                    1. 7

                                      Reverse engineering is a basic human right, yes. It’s an aspect of freedom of thought.

                                      1. 1

                                        Is it thinking or experimenting? Is there a difference?