Threads for dchest
-
Who generates the key in this case - the host or the token?
-
-
-
-
-
I think one does not exclude the other. E.g. ESP has eFuses for storing encryption keys, which can be read-projected (only readable by the hardware encryption support): https://github.com/espressif/esptool/wiki/espefuse
There are probably more secure elements that support this mode of operation.
-
there is some concern that one has to “trust” Infineon, the maker of the cpu/chip that:
- there is no NIST/NSA style backdoor for their generated keys
- there is no way to exfiltrate or extract a private key without your knowledge (e.g. confiscated & copied by evil agent at airport security, but on-key validation doesn’t show that this happened
That said, I’m happier with an ECC yubikey than a filesystem password protected private key.
-
-
-
-
-
Gonna physically visit my 1U server in my buddy’s basement, reboot it, edit the kernel command line to include
init=/bin/sh, remount/asrw, and update my user account’s password. I should remember to tote a monitor and keyboard over. Might as well have a beer while I’m there.-
-
-
-
UPDATE: wow, such fail. :) First off, I just sat there and watched him play minecraft (connected to the server in his basement) for a few hours. When I finally got around to kicking him off and doing the
init=/bin/shthing, I discovered that the only USB keyboard we had doesn’t work in that mode… Despite it having worked perfectly in GRUB just moments before. Ok, the D-I CD was still sitting there from when we set the machine up. I booted into that, selected Rescue Mode, and started a shell–keyboard worked (of course), but the raid device wasn’t listed in/dev(also network configuration was failing). Then I remembered that we’d used firmware.tar.gz to set up this machine… So I say ‘You still got that USB stick?’ and he replies ‘Hold on, we’ve been trying this for longer than I tried to remember my password–get me back to a login prompt!’ … We get there and I see him try to log in with his username in all caps. …His username is actually lowercase! “Aaannnd we’re back!”. I use his sudo powers to change my password. Done. Beers were consumed (not necessarily in the same order that was originally implied). -
-
-
-
-
-
Nice ad. :|
-
-
-
It’s advertising an open source project, Source Hut, but also Janet, Zig, Nim, Samurai, Sway and other open source projects I like. Projects that get very little payment or gratitude for the work they do.
Yes sr.ht is a service too, a useful one at that. They support BSD well, unlike other companies, how else are they supposed to let people know this fact? Should they be paying largely unethical companies like google for ad space? Or should they just be more subversive so people don’t complain.
Let me put it this way, if every open source project was also a business, should we hate on every single one for advertising? sr.ht didn’t game the upvotes to get on the front page, people upvoted it by themselves.
I suppose there could be a tag ‘sponsored’ so people can ignore them. Not suggesting allowing lower quality from sponsored content either, probably the inverse.
-
The issue is that I see a Sourcehut “ad” every few days: “Sourcehut supports OpenBSD”, “Sourcehut supports migrations from Bitbucket”, “Sourcehut supports ASCII”. Yeah … we got it … A lot of these posts don’t have a lot of meat to them and at this point, it’s just getting spammy.
-
Yeah … we got it … A lot of these posts don’t have a lot of meat to them and at this point, it’s just getting spammy.
They don’t always have a lot of “meat,” but posts about SourceHut represent a capitalist ideology I can actually get behind. A single proprietor, working their ass off to try to change the software world, which has gotten extremely out of hand with regards to complexity, and the marketing of products that fix the complex systems we don’t need, at all, to begin with.
What’s the difference between a SourceHut post, and an
postad that complains that as an open source author I am not compensated fairly? Hint: one should be inspiration, for the other is actually possible.-
-
optional payments
It’s optional, right now, while in Alpha. It doesn’t seem the plan is that forever. Also, if it wasn’t clear, I’m extremely in favor of this model of charging people for a service, but releasing your software under a permissive license.
-
-
-
Just let me other another data point here. It was thanks to the “migration from Bitbucket” post that I found out Sourcehut had a nifty script to help migrations from Bitbucket and that saved hours of work as I migrated 20+ repos effortlessly. This current post made me realize that maybe I should be paying more attention to their CI system as it looks much simpler than others I’ve used. So, in the end, I’m appreciating these blog posts a lot. Yes they are related to a commercial venture but so what? You can self-host it if you’re not into SaaS outside your control. If we set a hard line like this, then it becomes impossible to post about any commercial project at all. It is already hard to monetize FOSS projects to make them sustainable, now imagine if they are not even allowed blog posts…
-
Same here. This string of posts made me aware of sourcehut and when I had to migrate from bitbucket, I then gave them a hard eval. I like their human, non-shitty business model of “I give them money and they give me services”, and that their products are professionally executed and no-frills.
I don’t know how to reconcile it. These articles were very useful to me, when most product ads weren’t and I’d be disappointed if this site became a product advert platform. I think people are right for flagging it is almost-an-ad, but in this one vendor’s case I’m glad I saw them and am now a happy sourcehut customer.
-
-
-
-
if every open source project was also a business, should we hate on every single one for advertising?
Yes. I flag those too. Advertising is a mind killer.
-
-
But there is no other way to get large numbers of people to know about something, following your advice would be suicide.
All these conversations are done like it’s all or nothing. We allow politics/marketing/etc on Lobsters or… it never happens anywhere with massive damage to individuals and society. Realistically, this is a small site with few monetary opportunities for a SaaS charging as little as he does. If the goal is spreading the word, it’s best done on sites and platforms with large numbers of potential users and (especially) paying customers. Each act of spreading the word should maximize the number of people they reach for both societal impact and profit for sustainability.
Multiple rounds on Lobsters means, aside from the first announcement with much fan fare, the author sacrificed each time opportunities to reach new, larger audiences to show the same message again to the same small crowd. Repeating it here is the opposite of spreading the word. Especially since most here that like Sourcehut are probably already following it. Maybe even buying it. He’s preaching to the choir here more than most places.
Mind-killer or not, anyone talking about large-scale adoption of software, ideology, etc should be using proven tactics in the kinds of places that get those results. That’s what you were talking about, though. I figured he was just trying to show latest BSD-related progress on one of his favorite tech forums. More noise than signal simply because he was sharing excitement more than doing technical posts or focused marketing.
-
-
Every blog post is an ad for something. It may not be a product, directly, but it’s advertising an idea, the person, or persons the idea was thought by, the writing (which, btw can be a product) of the author, etc.
If you want to sincerely flag advertising, you might as well get offline—it’s pervasive.
-
It may not be a product, directly, but it’s advertising an idea
Not a native english speaker here. I may be wrong, but after looking at the dictionnary definition:
advertisement
noun
A paid notice that tells people about a product or service.
it seems that an advertisement has a precise definition: an ad is directly related to a paid product, not an idea.
-
it seems that an advertisement has a precise definition: an ad is directly related to a paid product, not an idea.
This is a fairly pedantic interpretation. A person promotes an idea to sell something, if even themselves. That “sale” might only come later in the form of a job offer, or support through Patreon, etc, etc.. But, to say that you can’t advertise an idea is wrong. The cigarette industry’s ad campaigns have always been about selling an image, an idea that if you smoke you become part of something bigger. Oh, and btw, you’ll probably remember the brand name, and buy that kind instead of something else.
iPods were sold on the very basis of white headphones, TO THE POINT, that people without iPods started wearing white headphones to be part of the “club.” Advertisements sell you the idea of a better life, and hopefully you’ll buy my product to get it.
-
-
-
-
-
-
-
I’ve been trying to engage more with Reddit for this reason. I don’t really like it as a platform or see it as doing a social good, but there are users there and I’d like to be there to answer their questions. I was going to make a Twitter account, too, but they wanted my phone number and a pic of my ID and a blood sample to verify my account so I abandoned that. Finding good ways to ethically grow Sourcehut’s audience is not an entirely solved problem.
-
The reason Twitter – and many platforms – asks for phone numbers is because spam and trolls are a persistent problem. Ban one neo-Nazi troll tweeting obscenities at some black actor for DesTROyinG WhITe SocIEtY and they’ll create a new account faster than you can say “fuck off Nazi”.
Reddit is often toxic as hell by the way, so good luck with that.
-
-
-
-
-
-
https://sourcehut.org/blog/index.xml
I’ve been meaning to make this more visible…
hold pleasedone.
-
-
-
-
Said FOSS service has been on the Lobsters front page multiple times recently. I suspect the reaction is: “We get it, sr.ht exists and SirCmpwn is apparently desperate to attract a paying customerbase, but a clickbaity title for a blogspam ad on the usual suspect’s software is probably crossing the line.”
- today: https://lobste.rs/s/ziythj/sourcehut_makes_bsd_software_better
- 26 days ago: https://lobste.rs/s/dqpafi/shell_access_for_builds_sr_ht_ci
- Aug 14, just about a month ago: https://lobste.rs/s/j2wdwl/sourcehut_hacker_s_forge
- Jul 24, though only tangentially related (but has an ad in the paragraph right after the table of contents): https://lobste.rs/s/ktvzwl/use_plaintext_email
-
-
-
Just another example of the “every programmer before be was stupid and wrong” attitude.
What a silly rant. People pay to watch talks like this 🙄
-
This is not a fair characterization of the lightning talk, imo. The whole talk centers around his frustration with dealing with an edge case of
tail. At the end basically gives up and says that some kid will be complaining the same way about his own decisions and he’s so frustrated he doesn’t care, mostly as a result of wrestling with this intractable problem.The problem was pretty interesting, too!
-
There are a number of “JESUS CHRIST ARE YOU SERIOUS?!?!” remarks shouted though, which, to me, basically just seem like a slightly nicer way of saying “JESUS CHRIST WERE THESE PEOPLE STUPID?!?!”
I didn’t watch the last minute or so as I thought the entire thing was obnoxious as hell. I tried watching it again after your comment – just to make sure I wasn’t in some particularly grouchy mood yesterday evening and mistook his remarks – and found it even more obnoxious.
-
Last minute was the most spicy part ;).
It’s just his sense of humour.
-
To be honest, I went into the talk wanting to disagree with you and be upset with your original characterization of it.
But yeah, your criticisms are fair.
It sounds like the commenter was expecting unix to be plan 9. Unix wasn’t plan 9, nor was BSD. It’s entirely reasonable for
tail’s manpage to say “it sits there waiting in hopes that the file will grow.”I had a vivid moment recently when I realized this kind of humor just doesn’t translate very well: I was banned from contributing to homebrew for mocking what I perceived to be a poor design decision.
https://github.com/Homebrew/brew/issues/6154
Boop! Banned for life. No appeal, no one cared. And to be honest, I guess it was fair.
Now I wish I’d done a lightning talk on Homebrew forcing devs to upgrade their entire OS with no workaround.
-
-
That’s precisely the point. Many engineers aren’t funny when they think they’re being funny. It’s obnoxious and unprofessional. Mocking others’ work also implies you’re weak: the people who do great work know how hard it is to get every aspect correct.
Either you believe that there is humor in that issue thread, or you believe I unironically prayed to the gods of hacking for a mercifully short OS update. Possibly while sacrificing a terminal session to earn their favor.
-
-
-
-
-
-
-
-
Bad code!
- There’s
des.NewTripleDESCipher. - Implement
NewECBEncrypterandNewECBDecrypterfor it! - Never report padding errors. Make sure unpadding is constant time.
- There’s
-
peter
4 years ago
|
link
| on:
At 22 years old, Postgres might just be the most advanced database yet
yet in many respects, it is the most modern database management system there is
It’s not though. No disrespect to PostgreSQL, but it just isn’t. In the world of free and open source databases it’s quite advanced, but commercial databases blow it out of the water.
PostgreSQL shines by providing high quality implementations of relatively modest features, not highly advanced state of the art database tech. And it really does have loads of useful features, the author has only touched on a small fraction of them. Almost all those features exist in some other system. But not necessarily one single neatly integrated system.
PostgreSQL isn’t great because it’s the most advanced database, it’s great because if you don’t need anything state of the art or extremely specialized, you can just use PostgreSQL for everything and it’ll do a solid job.
-
but commercial databases blow it out of the water
Can you provide some specific examples?
-
Oracle has RAC, which is a basic install step for any Oracle DBA. Most Postgres users can’t implement something similar, and those that can appreciate it’s a significant undertaking that will lock you into a specific workflow so get it right.
Oracle and MS-SQL also have clustered indexes. Not what Postgres has, but where updates are clustered as well. Getting Pg to perform sensibly in this situation is so painful, it’s worth spending a few grand to simply not worry about it.
Ever run Postgres on a machine with over 100 cores? It’s not much faster than 2 cores without a lot of planning and partitioning, and even then, it’s got nothing on Oracle and MS-SQL: Open checkbook and it’s faster might sound like a lose, but programmers and sysadmins cost money too! Having them research how to get your “free” database to perform like a proper database isn’t cost effective for a lot of people.
How about big tables. Try to update just one column, and Postgres still copies the whole row. Madness. This turns something that’s got to be a 100GB of IO into 10s of TBs of IO. Restructuring this into separate partitions would’ve been the smart thing to do if you’d remembered to do it a few months ago, but this is a surprise coming from commercial databases which haven’t had this problem for twenty years. Seriously! And don’t even try to VACUUM anything.
MS-SQL also has some really great tools. Visual Studio actually understands the database, and its role in development and release. You can point it at two tables and it can build ALTER statements for you and help script up migrations that you can package up. Your autocomplete can recognise what version you’re pointing at. And so on.
…and so on, and so on…
-
Thanks for the detailed response. Not everyone has money to throw at a “real” enterprise DB solution, but (having never worked with Oracle and having only administered small MSSQL setups) I did wonder what some of the specific benefits that make a DBA’s life easier were.
Of course, lots of the open source tools used for web development and such these days seem to prefer Postgres (and sometimes MySQL), and developers like Postgres’ APIs. With postgres-compatible databases like EnterpriseDB and redshift out there, my guess is we’ll see a Postgres-compatible Oracle offering at some point.
-
Not everyone has money to throw at a “real” enterprise DB solution
I work for a commercial database company, so I expect I see a lot more company-databases than you and most other crustaceans: Most companies have a strong preference to rely on an expert who will give them a fixed cost (even if it’s “money”) to implement their database, instead of trying to hire and build a team to do it open-source. Because it’s cheaper. Usually a lot cheaper.
Part of the reason why: An expert can give them an SLA and has PI insurance, and the solution generally includes all costs. Building a engineering+sysadmin team is a big unknown for every company, and they usually need some kind of business analyst too (often a contractor anyway; more £££) to get the right schemas figured out.
Professional opinion: Business logic may actually be some of the least logical stuff in the world.
lots of the open source tools used for web development and such these days seem to prefer Postgres
This is true, and if you’re building an application, I’d say Postgres wins big. Optimising queries for dbmail’s postgres queries was hands down much easier than any other database (including commercial ones!).
But databases are used for a lot more than just applications, and companies who use databases don’t always (or even often) build all (or even much) of the software that interacts with the database. This should not be surprising.
With postgres-compatible databases like EnterpriseDB and redshift out there, my guess is we’ll see a Postgres-compatible Oracle offering at some point.
I’m not sure I disagree, but I don’t think this is a good thing. EnterpriseDB isn’t Postgres. Neither is redshift. Queries that work fine in a local Pg installation run like shit in redshift, and queries that are built for EnterpriseDB won’t work at all if you ever try and leave. These kinds of “hybrid open source” offerings are an anathema, often sold below a sustainable price (and much less than what a proper expert would charge), leaving uncertainty in the SLA, and with none of the benefits of owning your own stack that doing it on plain postgres would give you. I just don’t see the point.
-
Professional opinion: Business logic may actually be some of the least logical stuff in the world.
No kidding. Nice summary also.
-
-
-
-
And yet the cost of putting your data into a proprietary database format is enough to make people find other solutions when limitations are reached.
Don’t forget great database conversion stories like WI Circuit Courts system or Yandex where the conversion to Postgres from proprietary databases saved millions of dollars and improved performance…
-
Links to those stories?
-
-
-
ClickHouse has nothing to do with Postgres, it’s a custom column oriented database for analytics. Yandex Mail actually migrated to Postgres. Just Postgres.
-
-
-
-
Not Oracle-related, but a friend of mine tried to replace a disk-based kdb+ with Postgres, and it was something like 1000x slower. This isn’t even a RAC situation, this is one kdb+ core, versus a 32-core server with Postgresql on it (no failover even!).
Postgres is getting better. It may even be closing the gap. But gosh, what a gap…
-
-
You remember Mark’s benchmarks.
kdb doing 0.051sec what postgres was taking 152sec to complete.
1000x is nothing.
Nobody should be surprised by that. It just means you’re asking the computer to do the wrong thing.
Btw, starting a sentence with “not to be that guy” means you’re that guy. There’s a completely normal way to express curiosity in what my friend was doing (he’s also on lobsters), or to start a conversation about why it was so much easier to get right in kdb+. Both could be interesting, but I don’t owe you anything, and you owe me an apology.
-
Thanks for sharing the source, that helps in understanding.
That’s a benchmark comparing a server grade setup vs essentially laptop grade hardware (quad-core i5), running the default configuration right out of the sample file from the Git repo, with a query that reads a single small column out of a very wide dataset without using an index. I don’t doubt these numbers, but they aren’t terribly exciting/relevant to compare.
Also, there was no disrespect intended, not being a native english speaker I may have come off clumsy though.
-
kdb doing 0.051sec what postgres was taking 152sec to complete.
That benchmarks summary points to https://tech.marksblogg.com/billion-nyc-taxi-rides-postgresql.html which was testing first a pre-9.6 master and then a PG 9.5 with cstore_fdw. Seems to me that neither was fair and I’d like to do it myself, but I don’t have the resources.
-
If you think a substantially different disk layout of Pg, and/or substantially different queries would be more appropriate, I think I’d find that interesting.
I wouldn’t like to see a tuning exercise including a post-query exercise looking for the best indexes to install for these queries though: The real world rarely has an opportunity to do that outside of applications (i.e. Enterprise).
-
-
-
-
-
Sort of? Kdb+ isn’t a big program, and most of what it does is the sort of thing you’d do in C anyway (if you liked writing databases in C): Got some tall skinny table? Try mmaping as much as possible. That’s basically what kdb does.
What was surprising was just how difficult it was to get that in Pg. I think we expected, with more cores and more disks it’d be fast enough? But this was pretty demoralising! I think the fantasy was that by switching the application to Postgres it’d be possible to get access to the Pg tooling (which is much bigger than kdb!), and we massively underestimated how expensive Pg is/can be.
-
Kdb+ isn’t a big program, and most of what it does is the sort of thing you’d do in C anyway (if you liked writing databases in C)
Well, kdb+ is columnar, which is pretty different than how most people approach naive database implementation. That makes it very good for some things, but really rough for others. Notably, columnar storage is doesn’t deal with
updatestatements very well at all (to the degree that some columnar DBs simply don’t allow them).Even on reads, though, I’ve definitely seen postgres beat it on a queries that work better on a row-based system.
But, yes, if your primary use cases favor a columnar approach, kdb+ will outperform vanilla postgres (as will monetdb, clickhouse, and wrappers around parquet files).
You can get the best of both worldsYou can get decent chunks of both worlds by using either the cstore_fdw or imcs extensions to postgres.-
which is pretty different than how most people approach naive database implementation.
I blame foolish CS professors emphasising linked lists and binary trees.
If you simply count cycles, it’s exactly how you should approach database implementation.
Notably, columnar storage is doesn’t deal with update statements very well at all (to the degree that some columnar DBs simply don’t allow them).
So I haven’t done that kind of UPDATE in any production work, but I also don’t need it: Every customer always wants an audit trail which means my database builds are INSERT+some materialised view, so that’s exactly what kdb+ does. If you can build the view fast enough, you don’t need UPDATE.
Even on reads, though, I’ve definitely seen postgres beat it on a queries that work better on a row-based system.
If I have data that I need horizontal grabs from, I arrange it that way in memory. I don’t make my life harder by putting it on the disk in the wrong shape, and if I do run into an application like that, I don’t think gosh using postgres would really speed this part up.
-
-
-
-
-
-
-
Spanner provides globally consistent transactions even across multiple data centers.
Disclosure: I work for Google. I am speaking only for myself in this matter and my views do not represent the views of Google. I have tried my best to make this description factually accurate. It’s a short description because doing that is hard. The disclosure is long because disclaimers are easier to write than useful information is. ;)
-
@geocar covered most of what I wanted to say. I also have worked for a commercial database company, and same as @geocar I expect I have seen a lot more database use cases deployed at various companies.
The opinions stated here are my own, not those of my former or current company.
To put it bluntly, if you’re building a Rails app, PostgreSQL is a solid choice. But if you’ve just bought a petabyte of PCIe SSDs for your 2000 core rack of servers, you might want to buy a commercial database that’s a bit more heavy duty.
I worked at MemSQL, and nearly every deployment I worked with would have murdered PostgreSQL on performance requirements alone. Compared to PostgreSQL, MemSQL has more advanced query planning, query execution, replication, data storage, and so on and so forth. It has state of the art features like Pipelines. It has crucial-at-scale features like Workload Profiling. MemSQL’s competitors obviously have their own distinguishing features and qualities that make them worth money. @geocar mentioned some.
PostgreSQL works great at smaller scale. It has loads useful features for small scale application development. The original post talks about how Arcentry uses NOTIFY to great effect, facilitating their realtime collaboration functionality. This already tells us something about their scale: PostgreSQL uses a fairly heavyweight process-per-connection model, meaning they can’t have a huge number of concurrent connections participating in this notification layer. We can conclude Arcentry deployments using this strategy probably don’t have a massive number of concurrent users. Thus they probably don’t need a state of the art commercial database.
There are great counterexamples where specific applications need to scale in a very particular way, and some clever engineers made a free database work for them. One of my favorites is Expensify running 4 million queries per second on SQLite. SQLite can only perform nested loop joins using 1 index per table, making it a non-starter for applications that require any kind of sophisticated queries. But if you think about Expensify, its workload is mostly point look ups and simple joins on single indexes. Perfect for SQLite!
-
-
No. The author’s claims “Postgres might just be the most advanced database yet.” MemSQL is a database. If you think they’re apples and oranges different, might that be because MemSQL is substantially more advanced? And I used MemSQL as one example of a commercial database. For a more apples-to-apples comparison, I also think MSSQL more advanced than PostgreSQL, which geocar covered.
And MemSQL’s in-memory rowstore serves the same purpose as PostgreSQL’s native storage format. It stores rows. It’s persistent. It’s transactional. It’s indexed. It does all the same things PostgreSQL does.
And MemSQL isn’t only in-memory, it also has an advanced on-disk column store.
-
-
-
-
-
This has been known for some time: https://twitter.com/dchest/status/720964646472183809 (2016)
We discussed it with Cryptocat author. The general attitude, as far as I remember, was that it’s hard to make sure any crypto in JavaScript is constant-time anyway. My point was that while it’s true, it should be “algorithmically constant time”, that is, there are no branches or secret table index accesses in the code even if VM can fuck that up.
As @freddyb said, it’s probably very hard (may even be impossible?) to exploit this in Cryptocat. However, I disagree that it’s not embarrassing [edit: it = Curve25519 implementation]: one of the goals of Curve25519 was to make it easier to write constant-time implementations; writing a variable-time one from scratch for non-educational purposes is embarrassing.
Edit: I should add that this “vulnerability” is not on par with previous ones in the old Cryptocat. You may be surprised how many non-constant time crypto there is in production. In fact, many wouldn’t call this “vulnerability” unless it’s demonstrably exploitable.
-
even if VM can fuck that up
Why deploy a “fix” that doesn’t actually work?
-
It’s not guaranteed to work, but from my very limited tests in specific VMs at specific point of time, it does work.
In general, it’s not different from the current practice: there’s no guarantee that any crypto written in any language (even C or assembly) will be constant-time when it’s executing unless you write for and deploy on specific hardware. E.g. some ARM CPUs have variable-time instructions that other ARM versions implement in constant time.
See https://www.bearssl.org/constanttime.html and https://www.bearssl.org/ctmul.html
-
-
However, I disagree that it’s not embarrassing [edit: it = Curve25519 implementation]: one of the goals of Curve25519 was to make it easier to write constant-time implementations; writing a variable-time one from scratch for non-educational purposes is embarrassing.
Ah! Point taken :)
-
-
It must be noted that this has happened a lot more times if one considers ccTLDs (which are TLDs). In fact, .cs has died twice. Once for Czechoslovakia and once for Serbia-Montenegro https://en.wikipedia.org/wiki/.cs
-
-
I don’t like the design of Enchive.
The process for encrypting a file:
- Generate an ephemeral 256-bit Curve25519 key pair.
- Perform a Curve25519 Diffie-Hellman key exchange with the master key to produce a shared secret.
OK.
- SHA-256 hash the shared secret to generate a 64-bit IV.
Kinda OK, can justify this complexity by the need for a quick check before decryption (“validate the IV against the shared secret hash and format version”) if we got the correct key.
- Add the format number to the first byte of the IV.
OK.
- Initialize ChaCha20 with the shared secret as the key.
This is using raw multiplication result as a key. It’s recommended to hash the result (but not pure SHA256 as we’re already exposing 56 bits of it as IV) before using is as a cipher key (for example, NaCl uses HSalsa20 as a quick hash for that).
- Write the 8-byte IV.
- Write the 32-byte ephemeral public key.
- Encrypt the file with ChaCha20 and write the ciphertext.
OK. But for big files, it may be worth using chunked authenticated encryption to avoid spilling out unauthenticated plaintext or wasting time (see https://www.imperialviolet.org/2014/06/27/streamingencryption.html and my implementation https://github.com/dchest/nacl-stream-js).
- Write HMAC(key, plaintext).
Here we have three problems.
First is that is uses the same key for HMAC as for encryption. I don’t think there’s a particular interaction problem between HMAC-SHA-256 and ChaCha20 that would lead to something scary, but this design is not ideal. To fix this and previous issue in one shot, the authors could use a 64-byte hash function to derive both encryption and authentication keys from Curve25519 shared key: encr_key || mac_key = SHA512(shared_key), or use HMAC-SHA256 with different personalization strings (encr_key = HMAC-SHA256(“EncrKey”, shared_key) and mac_key = HMAC-SHA256(“AuthKey”, shared_key), or HKDF.
Secondly, it’s MAC-then-encrypt, which exposes cipher to various attacks before there’s a chance of authenticating. Finally, I would also authenticate everything, not just the ciphertext. So I’d use HMAC(mac_key, everything) where everything is IV, ephemeral public key, and ciphertext. This way, HMAC will be checked before decrypting, and malicious payload will be rejected early.
Enchive uses an scrypt-like algorithm for key derivation, requiring a large buffer of random access memory.
If it’s scrypt-like, why not just use scrypt? I haven’t checked the whole algorithm, but I can already see a drawback: it uses SHA-256 to perform work on memory. Scrypt specifically uses a very fast function (8-round Salsa20) so that it can perform this computation as quickly as possible, which is very important for a memory-hard function.
To summarize: there’s nothing particularly broken with this design, as far as I can tell from a quick look, but it’s not a solid design, unfortunately.
-
I get the feeling most of those shortcomings are caused by direct use of primitives. I suspect that the author was trying to:
- minimize dependencies – especially looking at
optparse.h, which is (mostly) redundant on a POSIX system due to getopt(3) existing – and source files, and - keep the license unencumbered (all third party code seems to be in the public domain:, but then ended up making dangerous decisions given raw primitives.
argon2 not being in there is probably not an accident but a result of how difficult it is to implement and how he’d have two hash functions (SHA-256 and BLAKE2 for the argon2 state).
The author might’ve had a better result and less work with naive use of Monocypher, libsodium or TweetNaCl, though TweetNaCl still would’ve let him shoot himself in the foot with raw X25519.
- minimize dependencies – especially looking at
-
Enchive’s author here. These are all good points. Most of the mistakes are me not knowing any better when I designed it, but, fortunately, none of them fatal as far as I know.
But for big files, it may be worth using chunked authenticated encryption to avoid spilling out unauthenticated plaintext
I did eventually figure out chunked authentication for myself months later, but too late for Enchive. If I ever redesign the file format, it would definitely use chunked authentication, among other corrections like using EtM.
If it’s scrypt-like, why not just use scrypt?
At the time (early 2017) I couldn’t find a drop-in scrypt library with a friendly license, and I didn’t want to try implementing it myself. A major design goal was ANSI C and no dependencies. As a result, Enchive can easily be compiled just about anywhere, probably even decades into the future (to, say, decrypt some old archives). As evidence of this, you can build it and run it on Windows 98 decades in the past.
-
-
-
So, if you are using FreeBSD, and Gimp, and working with FLIC files, and are dumb enough to either run something random from the net, or let a bad person access your machine with such a file… you’re in trouble.
Somehow I think the union of these sets is in the low 10s, if that.
But hey, this gets attention for the project, and the logo is kinda cute!
-
-
By “dumb”, I mean someone blindly loading a file they got from someone saying “hey d00d check out this cool FLIC!!!”. It’s of course entirely possible for a malicious actor to trick even a vigilant user to load a file by social engineering, MiTM attacks, or spoofing in general.
My point is that the attack surface for this particular vulnerability is very small.
I’m all for better security for all users, and the techniques the project are using seem to bearing fruit. But it doesn’t detract from the fact that this issue is relatively less serious than other issues.
-
-
Yeah — I use FreeBSD, sometimes GIMP (but more often Krita), occasionally with files from the internet…
but I have NOT even heard of FLIC before
-
-
Do you know if the author of Fossil is looking for help contributing, or if they’re just thinking out loud?
-
-
dchest
5 years ago
|
link
| on:
OberonScript - A Lightweight Compiler and Runtime System for the Web (2006)
There’s also https://github.com/vladfolts/oberonjs
-
Hah, I was actually curious whether AST will make a move. Good to see he did.
Still, it’s sad that he doesn’t seem to care about ME.
-
Whether he cares about ME is irrelevant here. By releasing the software under most (all?) free software and open source licenses, you forfeit the right to object even if the code is being used to trigger a WMD - with non-copyleft licenses you agree not to even see the changes to the code. That’s the beauty of liberal software licenses :^)
All that he had asked for is a bit of courtesy.
-
-
No, it is not - hence the open letter. You are most likely confused by the original BSD License which contained the so called, advertising clause.
-
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
-
Correct. The license requires Intel to reproduce what’s mentioned in the parent comment. The distribution of Minix as part of the IME is a “redistribution in binary form” (i.e., compiled code). Intel could have placed the parts mentioned in the license into those small paper booklets that usually accompany hardware, but as far as I can see, they haven’t done so. That is, Intel is breaching the BSD license Minix is distributed under.
There’s no clause in the BSD license to inform Mr. Tanenbaum about the use of the software, though. That’s something he may complain about as lack of courtesy, but it’s not a legal requirement.
What’s the consequence of the license breach? I can only speak for German law, but the BSD license does not include an auto-termination clause like the GPL does, so the license grant remains in place for the moment. The copyright holder (according to the link above, this is Vrije Universiteit, Amsterdam) may demand compensation or acknowledgment (i.e. fulfillment of the contract). Given the scale of the breach (it’s used in countless units of Intel’s hardware, distributed all over the globe by now), he might even be able to revoke the license grant, effectively stopping Intel from selling any processor containing the then unlicensed Minix. So, if you ever felt like the IME should be removed from this world, talk to the Amsterdam University and convince them to sue Intel over BSD license breach.
That’s just my understanding of the things, but I’m pretty confident it’s correct (I’m a law student).
-
-
-
-
-
So, they distribute a binary form of the OS.
-
This is the “tivoization” situation that the GPLv3 was specifically created to address (and the BSD licence was not specifically updated to address).
-
No, it was created to address not being able to modify the version they ship. Hardware vendors shipping GPLv2 software still have to follow the license terms and release source code. It’s right in the article you linked to.
BSD license says that binary distribution requires mentioning copyright license terms in the documentation, so Intel should follow it.
-
-
You have a point with the manufacturers being in-between Intel and the end users that I didn’t see in my above comment, but the outcome is similar. Intel redistributes Minix to the manufacturers, which then redistribute it to the end-users. Assuming Intel properly acknowledges things in the manufacturer’s docs, it’d then be the manufacturers that were in breach of the BSD license. Makes suing more work because you need to sue all the manufacturers, but it’s still illegal to not include the acknowledgements the BSD license demands.
Edit:
Does including a CREDITS file in the firmware count?
No. “Acknowledging” is something that needs to be done in a way the person that receives the software can actually take notice of.
-
-
You’re correct, my bad. But “reproduce the above copyright notice” etc. aims at the same. Any sensible interpretation of the BSD license’s wording has to come to the result that the receivers of the source code must be able to view those parts of the license text mentioned, because otherwise the clause would be worthless.
-
-
-
-
-
-
If they don’t distribute that copyright notice (I can’t remember last seeing any documentation coming directly from Intel as I always buy pre-assembled hardware) and your reasoning is correct, then they ought to fix it and include it somewhere.
However, the sub-thread started by @pkubaj is about being courteous, i.e. informing the original author about the fact that you are using their software - MINIX’s license does not have that requirement.
-
-
-
-
-
-
-
-
-
Do you have a cohesive argument behind that or are you just being negative?
First off, governments aren’t using IME for dragnet surveillance. They (almost certainly) have some 0days, but they aren’t going to burn them on low-value targets like you or me. They pose a giant risk to us because they’ll eventually be used in general-purpose malware, but the government wouldn’t actually fight much (or maybe at all, publicly) to keep IME.
Second off, security engineering is a sub-branch of economics. Arguments of the form “the government can hack anyone, just give up” are worthless. Defenders currently have the opportunity to make attacking orders of magnitude more expensive, for very little cost. We’re not even close to any diminishing returns falloff when it comes to security expenditures. While it’s technically true that the government (or any other well-funded attacker) could probably own any given consumer device that exists right now, it might cost them millions of dollars to do it (and then they have only a few days/weeks to keep using the exploit).
By just getting everyday people do adopt marginally better security practices, we can make dragnet surveillance infeasibly expensive and reduce damage from non-governmental sources. This is the primary goal for now. An important part of “marginally better security” is getting people to stop buying things that are intentionally backdoored.
-
Do you have a cohesive argument behind that or are you just being negative?
Behind what? The idea that governments won’t give up on spying on us? Well, it’s quite simple. Police states have happened all throughout history, governments really really want absolute power over us, and they’re free to work towards it in any way they can.. so they will.
They (almost certainly) have some 0days, but they aren’t going to burn them on low-value targets like you or me.
Sure, but do they even need 0days if they have everyone ME’d?
They pose a giant risk to us because they’ll eventually be used in general-purpose malware
Yeah, that’s a problem too!
Defenders currently have the opportunity to make attacking orders of magnitude more expensive, for very little cost. [..] An important part of “marginally better security” is getting people to stop buying things that are intentionally backdoored
If you mean using completely “libre” hardware and software, that’s just not feasible for anyone who wants to get shit done in the real world. You need the best tools for your job, and you need things to Just Work.
By just getting everyday people do adopt marginally better security practices, we can make dragnet surveillance infeasibly expensive and reduce damage from non-governmental sources.
“Just”? :) I’m not saying we should all give up, but it’s an uphill battle.
For example, the blind masses are eagerly adopting Face ID, and pretty soon you won’t be able to get a high-end mobile phone without something like it.
People are still happily adopting Google Fiber, without thinking about why a company like Google might want to enter the ISP business.
And maybe most disgustingly and bafflingly of all, vast hordes of Useful Idiots are working hard to prevent the truth from spreading - either as a fun little hobby, or a full-time job.
-
-
It reads to me like he just doesn’t want to admit that he’s wrong about the BSD license “providing the maximum amount of freedom to potential users”. Having a secret un-auditable, un-modifiable OS running at a deeper level than the OS you actually choose to run is the opposite of user freedom; it’s delusional to think this is a good thing from the perspective of the users.
-
And the BSD code supported that by making their secret box more reliable and cheaper to develop.
-
-
-
-
-
-
We had to build a custom updater for Peerio (reusing parts from electron-builder and the Electron-native updater), because none of the current ones satisfied our requirements. While electron-builder’s updater and Squirrel.Mac verify signatures, they do so using the native code signing tools and checking that the company name in the certificate matches. Instead, we publish a plain-text manifest signed with OpenBSD’s signify (our version, but compatible format), which looks like this:
untrusted comment: Peerio Updater manifest RWRwKJ91Y/oYjMqOB16Jf5oLxuCkUGwPCM8JOMNtvDTwNuq0SbTdMMPRTfHcVX438LUCx39fAi2rirgq1MoG9dVDxT1goV6omwE= version: 2.37.1 urgency: mandatory date: 2017-09-15T18:16:09.343Z linux-x64-file: https://github.com/PeerioTechnologies/peerio-desktop/releases/download/v2.37.1/peerio-2-linux-x86_64.AppImage linux-x64-sha512: d135a90809eace24cd741c97bb0044c5ab9b76c65e8bd6d6a8711f47e36e6070310b9e40b08f43660491ff3a29c89ce2a8bd452bf9912ee615a9c378db7b33d9 linux-x64-size: 69271552 ...We distribute two public keys with the program (one main and one backup); the program downloads this manifest, verifies signature, downloads the installer file for the current platform, verifies its hash and then perform the installation. As soon as we have the update, we just publish the new manifest into the location that the app checks and that’s it. Static files everywhere.
Nothing new to this, it’s what secure updaters have been doing for a long time (e.g. Sparkle on Mac), but somehow with the web world, this has been forgotten and rewritten, with Node update servers and complicated code signing with unreliable PKI.
This also allows us to implement a public chain of trusted hashes in the future, making sure everyone gets the same update in a verifiable way.
Simple signed plaintext manifest is also convenient to parse with Unix tools, making it easy to verify signature manually before downloading binaries (you’ll have to get our
signifypublic keys for that, of course).
-
Is it time to move back to jQuery and Prototype.js? If these mysterious patents are about things like “virtual DOM”, comparing trees of state or something derived from FRP, then using Vue, Preact, Angular 2, Cycle, Riot, Elm, reflex-dom will infinge them too.
Then let’s wait 20-30 years until these patents expire and everyone finally can use these nice state-management things.
-
-
-
And yet, there are a number of big companies which undoubtedly have big legal teams, and which seem to be okay with using React somewhere. Just cherry-picking some from the list [0]
Airbnb, American Express, Chrysler, Atlassian, eBay, Expedia, Microsoft, NHL, Netflix, New York Times, Salesforce, Twitter, Visa, Walmart… At least some of these companies must have had their legal teams look at the license and decide it was okay to use to use React. Which makes me wonder if the hysteria (this is a bit of hyperbole, but it does seem to have some people really worked up) is justified.
[0] https://github.com/facebook/react/wiki/sites-using-react
-
You’re assuming they’re using the “off-the-shelf” license. There’s nothing preventing them from negotiating a different license with Facebook. Now, I haven’t seen anything showing that this has happened, but it’s a fairly common practice to have individualized contracts with traditional commercial software, so it wouldn’t shock me.
-
It would surprise me, though - why would Facebook enter into an agreement with these big name companies that altered the React license out of Facebook’s favor? I don’t think all these companies did that (and I didn’t list every large or well-known company that’s on that link, by the way), and unless they’re paying FB to use React I just don’t know why FB legal would bother with all the work. Individual negotations with legal teams at all these big companies to reach a mutually agreeable license, just so a dev team can use React? It seems really unlikely. Just as unlikely as all these companies paying FB to get some kind of commerical license for React - when there is no suggestion that such a thing exists.
-
-
-
-
Soon PHP will be hip again with a few sprinkles of XmlHttpRequest in the browser ;)
Well, https://laravel-livewire.com