-
It’s worth noting that some people consider inheriting from
Exceptionrather thanStandardErrorto be bad style. Because catchingExceptionincludes things likeScriptError::SyntaxErrorandSignalException::Interrupt(^C), the advice seems to be to catchStandardErrorinstead if you don’t know what exceptions to expect and thus your custom exception classes would need to inherit fromStandardErrorto match that.-
-
Actually, It’s better to create a custom error that makes sense within the execution context.
Anyway, Thank you for the precision. :-)
But the purpose of my articles are just to explain how it works, not how to use it.
I believe in the fact that developers are not robots and they’ll learn all the aspects of a notion by using it, repeatedly. Instead of telling them what’s good or not, I prefer let them figure it out by themselves.
I prefer put all of my energy on explaining the main concept and let the 5% of edge-cases on side.
But, again, thanks for your precious feedback :-)
-
-
There’s another part of note that this post seems to ignore.
It’s a common occurrence that Google serves an intentionally unsolveable (correct solution is rejected) CAPTCHA or even straight up refuses to serve a CAPTCHA at all. In that case, there’s just no way to access your site. And these things usually happen for months at a time.
-
A better test bed. Although my work focus on developing programs on Linux, I will try to compile and run applications on OpenBSD if it is possible.
I feel like the lack of valgrind does hurt OpenBSD as a testbed. I know there’s malloc.conf(5), but that doesn’t seem to help much in the case of, say, out of bounds access of a stack-allocated variable.
a) Patches. Although most of them are trivial modifications, they are still my contributions.
Don’t claim it’s just trivialities. The small things and adding polish is what really makes OpenBSD stand out (or any software project, really), and every “trivial” modification helps.
-
-
I’m so happy to read about the progress on this, we need this on linux though! Who is interested in developing this?
-
I’d assume that the answer you’d get if you ask a Linux kernel developer is “use SELinux or AppArmor”.
But if we’re wishing for arbitrary things, then I’d also like for strlcpy, strlcat and the arc4random family to be an actual part of POSIX and subsequently adopted by glibc/Linux.
-
Or Smack for those favoring simplicity. Looking at the link, I just found out it got big in automotive Linux, too.
-
-
The closest equivalent on Linux are probably the systemd filesystem sandboxing options: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#ReadWritePaths=
-
-
Culture and priorities are different. Linux’s specifically targets what brings in mainstream and corporate audiences. OpenBSD explicitly rejects a lot of that to be simpler, more UNIX like, or quality/security. Lastly, there’s more attempts to sell Linux-based systems that generate revenue that can fund development.
-
-
-
nickpsecurity
1 month ago
|
link
| on:
Nanix: an idea for a modern, small, UNIX-like operating system
Don’t forget that performance enhancements, security enhancements, and increased hardware support all add to the size over what was done a long time ago with some UNIX or Linux. There’s cruft and necessary additions that appeared over time. I’m actually curious what a minimalist OS would look like if it had all the necessary or useful stuff. I especially curious if it would still fit on a floppy.
If not security or UNIX, my baseline for projects like this is MenuetOS. The UNIX alternative should try to match up in features, performance, and size.
-
-
-
Can you fit it with a desktop experience on a floppy like MenuetOS or QNX Demo Disc? If not, it’s not as minimal as we’re talking about. I am curious how minimal OpenBSD could get while still usable for various things, though.
-
-
-
Re: enhancements, I’m thinking Nanix would be more single-purpose, like muLinux, as a desktop OS that rarely (or never) runs untrusted code (incl. JS) and supports only hardware that would be useful for that purpose, just what’s needed for a CLI.
Given that Linux 2.0.36 (as used in muLinux), a very functional UNIX-like kernel, fit with plenty of room to spare on a floppy, I think it would be feasible to write a kernel with no focus on backwards hardware or software compatibility to take up the same amount of space.
-
Your OS or native apps won’t load files that were on the Internet or hackable systems at some point? Or purely personal use with only outgoing data? Otherwise, it could be hit with some attacks. Many come through things like documents, media files, etc. I can imagine scenarios where that isn’t a concern. What’s your use cases?
-
To be honest, my use cases are summed up in the following sentence:
it might be a nice learning exercise to get a minimal UNIX-like kernel going and a sliver of a userspace
But you’re right, there could be attacks. I just don’t see something like Nanix being in a place where security is of utmost importance, just a toy hobbyist OS.
-
If that’s the use, then I hope you have a blast building it. :)
-
-
-
-
-
It seems to work, just booted the ISO (admittedly not the floppy, don’t have what is needed to make a virtual image right now) of muLinux in Hyper-V and it seems to work fine, even having 0% CPU usage on idle according to Hyper-V.
-
-
-
-
-
I don’t like the design of Enchive.
The process for encrypting a file:
- Generate an ephemeral 256-bit Curve25519 key pair.
- Perform a Curve25519 Diffie-Hellman key exchange with the master key to produce a shared secret.
OK.
- SHA-256 hash the shared secret to generate a 64-bit IV.
Kinda OK, can justify this complexity by the need for a quick check before decryption (“validate the IV against the shared secret hash and format version”) if we got the correct key.
- Add the format number to the first byte of the IV.
OK.
- Initialize ChaCha20 with the shared secret as the key.
This is using raw multiplication result as a key. It’s recommended to hash the result (but not pure SHA256 as we’re already exposing 56 bits of it as IV) before using is as a cipher key (for example, NaCl uses HSalsa20 as a quick hash for that).
- Write the 8-byte IV.
- Write the 32-byte ephemeral public key.
- Encrypt the file with ChaCha20 and write the ciphertext.
OK. But for big files, it may be worth using chunked authenticated encryption to avoid spilling out unauthenticated plaintext or wasting time (see https://www.imperialviolet.org/2014/06/27/streamingencryption.html and my implementation https://github.com/dchest/nacl-stream-js).
- Write HMAC(key, plaintext).
Here we have three problems.
First is that is uses the same key for HMAC as for encryption. I don’t think there’s a particular interaction problem between HMAC-SHA-256 and ChaCha20 that would lead to something scary, but this design is not ideal. To fix this and previous issue in one shot, the authors could use a 64-byte hash function to derive both encryption and authentication keys from Curve25519 shared key: encr_key || mac_key = SHA512(shared_key), or use HMAC-SHA256 with different personalization strings (encr_key = HMAC-SHA256(“EncrKey”, shared_key) and mac_key = HMAC-SHA256(“AuthKey”, shared_key), or HKDF.
Secondly, it’s MAC-then-encrypt, which exposes cipher to various attacks before there’s a chance of authenticating. Finally, I would also authenticate everything, not just the ciphertext. So I’d use HMAC(mac_key, everything) where everything is IV, ephemeral public key, and ciphertext. This way, HMAC will be checked before decrypting, and malicious payload will be rejected early.
Enchive uses an scrypt-like algorithm for key derivation, requiring a large buffer of random access memory.
If it’s scrypt-like, why not just use scrypt? I haven’t checked the whole algorithm, but I can already see a drawback: it uses SHA-256 to perform work on memory. Scrypt specifically uses a very fast function (8-round Salsa20) so that it can perform this computation as quickly as possible, which is very important for a memory-hard function.
To summarize: there’s nothing particularly broken with this design, as far as I can tell from a quick look, but it’s not a solid design, unfortunately.
-
Enchive’s author here. These are all good points. Most of the mistakes are me not knowing any better when I designed it, but, fortunately, none of them fatal as far as I know.
But for big files, it may be worth using chunked authenticated encryption to avoid spilling out unauthenticated plaintext
I did eventually figure out chunked authentication for myself months later, but too late for Enchive. If I ever redesign the file format, it would definitely use chunked authentication, among other corrections like using EtM.
If it’s scrypt-like, why not just use scrypt?
At the time (early 2017) I couldn’t find a drop-in scrypt library with a friendly license, and I didn’t want to try implementing it myself. A major design goal was ANSI C and no dependencies. As a result, Enchive can easily be compiled just about anywhere, probably even decades into the future (to, say, decrypt some old archives). As evidence of this, you can build it and run it on Windows 98 decades in the past.
-
I get the feeling most of those shortcomings are caused by direct use of primitives. I suspect that the author was trying to:
- minimize dependencies – especially looking at
optparse.h, which is (mostly) redundant on a POSIX system due to getopt(3) existing – and source files, and - keep the license unencumbered (all third party code seems to be in the public domain:, but then ended up making dangerous decisions given raw primitives.
argon2 not being in there is probably not an accident but a result of how difficult it is to implement and how he’d have two hash functions (SHA-256 and BLAKE2 for the argon2 state).
The author might’ve had a better result and less work with naive use of Monocypher, libsodium or TweetNaCl, though TweetNaCl still would’ve let him shoot himself in the foot with raw X25519.
- minimize dependencies – especially looking at
-
-
-
Fortunately, arc4random_uniform(3) has mostly solved the range problem. If you have it available.
-
“missing” out of the box for composition and revision are tools for version control
There’s RCS and CVS in the base system for that.
One thing that I find somewhat unfortunate is that OpenBSD has a lot of great text editing tools, yet it’s missing any kind of typesetter (troff, TeX) in the base system.
-
…editing because @xorhash had been kind enough to remind me of rcs(1) and cvs(1)…
OpenBSD’s base system doesn’t provide dictionary searches or spell check, either, but I’m fine with that. I’m grateful they provide X Window as part of the base system. Stuff like git, troff, aspell, diction, pandoc, and dictd I’m happy to install using the package mangler.
What I would love to know is why OpenBSD ports has the dict server but none of the dictionaries. If I want a dict daemon on my laptop so I can check definitions offline, I have to get the actual dictionary archives out of the FreeBSD port’s distfiles because ftp.dict.org is dead. While I can do that, I’d rather not have to. :)
-
-
That’s an excellent introduction. Thanks.
However, RCS isn’t actually suited to my use case because I don’t use one file per novel. Instead, I write novels the way I code at my day job, with text distributed across various files in a directory tree. Yes, it’s probably overkill, but it beats paying a shitload of money for a Mac so I can use Scrivener or Ulysses.
My hierarchy currently looks somewhat like this:
$SERIES/ $TITLE/ title dedication disclaimer acknowledgements $SUBPLOT1/ 01.scene 02.scene 01.revision01.sed $SUBPLOT2/ 01.sceneWhen I’m ready to read what I’ve done as a whole, I’ll assemble the whole mess using
catandfmt. Likewise when I’m done with all revisions and am ready to submit to a publisher. At that point I’ll put everything together into a file like “submission01”, mark it up with with Markdown or reStructuredText (depending on whether I was pretentious enough to include footnotes), run it through pandoc and convert it to Word format (unless the publisher is hip enough to accept an OpenDocument Text file, and then edit the output in LibreOffice to suit the publisher’s house style.You can’t manage something like this with RCS. CVS would be more appropriate, but as I mentioned in another comment I’m already familiar with git. I use it when tinkering with static site generators, build websies, and at my day job.
-
-
Apparently there’s an AppImage of the unfinished Linux version for people who don’t want to use WINE.
Believe it or not, I’ve tried Scrivener. It’s not a bad app, but I don’t like that it stores everything in RTF files. When I’m drafting something, I’d rather work in plain text.
Also, as @qznc noted, a tool like ed(1) is great if you have a tendency to go back and edit unfinished work. I have this tendency in spades.
-
-
I don’t see why you can’t use RCS.
% ed test a this is a test of using RCS for version control. . w 49 !ci -l % ci -l test test,v <-- test enter description, terminated with single '.' or end of file: NOTE: This is NOT the log message! >> test check in >> . initial revision: 1.1 done ! ,n 1 this is a test of using 2 RCS for version control. a Now we add a new paragraph. . w 78 !ci -l % ci -l test test,v <-- test new revision: 1.2; previous revision: 1.1 enter log message, terminated with single '.' or end of file: >> new paragraph >> . done ! ,n 1 this is a test of using 2 RCS for version control. 3 4 Now we add a new paragraph. qCompared to git, the only thing that’s missing is keeping track of contents that get moved from one file to another.
-
RCS is one repository per file. That’s not what I want. I want one repository for the entire project. And I want the master repository to live on BitBucket (or some other provider I trust because I’m too lazy to self-host on a VPS). This lets me sync between multiple machines.
This way, when I’m dead because somebody got upset about me typing in public and decided to beat me into the ground with my laptop, it’s possible that some other nerd who overdosed on JRPGs and Blue Öyster Cult albums as a kid might find it and take over. :)
-
In the true spirit of unix, you use one tool for one purpose only. Just use a separate tool for syncing. scp(1) works. rsync(1) works better. unison(1) beats everything.
You can’t really call RCS a ‘repository’. It is, after all, just one ‘,v’ file for the version history of a single file. You can setup rsync or unison to sync up ‘,v’ files exclusively, which essentially transforms rcs to a hand-rolled cvs.
-
-
-
-
-
-
-
(Preface: I didn’t know much, and still don’t, about the *Solaris ecosystem.)
So it seems like the evolution of *Solaris took an approach closer to Linux? Where there’s a core chunk of the OS (kernel and core build toolchain?) that is maintained as its own project. Then there’s distributions built on top of illumos (or unleashed) that make them ready-to-use for endusers?
For some reason, I had assumed it was closer to the *BSD model where illumos is largely equivalent to something like FreeBSD.
If I wanted to play with a desktop-ready distribution, what’s my best bet? SmartOS appears very server oriented - unsurprising given *Solaris was really make more in-roads there in recent years. OpenIndiana?
-
If Linux (kernel only) and BSD (whole OS) are the extremes of the scale, illumos is somewhere in the middle. It is a lot more than just a kernel, but it lacks some things to even build itself. It relies on the distros to provide those bits.
Historically, since Solaris was maintained by one corporation with lots of release engineering resources and many teams working on subsets of the OS as a whole, it made sense to divide it up into different pieces. The most notable one being the “OS/Net consolidation” which is what morphed into what is now illumos.
Unleashed is still split across more than one repo, but in a way it is closer to the BSD way of doing things rather than the Linux way.
Hope this helps clear things up!
If I wanted to play with a desktop-ready distribution, what’s my best bet? SmartOS appears very server oriented - unsurprising given *Solaris was really make more in-roads there in recent years. OpenIndiana?
OI would be the easiest one to start with on a desktop. People have gotten Xorg running on OmniOS (and even SmartOS), but it’s extra work vs. just having it.
-
-
-
illumos itself doesn’t have an actual release. You’re expected to use one of its distributions as far as I can tell, which should arguably be called “derivatives” instead. OpenIndiana seems to be the main desktop version.
-
-
-
-
-
If the book is so bad, then what is the publisher doing? Isn’t it their job to weed out bad content?
-
I wanted to explore that question some more in the post, but it got out of scope and is really its own huge topic.
The short version is that perhaps, as readers, we think they are asking “Is this content any good?” when what they’re really asking is, “Will this sell?”
-
In the preface of the second edition it says that the first edition was reviewed “by a professional C programmer hired by the publisher.” That programmer said it should not be published. That programmer was right, but the publisher went ahead and published it anyway.
Can you expand slightly on this? I understand that the second edition contains a blurb that someone they hired reviewed the 1st edition and decided it should never be published. I’m slightly lost in meaning here.
- Did they hire a person for the second edition, to review the first edition where the conclusion was ‘that should have not been published’?
- Hired a person to review the first edition, the conclusion was to not publish but they still decided to publish and included a blurb about it in the second edition?
I guess the question is, did they knew before publishing that it’s this bad.
Additionally was the second edition reviewed by the same person and considered OK to be published?
-
Here’s a longer excerpt from the second edition’s preface.
Prior to the publication of the first edition, the manuscript was reviewed by a professional C programmer hired by the publisher. This individual expressed a firm opinion that the book should not be published because “it offers nothing new—nothing the C programmer cannot obtain from the documentation provided with C compilers by the software companies.”
This review was not surprising. The reviewer was of an opinion that was shared by, perhaps, the majority of professional programmers who have little knowledge of or empathy for the rigors a beginning programmer must overcome to achieve a professional-level knowledge base of a highly technical subject.
Fortunately, that reviewer’s objections were disregarded, and “Mastering C Pointers” was released in 1990. It was an immediate success, as are most books that have absolutely no competition in the marketplace. This was and still is the only book dedicated solely to the subject of pointers and pointer operations using the C programming language.
To answer your question, then, all we can conclude is that a “professional C programmer” reviewed the first edition before it was published, recommended against publishing it, but the book was published anyway. If the quoted portion were the reviewer’s only objection, then we could surmise that the reviewer didn’t know much either, or didn’t actually read it.
-
little knowledge of or empathy for … a beginning programmer
This is an important point I feel that has been left out of the discussion of this book. Yes the book contains harmful advice that should not be followed. It is probably a danger to make this text available to beginners, and it serves as little more than an object of ridicule for more experienced readers.
However, I think there is something to be gained from a more critical analysis that doesn’t hinge on the quality or correctness of the example. This reviewer takes a step in the right direction by trying to look at Traister’s background and trying to interpret how he arrived at holding such fatal misconceptions about C programming from a mental model seemingly developed in BASIC.
Traister’s code examples are in some cases just wrong and non-functioning, but in other cases I can understand what he wanted to achieve even if he has made a serious mistake. An expert C programmer has a mental model informed by their understanding of the memory management and function call semantics of C. A beginner or someone who has experience in a different sort of language will approach C programming from their own mental model.
Rather than pointing and laughing at his stupidity, or working to get this booked removed from shelves, maybe there’s something to be gained by exercising empathy for the author and the beginner programmer. Are the mistakes due to simple error, or do they arise from an “incorrect” mental model? Does the “incorrect” mental model actually make some sense in a certain way? Does it represent a possibly common misconception for beginners? Is it a fault of the programmer or the programming language?
-
…an opinion that was shared by, perhaps, the majority of professional programmers who have little knowledge of or empathy for the rigors a beginning programmer must overcome…
What utter nonsense. This is inverse-meritocracy: claiming that every single expert is blinded by their knowledge & experience. Who are we to listen to then?
It seems like they’d prefer lots of terrible C programmers cropping up right away, to a moderate number of well-versed C programmers entering the craft over time. Which, now that I think about it, is a sensible approach for a publisher to take.
-
-
-
-
Exactly. There’s tons of news outlets, magazines, and online sites that make most of their money on fluff. Shouldn’t be surprised if computer book publishers try it. The managers might have even sensed IT books are BS or can risk being wrong individually given how there’s piles of new books every year on the same subjects. “If they want to argue about content, let them do it in the next book we sell!” ;)
-
I recommend a scene from Hal Harley’s film “Fay Grim” (the sequel to “Henry Fool”) here. At a point, Fay questions the publishers decision to publish a work (‘The Confessions’) of her husband - she only read “the dirty parts” but still recognized the work as “really, really bad”.
Excerpted from a PopMatters review: “One proposal, from Simon’s publisher Angus (Chuck Montgomery), will lead to publication of Henry’s (admittedly bad) writing and increased sales of Simon’s poetry (on which royalties Fay and Ned depend to live). (Though the writing is, Fay and Angus agree, “bad,” he asserts they must press on, if only for the basest of reasons: “We can’t be too hard-line about these things, Fay. Anything capable of being sold can be worth publishing.”)”
-
-
-
-
You’d save yourself a lot of trouble upfront not borrowing the filezilla name - it’s trademarked. Already there’s an argument for whether “-ng” postfix constitutes a new mark, why bother even having it. Just completely rename it
Hilariously their trademark policy seems to prohibit their use of their own name
-
-
-
-
You’d be inviting lawsuits over the term “godzilla” instead.
-
Just remember to make sure it’s easy for random people to remember and spell. They’ll be Googling it at some point.
-
-
-
Nice. If you distribute pre-compiled binaries, please gpg-sign them and perhaps provide sha512 checksums of them as well.
-
-
Reproducible builds would be awesome.
As for SHA256 vs. SHA512, from a performance point of view, SHA512 seems to perform ~1.5x faster than SHA256 on 64-bit platforms. Not that that matters much in a case like this, where we’re calculating it for a very small file, and very infrequently. Just thought I’d put it out there. So, yeah, SHA256 works too if you want to go with that :)
-
Also remember defaulting on SHA-1 or SHA-256 means hardware acceleration might be possible for some users.
-
SHA-1 has been on the way out for a while, and browsers refuse SHA-1 certificates these days. It might be a good idea to just skip SHA-1 entirely and rely on the SHA-2 family.
-
True. I was just noting there’s accelerators for it in many chips.
-
-
Isn’t SHA-512 faster on most modern hardware? ZFS uses SHA-512 cut down to SHA-256 for this reason, AFAIK.
A benchmark: https://crypto.stackexchange.com/questions/26336/sha512-faster-than-sha256
-
Oh idk. I havent looked at the numbers in a while. I recall some systems, esp cost- or performance-sensitive, stuck with SHA-1 over SHA-256 years ago when I was doing comparisons. It was fine if basic collisions weren’t an issue in thd use case.
-
Anecdotal, but I just timed running sha 512 and 256 10 times each, on a largeish (512MB) file. Made sure to run them a couple of times before starting the timer to make sure it was in cache. Results for sha-512 were:
27.66s user 2.86s system 99% cpu 30.562 totalAnd 256:
42.18s user 2.72s system 99% cpu 44.943 totalSo it looks like sha-512 pretty clearly wins. (CPU is an i3-5005u).
-
Cool stuff. Modern chips handle good algorithms pretty well. What I might look up later is where the dirt-cheap chips are on offload performance and if they’ve upgraded algorithms yet. That will be important for IoT applications as hackers focus on them more.
-
-
-
I said there’s hardware accelerators for SHA-1 and SHA-2. Both are in use in new deployments with one used sometimes for weak-CPU devices or legacy support. Others added more points to the discussion with current, performance stats something I couldnt comment on.
Now, which of my two claims do you think is wrong?
-
- As noted, SHA-1 has been on its way out for awhile and shouldn’t be suggested.
- I don’t know if your claim on weak-CPU devices or legacy support is true, plus you mentioned IoT in response elsewhere, it clearly doesn’t apply in the context of filezilla, an FTP app people will be running on desktops/laptops. Even if one is using the a new ARM laptop that is somewhat under powered…
- As the comment you responded to points out, one installs new software quite infrequently, so the suggestion based on performance seems odd, especially since the comment you responded to already points out that SHA-512 is generally faster to compute than SHA-256. In any case, suggesting SHA-1 for performance reasons seems unsecure.
-
-
-
-
-
-
-
-
-
-
Somehow, this reminds me of shoutboxes from back in the day.
-
-
-
-
Thanks for reminder to check it. The repealing net neutrality one w/ “executive fiat’ was great haha.
-
No such thing in our industry, regrettably, due to the extensive marketing and cultural issues.
-
-
-
-
-
No, you don’t need C aliasing to obtain vector optimization for this sort of code. You can do it with standards-conforming code via memcpy(): https://godbolt.org/g/55pxUS
-
Wow, it’s actually completely optimizing out the memcpy()? While awesome, that’s the kind of optimization I hate to depend on. One little seemingly inconsequential nudge and the optimizer might not be able to prove that’s safe, and suddenly there’s an additional O(n) copy silently going on.
-
memset/memcpy get optimized out a lot, hence libraries making things like this: https://monocypher.org/manual/wipe
-
-
-
You can - and thanks for the illustration - but the memcpy is antethical to the C design paradigm in my always humble opinion. And my point was not that you needed aliasing to get the vector optimization, but that aliasing does not interfere with the vector optimization.
-
I’m sorry but the justifications for your opinion no longer hold. memcpy() is the only unambiguous and well-defined way to do this. It also works across all architectures and input pointer values without having to worry about crashes due to misaligned accesses, while your code doesn’t. Both gcc and clang are now able to optimize away memcpy() and auto vars. An opinion here is simply not relevant, invoking undefined behavior when it increases risk for no benefit is irrational.
-
Au contraire. As I showed, C standard does not need to graft on a clumsy and painful anti-alias mechanism and programmers don’t need to go though stupid contortions with allocation of buffers that disappear under optimization , because the compiler does not need it. My code does’t have alignment problems. The justification for pointer alias rules is false. The end.
-
-
In those cases, you need to use an alignment modifier or sizeof. No magic needed. There is a reason that both gcc and clang have been forced to support -fnostrict_alias and now both support may_alias. The memcpy trick is a stupid hack that can easily go wrong - e.g one is not guaranteed that the compiler will optimize away the buffer, and a large buffer could overflow stack. You’re solving a non-problem by introducing complexity and opacity.
-
In what world is memcpy() magic and alignment modifiers aren’t? memcpy() is an old standard library function, alignment modifiers are compiler-specific syntax extensions.
memcpy() isn’t a hack, it’s always well-defined while aliasing can never be well-defined in all cases. Promoting aliasing as a rule is like promoting using the equality operator between floats – it can never work in all cases, though it may be possible to define meaningful behavior in specific cases. Promoting aliasing as a rule is promoting the false idea that C is a thin layer above contemporary architectures, it isn’t. Struct memory is not necessarily the same as array memory, not every machine that C supports can deference an int32 inside of an int64, not every machine can deference an int32 at any offset. Do you want C to die with x86_64 or do you want C to live?
Optimizations don’t need to be guaranteed when the code isn’t even correct in the first place. First make sure your code is correct, then worry about optimizing. You talk about alignment modifiers but they are rarely used, and usually they are used after a bug has already occurred. Code should be correct first, and memcpy() is the rule we should be promoting since it is always correct. Optimizers can meticulously add aliasing for specific cases once a bottleneck has been demonstrated. You’re solving a non-problem by indulging in premature optimization.
-
-
The memcpy hack is a hack because the programmer is supposed to write a copy of A to B and then back to A and rely on the optimizer to skip the copy and delete the buffer. So unoptimized the code may fault on stack overflows for data structures that exist only to make the compiler writers happier. And with a novel architecture, if the programmer wants to take advantage of a new capability - say 512 bit simd instructions , she can wait until the compiler has added it to its toolset and be happy with how it is used.
As for this not working in all cases: Big deal. C is not supposed to hide those things. In fact, the compiler has no idea if the memory is device memory with restrictions on how it can be addressed or memory with a copy on write semantics or …. You want C to be Pascal or Java and then announce that making C look like Pascal or Java can only be solved at the expense of making C unusable for low level programming. Which programming communities are asking for such insulation? None. C works fine on many architectures. C programmers know the difference between portable and non-portable constructs. C compilers can take advantage of SIMD instructions without requiring C programmers to give up low level memory access - one of the key advantages of programming in C. Basically, people who don’t like C are trying to turn C into something else and are offended that few are grateful.
-
You aren’t writing a copy of a buffer back and forth. In your example, you are reducing an encoding of a buffer into a checksum. You are only copying one way, and that is for the sake of normalization. All SIMD code works that way, you always must copy into SIMD registers first before doing SIMD operations. In your example, the aliasing code doesn’t resemble SIMD code both syntactically and semantically as much the memcpy() code does and in fact requires a smarter compiler to transform.
The chance of overflowing the stack is remote, since stacks now automatically grow and structs tend to be < 512 bytes, but if that is a legitimate concern you can do what you already do to avoid that situation, either use a static buffer (jeopardizing reentrancy) or use malloc().
By liberally using aliasing, you are assuming a specific implementation or underlying architecture. My point is that in general you cannot assume arbitrary internal addresses of a struct can always be dereferenced as int32s, so in general that should not be practiced. In specific cases you can alias, but those are the exceptions not the rule.
-
All copies on some architectures reduce to: load into register, store from register. So what? That is why we have a high level language which can translate *x = *y efficiently. The pointer alias code directly shows programmer intent. The memcpy code does not. The “sake of normalization” is just another way of saying “in order to cooperate with the fiction that the inconsistency in the standard produces”.
In many contexts, stacks do NOT automatically grow.Again, C is not Java. OS code, drivers, embedded code, even many applications for large systems - all need control over stack size. Triggering stack growth may even turn out to be a security failure for encryption which is almost universally written in C because in C you can assure time invariance (or you could until the language lawyers decided to improve it). Your proposal that programmers not only use a buffer, but use a malloced buffer, in order to allow the optimizer (they hope) not to use it, is ridiculous and is a direct violation of the C model.
“3. C code can be non-portable. Although it strove to give programmers the opportunity to write truly portable programs, the Committee did not want to force programmers into writing portably, to preclude the use of C as a “high-level assembler;” the ability to write machine-specific code is one of the strengths of C. It is this principle which largely motivates drawing the distinction between strictly conforming program and conforming program.” ( http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2021.htm)
Give me an example of an architecture where a properly aligned structure where sizeof(struct x)%sizeof(int32) == 0 cannot be accessed by int32s ? Maybe the itanium, but I doubt it. Again: every major OS turns off strict alias in the compilers and they seem to work. Furthermore, the standard itself permits aliasing via char* (as another hack). In practice, more architectures have trouble addressing individual bytes than addressing int32s.
I’d really like to see more alias analysis optimization in C code (and more optimization from static analysis) but this poorly designed, badly thought through approach we have currently is not going to get us there. To solve any software engineering problem, you have to first understand the use cases instead of imposing some synthetic design.
Anyways off the airport. Later. vy
-
I’m willing to agree with you that the aliasing version more clearly shows intent in this specific case but then I ask, what do you do when the code aliases a struct that isn’t properly aligned? There are a lot of solutions but in the spirit of C, I think the right answer is that it is undefined.
So I think what you want is the standard to define one specific instance of previously undefined behavior. I think in this specific case, it’s fair to ask for locally aliasing an int32-aligned struct pointer to an int32 pointer to be explicitly defined by the standards committee. What I think you’re ignoring, however, is all the work the standards committee has already done to weigh the implications of defining behavior like that. At the very least, it’s not unlikely that there will be machines in the future where implementing the behavior you want will be non-trivial. Couple that with the burden of a more complex standard. So maybe the right answer to maximize global utility is to leave it undefined and to let optimization-focused coders use implementation-defined behavior when it matters but, as I’m arguing, use memcpy() by default. I tend to defer to the standards committees because I have read many of their feature proposals and accompanying rationales and they are usually pretty thorough and rarely miss things that I don’t miss.
Everybody arguing here loves C. You shouldn’t assume the standards committee is dumb or that anyone here wants C to be something it’s not. As much as you may think otherwise, I think C is good as it is and I don’t want it to be like other languages. I want C to be a maximally portable implementation language. We are all arguing in good faith and want the best for C, we just have different ideas about how that should happen.
-
what do you do when the code aliases a struct that isn’t properly aligned? There are a lot of solutions but in the spirit of C, I think the right answer is that it is undefined.
Implementation dependent.
Couple that with the burden of a more complex standard.
The current standard on when an lvalue works is complex and murky. Wg14 discussion on how it applies shows that it’s not even clear to them. The exception for char pointers was hurriedly added when they realized they had made memcpy impossible to implement. It seems as if malloc can’t be implemented in conforming c ( there is no method of changing storage type to reallocate it)
C would benefit from more clarity on many issues. I am very sympathetic to making pointer validity more transparent and well defined. I just think the current approach has failed and the c89 error has not been fixed but made worse. Also restrict has been fumbled away.
-
-
-
The chance of overflowing the stack is remote, since stacks now automatically grow and structs tend to be < 512 bytes, but if that is a legitimate concern you can
… just copy the ints out one at a time :) https://godbolt.org/g/g8s1vQ
The compiler largely sees this as a (legal) version of the OP’s code, so there’s basically zero chance it won’t be optimised in exactly the same way.
-
-
-
-
You don’t need a large buffer. You can memcpy the integers used for the calculation out one at a time, rather than memcpy’ing the entire struct at once.
Your designation of using memcpy as a “stupid hack” is pretty biased. The code you posted can go wrong, legitimately, because of course it invokes undefined behaviour, and is more of a hack than using memcpy is. You’ve made it clear that you think the aliasing rules should be changed (or shouldn’t exist) but this “evidence” you’ve given has clearly been debunked.
-
Funny use of “debunked”. You are using circular logic. My point was that this aliasing method is clearly amenable to optimization and vectorization - as seen. Therefore the argument for strict alias in the standard seems even weaker than it might. Your point seems to be that the standard makes aliasing undefined so aliasing is bad. Ok. I like your hack around the hack. The question is: why should C programmers have to jump through hoops to avoid triggering dangerous “optimizations”? The answer: because it’s in the standard, is not an answer.
-
Funny use of “debunked”. You are using circular logic. My point was that this aliasing method is clearly amenable to optimization and vectorization - as seen
You have shown a case where, if the strict aliasing rule did not exist, some code could [edit] still [/edit] be optimised and vectorised. That I agree with, though nobody claimed that the existence of the strict aliasing rule was necessary for all optimisation and vectorisation, so it’s not clear what you do think this proves. Your title says that the optimisation is BECAUSE of aliasing, which is demonstrably false. Hence, debunked. Why is that “funny”? And how is your logic any less circular then mine?
The question is: why should C programmers have to jump through hoops to avoid triggering dangerous “optimizations”?
Characterising optimisations as “dangerous” already implies that the code was correct before the optimisation was applied and that the optimisation can somehow make it incorrect. The logic you are using relies on the code (such as what you’ve posted) being correct - which it isn’t, according to the rules of the language (which, yes, are written in a standard). But why is using memcpy “jumping through hoops” whereas casting a pointer to a different type of pointer and then de-referencing it not? The answer is, as far as I can see, because you like doing the latter but you don’t like doing the former.
-
-
-
-
-
-
-
-
-
-
I no longer believe that daemons should fork into the background. Most Unix systems now have better service control and it makes the code easier to deal with if it doesn’t call
fork(). This makes it easier to test (no longer do you have to provide an option not tofork()or an option tofork()) and less code is always better.-
-
-
-
-
It’s even simpler when you have daemon(3): http://man7.org/linux/man-pages/man3/daemon.3.html
-
Which you do on OpenBSD, actually.
Note that daemon(3) is a non-standard extension so it should be avoided for portable code. The implementation is simple enough, though.
-
-
-
I’m not sure this is accurate, at least on -current. There are several go “deamons” that as far as I understand don’t support
fork(2). These can still be managed by OpenBSD’s rc system:# cd /etc/rc.d # cat grafana #!/bin/ksh # # $OpenBSD: grafana.rc,v 1.2 2018/01/11 19:27:10 rpe Exp $ daemon="/usr/local/bin/grafana-server" daemon_user="_grafana" daemon_flags="-homepath /usr/local/share/grafana -config /etc/grafana/config.ini" . /etc/rc.d/rc.subr rc_bg=YES rc_reload=NO rc_cmd $1I’m not sure if there’s more to it that I don’t understand, I don’t write many deamons!
-
Well, it turns out, I can’t read! The key to this is
rc_bg, see https://man.openbsd.org/rc.subr#ENVIRONMENT
-
-
-
-
-
GeoffWozniak
2 months ago
|
link
| on:
Forget reusability, aim for perfection: New lessons from mandoc
Do not prematurely introduce dependencies.
Worrying about performance is vastly overvalued. Good performance is a by-product of pursuing other goals, mainly simplicity.
This is largely true, based on my experience. Good advice to follow.
Do not use Markdown. [The essay referred to in the slides.]
I think the author is a bit harsh on Markdown, especially if you read the linked essay. It was designed for writing for the web, and was meant to be converted to HTML only. The author criticizes it for lacking things it was never meant to have.
But I do agree that it’s terrible for technical documentation and it’s a travesty that it’s become the norm in that respect. I recently switched to using Markdown for some internal documentation at work mostly because Bitbucket will render it and it (hopefully) encourages others on the team to write docs. When I’m writing with it, though, it’s a mess. No description lists, linking between things is ugly, and you have to manually make thing like a table of contents. I’d switch to mdoc but then I’m worried no one else will bother to writing anything. Maybe that will happen anyway with the cruddiness of Markdown, so perhaps it won’t matter.
(Bonus: in the linked essay, I learned that GNU is apparently going to kill off info. No arguments here!)
-
it’s a travesty that it’s become the norm in that respect
I believe we mostly have GitHub becoming the norm to thank for that. Its native formatting for rich presentation is Markdown via the
README.md. With GitHub being massively adopted and Markdown being the least path of resistance, it’s hardly surprising how this came to be. So that hampers mdoc adoption.Because everybody writes their documentation in
README.mdnow (if at all), they also expect Markdown to man page converters. Those emit man(7) more or less by necessity. People unfamiliar with mandoc won’t care, but those that are may be annoyed by the semantic information that is lost. Not only because mandoc produces worse HTML output because of it, but also because mandoc’s semantic search won’t work for those man pages. However, the group that are unfamiliar with mandoc intersects more or less entirely with the group of people who writes Markdown exclusively. This further hampers mdoc adoption.We have a man page and documentation problem. And there doesn’t seem to be a way to help it.
-
You’ve long been able to render AsciiDoc, org, rST, among other lightweight markup languages, to HTML with GitHub. For example:
I dunno why Markdown “won”, but maybe it had something to do with:
- (at one time) an informal spec, easy to implement (though, probably, with bugs) in a couple days
- the use by Reddit for its comment system (compare w/ bbcode)
- good marketing - gruber was already a known figure when he unleashed markdown, was able to publicize it with his widely-read personal blog
-
-
(Bonus: in the linked essay, I learned that GNU is apparently going to kill off info. No arguments here!)
Looks like that was from 2014 and I can’t tell if the proposed replacement is genuine or a very bad joke. Either way, it doesn’t feel like much has changed in 4 years unfortunately.
-
you have to manually make thing like a table of contents
This is a specific solution (VSCode specific), but I discovered markdown-toc recently and I like it. There are other tools that can be used to add a TOC to a markdown based on headings.
I actually find Markdown convenient for Readmes and user land documentation. Because I can get by with very little markup and the markup reads fine as text it encourages me to write.
I can totally see how it would be a pain for technical documentation for code bases, but I suspect you would use .rst and an assist from an automatic documentation generator for that.
-
I am regretting the Markdown choice already and I’ll probably switch to something else that can export Markdown so it can be read directly while browsing the repo.
So yeah, whether that’s .rst or not remains to be seen, but the pattern is the same.
-
-
-
I love how “boring” syspatch is. freebsd-update is certainly more complex, but does a pretty solid job itself.
I am, however, a bit concerned by FreeBSD’s “packaging base” movement, both in terms of complexity (lots of tiny packages), and how long it seems to be taking.
-
-
Slide 15 mentions that deraadt@ signs the patch file. This made me wonder: What happens if he unexpectedly falls critically ill or dies? For at least one OpenBSD release cycle, there might just be no way to sign anything.
-
-
It bears noting that the OpenBSD project’s man page viewer is actually a CGI program written in C.
-
It’s fast, too. I wonder if it would be faster if they did something really hardcore like write it in a language that’s less tame. Like this.
-
It may be worth noting that copyright on Windows 95 is still intact. In all likelihood, this infringes upon Microsoft’s copyright.