1. 4

    And for those of us who use or want to use C/C++ for our Internet (specifically, web) systems, please let me know if https://learnbchs.org is helpful for tools and deployment strategies that aren’t limited to “use rust”. (Disclaimer: I wrote it.)

    1. 2

      Hi, yeah, personally I would be put off by that stack? OpenBSD is great and I’ve been meaning to deploying something with httpd for a minute now just to play with it, but using sqlite for a web app is a deal breaker and C as a web language…also a deal breaker. Just not what i would use personally, for a lot of different reasons. But thanks for the link and the site looks great.

    1. 2

      man abort on a Debian machine doesn’t give me any jokes. Is there a difference between the glibc html and the manpage? If so—why?

      1. 3

        The man page is not associated with Glibc and was written by other authors. It is part of the Linux man pages project. See https://www.gnu.org/software/libc/documentation.html:

        The glibc project does not maintain any man-pages; instead other projects document the glibc APIs as man-pages. The largest such collection is the Linux man-pages project. The glibc community keeps close ties with the Linux man-pages project, and many developers contribute to both sources of documentation. While the glibc manual remains the canonical source for API descriptions, the man-pages are an excellent reference. Every effort is made to keep the documentation harmonized between both projects.

        More importantly, though, the Glibc manual is generally more conversational in tone and style. There is no reason why non-technical information from the Glibc manual should be replicated in man pages.

      1. 1

        Not sure why the DB has to be on the servers (to be monitored), and not on the client/monitoring server. Personally I try to keep the monitoring workload on the servers as minimal as possible, and do all the aggregation ++ on a separate server.

        If they had done that, they also could have used snmpd from the base system, without the need for writing a collector daemon.

        1. 3

          It’s on both. The database is really just a set of circular buffers that’s transmitted in totality. But it’s on the system being monitored because hey, you might turn off the monitor. And then your history is gone. Also, even if the monitor is running, the network might fail.

          1. 3

            This is something I wish the Prometheus people had factored in to their design. I’d love to be able to restart the collectors or have network disruptions without losing data points. A locally spooled catch-up ring buffer is a great idea.

            1. 1

              A locally spooled catch-up ring buffer is a great idea.

              Would you persist it to disk? Just in case the monitored server crashes while the buffer is not empty.

              1. 2

                Persisting to disk would definitely be something I’d like to explore. I think the downside might be that it induces extra I/O load and storage pressure on what might already be a busy system.

                1. 1

                  I’m not sure you’re referring to slant, but it leaves the database on the disc. The point of slant is to be a simple tool for normative hosts: if your systems are so significantly under load, you probably want to do something like snmp.

                  That being said, I’m happy to work with folks with patches to make slant more performant. I know that ksql has lots of room for improvement!

                  1. 2

                    I’m not sure you’re referring to slant, but it leaves the database on the disc.

                    Sorry, I was talking more in the abstract – we’re using the Prometheus ecosystem pretty heavily at the moment.

                    It’s neat that slant persists to disk!

        1. 3

          One reason I think sblg isn’t as popular as this kind of tool is the auto-theme part. Like, being able to do git clone to pull down a template Makefile, blog-template.xml, and article-template.xml with associated CSS files with some kind of unifying theme. Is that something that folks use, or does one usually start from scratch when designing a blog site?

          1. 2

            I don’t know, I don’t think themes (as in layout) are important, but themes (as in example code for all the structures supported by the tool) are important. My website runs on an ancient version of hugo and probably won’t build with a current one without serious changes, but it’s a SSG with only user-supplied input by me, so I don’t care.

            I made my own theme but tbh I don’t remember if I did it from scratch or just ported it from hyde, which I was using before that. I probably did it from scratch for hyde, as I don’t have any acknowledgements in that repo. It was enormously helpful to see good examples for hugo and be able to see examples in themes, as for example good RSS support was very important for me.

            But I’ve only once run some kind of “quick, I need a website!” thing off of any SSG, and then I used jekyll because it was a team blog that should just work without any tinkering and only minimal effort to use a basic theme.

            1. 1

              Oh, Nice. I haven’t heard about this tool. I was just testing some of the static site generators. So, I did a small google search and found Hugo, Jekyll, etc. are one of the good static site generators. And, I have heard about ssg (https://www.romanzolotarev.com/ssg.html) - static site generator tool.

              I can’t compare these tools because I think I am not the right person to do so. And, as I am not a very huge fan of the website development, I was searching for something where I just need to write my content that’s all with some good minimal visibility and readability theme support. So, I have picked Hugo and started testing it.

            1. 4

              As a researcher, I worked a lot on flocking and game theory. See flocks for how beautiful these can look!

              1. 2

                Note that this was based upon OpenBSD 4.6.

                1. 10

                  If you want proper as in “canonical”, John Gruber’s original is your best choice. It is effectively the definition of markdown. But markdown has moved on beyond Gruber’s original vision, mostly without him. Still, his original vision in collaboration with Aaron Swartz is still very usable, especially for blogging.

                  Once you move beyond the original Perl code, you have numerous choices. Here are a few:

                  And there’s no shortage of online converters and other versions.

                  I’ve used pandoc for years. It doesn’t just convert markdown to html. It converts from multiple formats to multiple formats, including various slide formats as well as PDF. It supports Python and Haskell scripts, as well as Lua scripting. The Lua scripting is very interesting because it manipulates the AST in memory as opposed to Python and Haskell where pandoc marshalls the AST to a JSON representation, pipes it to the script, and then receives the output.

                  Lowdown is interesting because it’s a fork of hoedown. @kristaps went through it and added a proper AST representation internally as well as pledge and privilege separating it. He doesn’t expose the AST (yet?), but it’s fast and efficient.

                  Multimarkdown is a good choice if you want to stay in Markdown but target more than one output format. Like pandoc, mmd can export to several slide formats and pdf by way of LaTeX.

                  Lastly, the lua and awk versions are noted above because I like little languages that do so much. ;-)

                  For most of what markdown was originally designed for, text files converted to HTML, you almost can’t go wrong. Where you’re going to find your choices both expanded and somewhat complicated is when you move beyond basic markdown and start trying to write slides, documentation, books, or use add-on features like tables.

                  I’d suggest start with basic markdown as described on Gruber’s Markdown page, pick one of the simpler converters like his Perl script, and see whether you need anything beyond that. Then look at Pandoc and Multimarkdown. If you don’t need more features but want more speed, then look at Lowdown.

                  1. 2

                    lowdown does have its library exposed! pkg_add lowdown, man 3 lowdown.

                    1. 1

                      Thanks for the detailed answer!

                    1. 3

                      pkg_locate doesn’t find any of these. It does, however, find pelican and my own, sblg. I’ve also heard a lot about Roman’s ssg.

                        1. 4

                          How about “learn javascript” instead of any of its countless frameworks? It’s pretty easy to write compliant vanilla javascript without jumping through per-browser hoops. Same goes with CSS and HTML—maybe learn those too?

                          Though these days, I guess I’d rather suggest people write typescript.

                          1. 4

                            This doesn’t solve a contention problem: it merely notifies folks when the jakes are in use.

                            (A “bathroom contention problem” would probably be solved by some straightforward modelling and controlled distribution of coffee.)

                            1. 11

                              “Unfortunately, the fundamentalist FOSS mentality we encountered on Reddit is still alive and well.” Ok, so this “fundamentalist” attitute, according to the article, is from comments like “This is going to be a very hard sell being a proprietary closed source system to Linux users, many use Linux because they have bought into the idea of open source. Good luck with it anyway”. I understand that this article is just a closed-source product promo (whose claim to fame is interoperating with… another closed-source product), but name-calling folks (probably like myself) who use open source “fundamentalists” is wrong, especially when they give feedback as quoted.

                              Then we have, “we don’t store or process data online — strictly between you and your mail server”, but, “users can turn off what little data collection we do”. Um… pick one.

                              1. 1

                                i guess they meant “unfortunately for us proprietary software programmers who want to exploit users.”

                              1. 3

                                If anybody feels like having some weekend fun, BCHS is always wanting for articles on using these tools! (I use scan-build all the time, though I swear mostly by AFL and valgrind.)

                                  1. 1

                                    Sorry, too late to edit it now as well. I wonder if a moderator can change it

                                  1. 0

                                    I propose having several API releases with absolutely nothing in them so that browsers can catch up with the “extremely exciting” eye cancer already there. See caniuse browser scores for more. Also, I would like a pony.

                                    1. 3

                                      If you would like to criticise this API or web APIs in general, could you please do it in some sort of constructive manner? I have no idea what your comment was about, other than that you seem angry about something.

                                    1. 9

                                      Many of the author’s experiences speaking with senior government match my own.

                                      However, there’s one element that I think is very easily lost in this conversation, and which I want to highlight: there is no group I spend more time trying to convince of the importance of security than other software engineers.

                                      Software engineers are the only group of people I’ve ever had push back when I say we desperately need to move to memory safe programming languages. All manner of non-engineers, when I’ve explained the damages wrought by C/C++, and how nearly every mass-vulnerability they know about has a shared root cause, generally understand why this is an important problem, and want to discuss ideas about how do we resolve this.

                                      Engineers complain to me that rewriting things is hard, and besides if you’re disciplined in writing C and use sanitizers and fuzzers you’ll be ok. Rust isn’t ergonomic enough, and we’ve got a really good hiring pipeline for C++ engineers.

                                      If we want to build software safety into everything we do, we need to get engineers on board, because they’re the obstacle.

                                      1. 11

                                        People don’t even use sanitizers and fuzzers, so I’m not sure why you would expect them to rewrite in Rust. It’s literally 1000x less effort.

                                        As far as I can tell, CloudFlare’s CloudBleed bug would have been found if they compiled with ASAN and fed about 100 HTML pages into it. You don’t even have to install anything; it’s built right into your compiler! (both gcc and Clang)

                                        I also don’t agree that “nearly every mass vulnerability has a shared root cause”. For example, you could have written ShellShock in Rust, Python, or any other language. It’s basically a “self shell-code injection” and has very little to do with memory safety (despite a number of people being confused by this.)

                                        The core problem is the sheer complexity and number of lines of unaudited code, and the fact that core software like bash has exactly one maintainer. There are actually too many people trying to learn Rust and too few people maintaining software that everybody actually uses.

                                        In some sense, Rust can make things worse, because it leads to more source code. We already have memory-safe languages: Python, Ruby, JavaScript, Java, C#, Erlang, Clojure, OCaml, etc.

                                        Software engineers should definitely spend more time on security, and need to be educated more. But the jump to Rust is a non-sequitur. Rust is great for kernels where the above languages don’t work, and where C and C++ are too unsafe. But kernels are only a part of the software landscape, and they don’t contain the majority of security bugs.

                                        I would guess that most data breaches these days have nothing to do with memory safety, and have more to do with bugs similar to the ones in the OWASP top 10 (e.g. XSS, etc.)

                                        https://www.owasp.org/images/7/72/OWASP_Top_10-2017_%28en%29.pdf.pdf


                                        Edit: as another example, Mirai has nothing to do with memory safety:

                                        https://en.wikipedia.org/wiki/Mirai_(malware)

                                        All it does it try default passwords, which gives you some idea of where the “bar” is. Rewriting software in Rust has nothing to do with that, and will actually hurt because it takes effort and mindshare away from solutions with a better cost/benefit ratio. And don’t get me wrong, I think Rust has its uses. I just see people overstating them quite frequently, with the “why don’t more people get Rust?” type of attitude.

                                        1. 2

                                          There were languages like Opa that tried to address what happened on web app side. They got ignored just like people ignore safety in C. Apathy is the greatest enemy of security. It’s another reason we’re pushing the memory-safe, higher-level languages, though, with libraries for stuff likely to be security-critical. The apathetic programmers do less damage on average that way. Things that were code injections become denial of service. That’s an improvement.

                                        2. 2

                                          not only software engineers, almost the entire IT industry has buried it’s head in the sand and is trying desperately hard to hide from the problem, because “security is too hard”. We are pulling teeth to get people to even do the minimal upgrades to things. I recently had a software vendor refusing to support anything other than TLS 1.0. After many exchanges back and forth, including an article from Microsoft(and basically every other sane person) saying they were dropping all support of older TLS protocols because of their insecurity, they finally said, OK we will look into it. I’m sure we all have stories like this.

                                          If you can’t even bother to take the minimum of steps to upgrade your security stacks after more than a decade,(TLS1.0 released in 1999 and TLS 1.2 is almost exactly a decade old now) because it’s “too hard”, trying to get people to move off of memory unsafe languages like C/C++ is a non-starter.

                                          But I agree with you, and the author.

                                          1. 2

                                            I would like to use TLS 1.3 for an existing product. It’s in C and Lua. The current system is network driven using select() (or poll() or epoll() depending upon the platform). The trouble I’m having is finding a library that is easy, or even a bit complicated but sane to use. The evented nature means I an notified when data comes in, and I want to feed this to the TLS library instead of having the TLS library manage the sockets for me. But the documentation is dense, the tutorials only cover blocking calls, and that’s when they’re readable! Couple this with the whole “don’t you even #$@#$# think of implementing crypto” that is screamed from the roof tops and no wonder software engineers steer away from this crap.

                                            I want a crypto library that just handles the crypto stuff. Don’t do the network, I already have a framework for that. I just need a way to feed data into it, and get data out of it, and tell me if the certificate is good or not. That’s all I’m looking for.

                                            1. 2

                                              OpenBSD’s libtls.

                                              1. 2

                                                TLS 1.3 is not quite ready for production use, unless you are an early adopter like Cloudfare. Easy to use API’s that are well-reviewed are not there yet.

                                                Crypto Libraries: OpenBSD’s libtls like @kristapsdz mentioned, or libsodium/nacl or OpenSSL. If it’s just for your internal connections and don’t actually need TLS, just talking to libsodium or NaCL for an encrypted stream of bytes is probably your best bet, using XSalsa20+Poly1305. See: https://latacora.singles/2018/04/03/cryptographic-right-answers.html

                                                TLS is a complicated protocol(TLS1.3 reduces a LOT of complexity, it’s still very complicated).

                                                If you are deploying to Apple, Microsoft or OpenBSD platforms, you should just tie to the OS provided services, that provide TLS. Let them handle all of that for you(including the socket). Apple and MS platforms have high-level API’s that will do all the security crap for you. OpenBSD has libtls.

                                                On other platforms(Linux, etc), you should probably just use OpenSSL. Yes it’s a fairly gross API, but it’s pretty well-maintained nowadays(5 years ago, it would not qualify as well maintained.). The other option is libsodium/NaCL.

                                                1. 1

                                                  Okay, fine. Are there any crypto libraries that are easy to use for whatever is current today? My problem is: a company that is providing us information today via DNS has been invaded by a bunch of hipster developers [1] who drunk the REST Kool-Aid™ so I need a way to make an HTTPS call in an event driven architecture and not blow our Super Scary SLAs with the Monopolistic Phone Company (which would case the all-important money to flow the other way), so your advice to let OS provided TLS services control the socket is a non-starter.

                                                  And for the record, the stuff I write is deployed to Solaris. For reasons that exceed my pay grade.

                                                  So I read the Cryptographic Right Answers you linked to and … okay. That didn’t help me in the slightest.

                                                  The program I’m working on is in C, and not written by me (so it’s in “maintenance mode”). It works, and rewriting it from scratch is probably also a non-starter.

                                                  Are you getting a sense of the uphill battle this is?

                                                  [1] Forgive my snarky demeanor. I am not happy about this.

                                                  Edit: further clarification on what I have to work with.

                                                  1. 1

                                                    I get it, it sucks sometimes. I’m guessing you are not currently doing any TLS at all? So you can’t just upgrade the libraries you are currently using for TLS, whatever they are.

                                                    In my vendor example, the vendor already implemented TLS (1.0) and then promptly stopped. They have never bothered to upgrade to newer versions of TLS. I don’t know the details of their implementation, obviously, since it’s closed-source; but unless they went crazy and wrote their own crypto code, upgrading their crypto libraries is probably all that’s required. I’m not saying it’s necessarily easy to do that, but this is something everyone should do at least once every decade, just to keep the code from rotting a terrible death anyways. TLS 1.2 becomes a decade old standard next month.

                                                    I don’t work on Solaris platforms (and haven’t in at least a decade, so you are probably better off checking with other Solaris people). Oracle might have a TLS library these days, I have no clue. I tend to avoid Oracle land whenever possible. I’m sorry you have to play in their sandbox.

                                                    I agree the Crypto right-answers page isn’t useful for you, since you just want TLS, It’s target is for developers who need more than TLS. I used it here mostly as proof of why I recommended XSalsa20+Poly1305 for symmetric encryption. Again, you know you need TLS, so it’s a non-useful document for you at this point.

                                                    Event driven IO is possible with OpenSSL, but it’s not super easy see: https://www.openssl.org/docs/faq.html#PROG11. Then again, nothing around event driven IO is super easy. Haproxy and Nginx both manage to do it, and are both open-source implementations of TLS, so you have working code you can go examine. Plus it might give you access to developers who have done event driven IO with TLS. I haven’t ever written that implementation, so I can’t help with those specifics.

                                                    OpenSSL is working on making their API’s easier to use, but it’s a long, slow haul, but it’s definitely a known problem, and they are working on it.

                                                    As for letting the OS do the work for you, you are correct there are definitely use-cases where it won’t work, and it seems you fit the bill. For most applications, letting the OS do it for you is generally the best answer, especially around Crypto which can be hard to get right, and of course only applies to the platforms that offer such things(Apple, MS, etc). Which is why I started there ;)

                                                    Anyways, good luck! Sorry I can’t just point to a nice easy example, for you. Maybe someone else around here can.

                                                    1. 1

                                                      I’m not even using TCP! This is all driven with UDP. TCP complicates things but is manageable. Adding a crap API between TCP and my application? Yeah, I can see why no one is lining up to secure their code.

                                                      1. 1

                                                        I think there is a communication issue here.

                                                        The vendor you are connecting with over HTTPS supports UDP packets on a REST API interface? really? Crazier things have happened I guess.

                                                        I think what you are saying is you are doing DNS over UDP for now, but are being forced into HTTPS over TCP?

                                                        DNS over UDP is very far away from a HTTPS rest API.

                                                        Anyways, for being an HTTPS client, against a HTTPS REST API over TCP, you have 2 decent options:

                                                        Event driven/async: use libevent, example code: https://github.com/libevent/libevent/blob/master/sample/https-client.c

                                                        But most people will be boring, and use something like libcurl (https://curl.haxx.se/docs/features.html) and do blocking I/O. If they have enough network load, they will setup a pool of workers.

                                                        1. 2

                                                          Right now, we’re looking up NAPTR records over DNS (RFC-3401 to RFC-3404). The summary is that one can query name information for a given phone number (so 561-555-5678 is ACME Corp.). The vendor wants to switch to a REST API and return JSON. Normally I would roll my eyes at this but the context I’m working in is more realtime—as in Alice is calling Bob and we need to look up the information as the call is being placed! WE have a hard deadline with the Monopolistic Phone Company to provide this information [1].

                                                          We don’t use libevent but I’ll look at the code anyway and try to make heads and tails.

                                                          [1] Why are we querying a vendor this for? Well, it used to be in house, but now “we lease this back from the company we sold it to - that way it comes under the monthly current budget and not the capital account.” (at least, that’s my rational for it).

                                                          1. 2

                                                            Tell me how it goes. Fwiw, you might want to take a quick look at mbed TLS. Sure it wants to wrap a socket fd in its own context and use read/write on it, but you can still poll that fd and then just call the relevant mbedtls function when you have data coming in. It does also support non-blocking operation.

                                                            https://tls.mbed.org/api/net__sockets_8h.html#a2ee4acdc24ef78c9acf5068a423b8c30 https://tls.mbed.org/api/net__sockets_8h.html#a03af351ec420bbeb5e91357abcfb3663

                                                            https://tls.mbed.org/api/structmbedtls__net__context.html

                                                            https://tls.mbed.org/kb/how-to/mbedtls-tutorial (non-blocking io not covered in the tutorial but it doesn’t change things much)

                                                            I’ve no experience with UDP (yet – soon I should), but if you’re doing that, well, mbedtls should handle DTLS too: https://tls.mbed.org/kb/how-to/dtls-tutorial (There’s even a note relevant to event based i/o)

                                                            We use mbedtls at work in a heavily event based system with libev. Sorry, no war stories yet, I only got the job a few weeks ago.

                                                            1. 1

                                                              Right, let’s add MORE latency for a real-time-ish system. Always a great idea! :)

                                            1. 6

                                              Note: not about manpages.

                                                1. 20
                                                    Maybe
                                                  
                                                  
                                                    Less
                                                  
                                                   
                                                    Spread out?
                                                  
                                                  1. 1

                                                    no ahahahha

                                                  1. 1

                                                    Are there any benchmarks for this?

                                                    1. 2

                                                      Only thing I see is the the performance graph on the author’s page here: https://kristaps.bsd.lv/kcgi/

                                                      1. 1

                                                        I’ve long wanted to update these with some good measurements against, say, PHP. (And on OpenBSD, too.) It’s important to have a solid measure of the performance trade-off between CGI with a compiled binary and the FastCGI clones (Python’s, PHP’s, etc.) alongside the security benefits of ephemeral processes.

                                                        1. 1

                                                          Wow. Thanks for that.

                                                          15msec response sounds like an eternity. My server responds in micros over loopback, so what’s going on?

                                                          Is there an easy way to test this?