Threads for adsouza

  1. 4

    Here’s the definition I’ve been using:

    Software development is about using code to create value. Software engineering is doing that in the context of change.

    1. 4

      I prefer Rob Pike’s variation:

      Software engineering is what happens to programming when you add time and other programmers.

      1. 3

        I believe that credit for that in the Go community goes to Russ Cox, from his What is Software Engineering? essay. Russ goes on to cite Titus Winters (7 minutes of video here).

        1. 2

          Thanks for the correction. I had actually looked it up to check the reference, found the article you linked, but misread the opening paragraph as Russ quoting Rob Pike, which he was not doing.

          Titus Winter’s original:

          Software engineering is programming integrated over time

          And Russ gets credit for the variation I quoted.

          1. 2

            No problem. I’m fond of Russ’ quote. I lead a “Computational Engineering” team at a biotech, recruiters have told me that I should get the name changed because potential recruits can’t imagine their work.

            I finally settled on (giving Russ his nod):

            Computational Engineering is what happens to Bioinformatics and Infrastructure when they meet time and teammates.

      2. 2

        That’s 1 of the 2 important differences; the other is collaboration; software engineering involves working on a codebase with other people, which means you need to prioritize things like readability, integration tests, documentation, etc.

        1. 1

          That’s interesting, so do you think it’s impossible to do software engineering alone?

          1. 1

            Not impossible; just far less necessary (and atypical). There are rare cases of 1 person working on a project large & complex enough to be worth the effort of applying engineering principles (e.g. Austern Meyers’ X-Plane) but the vast majority of codebases with only 1 contributor are pretty small.

            1. 1

              My approach is, that a code written alone will be read by at least one person who thinks differently: the future you.

              1. 1

                That’s what the time factor is for.

      1. 2

        This is very impressive. Can’t wait to see it turned into a filter in Instagram or Google Photos…

        1. 1

          I tried this on my M1 Mac & it didn’t work. Is that because the binary code is x86-only?

          1. 1

            Yup, only works with x86 instruction set

            1. 23

              What is the most cursed thing you have used redbean for or seen it used for?

              1. 4

                This sounds amazingly useful. With what aspect of it are you least satisfied?

                1. 6

                  Thanks! Redbean has baked-in SSL support. I used MbedTLS and I love it, because it’s so much smaller and simpler than OpenSSL and BoringSSL which have huge dependencies like STL and Perl which would have needed to be ported beforehand. However its main drawback is it doesn’t have TLSv3 support. This makes me sad because it causes redbean to have a few extra milliseconds of connection latency if it’s used for self-contained public-facing serving. That’s something I intend to improve. However it’s not that big of an issue, since it’s very easy to use redbean in tandem with something like stunnel which can bolt on the SSL separately. But that only applies over long line internet connections. For local use cases (redbean’s primary and original intent) where CPU utilization is a greater factor than the speed of light, redbean’s SSL goes faster than any of the other stacks I’ve evaluated. For example, I haven’t posted SSL benchmarks on the website yet because wrk (Go) uses 2x as much CPU to establish ECDSA connections as redbean. So once it has the TLSv3 support, it’ll be a sight to behold.

                  1. 1

                    I thought wrk was written in C! Is the code at https://github.com/wg/wrk/tree/master/src not the correct wrk?

                2. 3

                  This is so f’in cool.

                  Question: I get the “serve static files” part, and “I can do dynamic stuff with Lua” part. How does the sqlite part work? Can I literally write a full-fledged web app in lua which is, say, saving new user records using sqlite, and those are just getting written to the filesystem in the folder redbean is running in?

                  1. 5

                    Indeed. Here’s the basic idea for how you’d implement a SQLite web service with redbean. https://gist.github.com/jart/f9b1a7ebe5c7ebea3abe8f98125c692c We’re in the process of adding features so that redbean can use its own .com file as the database. We have one function for just that, called StoreAsset() but right now SQLite is more stable.

                    1. 2

                      It would be cool to pair redbean with Litestream to get streaming backups of the SQLite db.

                  2. 2

                    I’m a huge fan of your work. Also fare’s, cool he invited you. Cool you are here.

                    1. 3

                      Thank you! Lobsters in a nice community I regularly visit. I was so happy to see that someone linked to redbean.

                    2. 2

                      How does it handle sqlite queries with database modification, e.g. INSERT / UPDATE / DELETE ?

                      1. 4

                        redbean simply embeds sqlite with lua bindings thereby enabling you to handle them in any manner you want. It’s expected that the user knows SQLite and is aware of its caveats. For example, redbean is a forking web server and SQLite file descriptors should never cross fork() boundaries (if the database is mutable). This isn’t a problem if you’re using Lua Server Pages. However if you’re using the faster OnHttpRequest() handler then you need to make sure you don’t call sqlite3.open() from the global scope and instead create it lazily as requests are handled. That means your database descriptor and prepared statements will persist across requests on a per-connection basis. This is nice since SQLite is significantly faster than other databases at doing this. Plus you get a nice process sandbox. So in case something goes wrong, like ASAN detects a bug, it’ll only impact one serving session. We’re also working on higher-level frameworks to abstract a lot of the handling aspects. Lastly I’ve also been focusing on creating single-file object store technology (see StoreAsset() and LoadAsset()) that should provide the fastest possible and most easily redistributable model of data storage under a forking model.

                        Example https://gist.github.com/jart/f9b1a7ebe5c7ebea3abe8f98125c692c

                        1. 1

                          However if you’re using the faster OnHttpRequest() handler then you need to make sure you don’t call sqlite3.open() from the global scope and instead create it lazily as requests are handled. That means your database descriptor and prepared statements will persist across requests on a per-connection basis.

                          Sorry, but I’m not sure what that means. If each request gets its own SQLite descriptor in OnHttpRequest(), why would “database descriptor and prepared statements persist across requests”?

                          1. 2

                            Because redbean supports HTTP keepalive pipelining and Lua variables are global unless you declare you function local. Forking happens when a TCP connection is received. That connection may send many HTTP messages. Each message for that connection is handled by the same process. If you’ve defined a hook like OnHttpRequest then that means any variables you use, unless you use local, will persist between messages so you can memoize operations on a per-session basis. When the TCP connection closes, it all gets garbage collected by the operating system when the process exits.

                            1. 1

                              I see. Does that imply we can call sqlite3.open() from the global scope in Lua, since by that time we are already in a fork()ed child process for that TCP connection? You said we shouldn’t do that and should “instead create it lazily as requests are handled” so I am a little confused.

                              1. 2

                                If it’s a foo.lua file in your redbean (i.e. a Lua Server Page) then you’re 100% OK. Those files are loaded and re-run on each request. There’s only one file that’s special and it’s .init.lua. That file is run before redbean calls bind() and no forks have been created yet. Anything you do in the global scope of that file, like import modules, will end up defining the main process “template” from which all forked processes are created. Therefore, if you create your SQLite object in the global context of that file, it needs to be strictly read-only. If it is, you can get like 800k RPS performance on read-only SQLite files. But any SQLite files you intend to mutate, it needs to happen after fork, otherwise corruption is possible. It also goes without saying, that you can have multiple SQLite file descriptors open simultaneously! So for example, if you built an Internet forum, all your years-old posts could be moved to immutable read-only SQLite files, for ultra-fast querying due to the pre-fork setup.

                                1. 1

                                  That’s awesome! I have an additional question: is it theoretically possible for redbean to support LuaJIT in the future? I’m not sure if the magic tricks you did with actually portable executables would interfere with JIT compilation/execution.

                                  1. 2

                                    It’s a work in progress. See https://github.com/ahgamut/LuaJIT-cosmo

                    1. 1

                      I couldn’t figure out which languages it supports.

                      1. 2

                        The README.md says:

                        Tagref works with any programming language

                        But I think I should make this more prominent! Thanks for the feedback.

                      1. 1

                        Conventional OSes have been doing a simpler version of this when they go into “hibernation” but the big difference here is that the OS updates that state constantly, essentially treating all of RAM as a cache for a virtual memory paging file. Perhaps even more intriguing is the cohesive simplicity of the programming language that accompanies this system: the demo app manages to load data from the Internet & render it graphically without needing to fiddle with a complex web of 3rd-party deps.

                        1. 4

                          Conventional OSes have been doing a simpler version of this when they go into “hibernation” but the big difference here is that the OS updates that state constantly, essentially treating all of RAM as a cache for a virtual memory paging file

                          The difference is that you can update the kernel but not lose any application state. Hypervisors have done this for a little while, loading a new version underneath running guests. And, of course, hypervisors have been able to suspend processes^W VMs to disk and resume them later. There have been patches floating around for a few years for both FreeBSD and Linux to allow suspending a single process to disk and resuming it but it gets a bit exciting when the process has open file handles because you might not be able to restore state. For a disk it’s mostly fine (unless that disk has gone away) because the filesystem is a shared resource and programs need to be somewhat robust in the presence of concurrent modification (they aren’t, in general, but this just increases the concurrency window). It’s mostly fine for networks because networks are expected to go away periodically and so processes generally handle that gracefully. It’s far more complex for local IPC (if I suspend a process with a pipe to another process, what happens? What happens if I restore it later without restoring the target?).

                          Perhaps even more intriguing is the cohesive simplicity of the programming language that accompanies this system

                          This kind of thing is great for simple demos but all single level store systems eventually end up inventing some kind of filesystem because you discover the need to:

                          • Update the code for the persistent process.
                          • Modify the internal data structures of the persistent process.

                          The first of these leads to the need for a clean separation between code and data and a model for stopping all threads and then starting them again. The second leads to a need for some data serialisation and deserialisation format.

                          That’s not to say that a conventional filesystem is the right abstraction. It conflates persistence, namespacing, and communication in some quite unfortunate ways.

                          1. 1

                            you discover the need to:

                            • Update the code for the persistent process.
                            • Modify the internal data structures of the persistent process

                            Common lisp provides in-band methods for achieving both of these; for the former, you can generally just redefine things; for the latter, you have tools like change-class and update-instance-for-redefined/different-class.

                            (I also do not buy your implicit premise that something like the traditional process model could not function in such a system.)

                        1. 1

                          Wow. Talk about hardware problems. The phone won’t even boot without a charged battery in some cases and you can’t charge the battery because it tries to boot when external power is attached.

                          Even if you get the phone booted, you need to do command line stuff to get charging working properly.

                          1. 2

                            I have a PinePhone as a toy, but never expect it to be viable as anything beyond a hotspot of sorts. I’ve basically always found their products to be marginal with some really poor hardware design choices or outright flaws. And, then the software is totally up to the community with no ongoing support. If you’re into that, great, and I am thankful for all the hard work people do put into the software. So, the battery charging thing is still an issue that was a problem on the non-pro. They created a new version but didn’t clean up basic stuff like this. And, some of what they sell is broken (SATA card for NAS) or I just don’t know why it exists (hard case for PinePhone which is just hard plastic that does nothing or USB keyboard which is among the worst I have ever used but I was tempted by the USB hub and punted on it). I can forgive the software situation since they are quite honest about it, but the hardware issues make them hard to buy from.

                            1. 1

                              I have a PineBook & it’s worked fine for the most part. A couple of times the wifi didn’t work on startup but a reboot fixed that.

                          1. 4

                            I think the post is a bit dismissive of ECS. I ended up going with ECS for my tiny company, because running two EKS clusters (staging and prod), at $0.10/hour each, added appreciably to our AWS bill. But then, we may also be in the minority in thinking that AWS NAT gateways are too expensive.

                            1. 5

                              I think ECS is a great choice for a small company. Sorry if the article gave you a different impression.

                              1. 4

                                No, I agree. 90% of projects would be fine with Heroku. 9% need ECS (and other AWS services). 1% need something even more complicated, like EKS or bare metal.

                                1. 1

                                  Is Heroku similar to AppEngine?

                                2. 3

                                  However if I were a very small company new to AWS I wouldn’t touch this with a ten foot pole. It’s far too complicated for the benefits and any sort of concept of “portability” is nonsense bullshit

                                  I think that’s pretty clear..

                                1. 7

                                  One of the things I wish this article highlighted more is the impact of team churn.

                                  Even with the generous assumption of five year service life for a website, I think that average tenure at a company now is down below something like two years.

                                  In such an environment, you really are best off just writing simple, disposable, easily tweaked code instead of trying to do things as beautifully as possible.

                                  Ablative coding is a whole interesting topic in and of itself.

                                  1. 3

                                    This is a great point.

                                    I’ve certainly done my share of hand-wringing about the long-term impact on the industry of constant job hopping, chiefly the possibility that it will lead to an industry full of people who have never seen their own code become the legacy ball of mud everyone hates to touch.

                                    But maybe there’s a local maximum to shoot for: treating “fully maintainable by a team where nobody has more than a year of experience with the code base” as a first-class design goal, something that we’d be willing to trade away other meaningful goals for. (Today, I think maintainability is often first on the chopping block when it conflicts with other goals.)

                                    What that looks like in practice, I don’t know. In some ways, maybe not too different from what top-flight teams already do: great automated tests, thorough documentation, strong modularity, and well-oiled onboarding processes. But in other ways, maybe pretty different: perhaps you would want to steer well clear of any tools or libraries or frameworks that require deep knowledge, even at the cost of significant developer productivity.

                                    1. 2

                                      perhaps you would want to steer well clear of any tools or libraries or frameworks that require deep knowledge, even at the cost of significant developer productivity.

                                      Highlighting this point because it’s incredibly important.

                                      Consider the difference in max productivity afforded by using, say, Go instead of Ruby or Closure or Haskell. Sure, your best developers are going to be more productive in the “better” languages, but your average developer will see little of that benefit and might go off the rails instead.

                                      1. 2

                                        Another thing to consider is the trade-off between readability & “writability”. Very powerful languages that allow a lot of abstraction can allow programmers to be very efficient/concise when writing new code but then newcomers to the codebase have a steep learning curve when they need to read that code later.

                                  1. 1

                                    I wonder what the best way to provide the author with proofreading feedback is.

                                    1. 1

                                      email works fine. Comments there do also work.

                                      1. 1

                                        I didn’t see a place to leave comments there but will send email.

                                    1. 3

                                      That’s what it says in big bold letters near the top of the Zig language web page. Zig is a recent experiment in cutting away all this derided complexity which plagues modern programming languages. Closures, function traits, operator overloading - programming is hard enough to begin with, why can’t we at least program in a simple language, without all that crap?

                                      That’s true, but not in the same way as, say, Go or a certain popular blogger’s secret WIP language.

                                      Regarding closures, it’s pretty difficult to have closures in a language that’s intended to be as low-level as Zig is. More so when that language aims to not have implicit allocations.

                                      Regarding operator overloading, this is a primary goal of Zig to not have hidden control flow via macros, operator overloading, exceptions, etc, and not so much as a general grudge to “complexity”. I quote:

                                      If Zig code doesn’t look like it’s jumping away to call a function, then it isn’t. This means you can be sure that the following code calls only foo() and then bar(), and this is guaranteed without needing to know the types of anything:

                                      var a = b + c.d;
                                      foo();
                                      bar();
                                      

                                      Examples of hidden control flow:

                                      • D has @property functions, which are methods that you call with what looks like field access, so in the above example, c.d might call a function.
                                      • C++, D, and Rust have operator overloading, so the + operator might call a function.
                                      • C++, D, and Go have throw/catch exceptions, so foo() might throw an exception, and prevent bar() from being called. (Of course, even in Zig foo() could deadlock and prevent bar() from being called, but that can happen in any Turing-complete language.) The purpose of this design decision is to improve readability.

                                      Regarding function traits… well, I’m afraid I have no idea of what those are. A quick Google search didn’t turn up any English I could parse at 10:30 PM.

                                      1. 3

                                        What are operators, if not functions with different syntax?

                                        1. 3

                                          The point is that you can’t define or redefine operators in Zig, so you don’t have to check what they are doing: it’s written in the manual.

                                        2. 1

                                          Nit: Go does not have exceptions.

                                        1. 1

                                          I believe the word you want here is “rendering”.

                                          1. 1

                                            Good grief just pick something already.

                                            This is turning into a undergrad designing their first language.

                                            1. 3

                                              With the number of bad choices requiring huge transition phases a various well designed languages have had why rush now?

                                              1. 2

                                                Quite the opposite. An undergrad would have just picked something & run with it. It takes many years of painful experience to know that this is a very risky proposition.

                                              1. 23

                                                Lol. I’ve spent weeks hunting bugs where the fix ended up literally being a single bit change. To be clear, that’s not intended to be a flex but rather an underlining of the original author’s point that a line of code is not really a metric of anything at all.

                                                1. 26

                                                  I came to say something similar. I recently found a bug in the Linux kernel in an obscure code path that no one except us and paravirtualised MIPS use (which, realistically, means no one except us). Debugging took several days. The fix was changing a 0 to a 1. Even in ASCII in the source code, this was a change of 0x30 to 0x31, so a one-bit fix in both the source and the compiled binary.

                                                  There’s the old (and possibly apocryphal) story of an engineer coming to service something, tapping the machine with a hammer, fixing it and charging $5,000. The customer objected and demanded an itemised bill. They got this:

                                                  • Hitting the machine with a hammer: $1
                                                  • Knowing where to hit it: $4,999

                                                  Fixing a bug is often trivial. Figuring out the correct fix and where to put it is the hard bit. My favourite bug fixes are the ones that involve just deleting code.

                                                  1. 4

                                                    There’s the old (and possibly apocryphal) story of an engineer coming to service something

                                                    Here you are

                                                    1. 1

                                                      This is one of those cases where the 1 byte change should be accompanied by a multi-paragraph explanation ;-)

                                                    2. 7

                                                      It’s not one line of code, it’s which line of code in the millions of them.

                                                    1. 3

                                                      Decentralised technologies like RSS never become mainstream because they don’t get any support from any major company. Google would prefer users to find what they need on google.com; Facebook and Twitter have feeds they control, Microsoft has Bing, Linkedin, etc. All those are services they fully control.

                                                      But RSS they wouldn’t be able to control or track, it’s completely decentralised and as such there’s nothing in it for them, and no point supporting it.

                                                      1. 1

                                                        For several years Google offered what was then generally considered the best RSS reader before they eventually shut it down.

                                                      1. 2

                                                        Long before QQ, ICQ also assigned numeric user IDs to people & we had to remember those to message people!

                                                        1. 17

                                                          That’s all email is to me. They’re mostly unimportant messages that I receive, deal with, and move on.

                                                          I’m 38, and I’m feeling 83 right now. An old fossil that remembers that distant time in my youth, where email was the main, sometimes the only way people could contact me online. To this day, I cannot afford to not check my emails. Many are important. So I tend to assume, by default, that people treat their inbox the same way. Or at least that they should.

                                                          10-15 years ago, a friend of mine asked me to get a Facebook account, so we could keep tabs on each other. I subscribed, tried it, then quickly shut it down: that thing was clearly a glorified second inbox, and I already had one. Why would I have two inboxes to check, when I can have only one? And I certainly don’t want 10 notifications a day about distant friends petting their cat.

                                                          Still, Facebook has taken over. Or Twitter. Or Instagram. Or whatever. A couple years ago, I was part of an orchestra, and they set up a Facebook group (even forced me to join), because “nobody reads their email”. Like, Facebook, whose notification items fall from your wall faster than you can check them, where messages are often missed even if you’re dilligent, is more reliable than email in practice because people just don’t use their email to communicate.

                                                          Has email become little more than a gateway to subscription services now?

                                                          1. 2

                                                            I feel with you, tough I am several years younger. But I still experienced the pre-facebook Internet era.

                                                            Has email become little more than a gateway to subscription services now?

                                                            It depends on the persons involved. With enough care it is possible to get many people to write you e-mails; I have got nearly all of my friends to do that by now. Problems arise if people want group functionality – someone will always set up a Facebook group or a WhatsApp chatroom. Many people today are unaware of mailing lists as they should be used (many open-source projects still use this medium properly). When they hear “mailing list” they think it refers to advertisement mail. It is also problematic for non-technical people to set up a mailing list. Services like freelists.org exist, but they’re only available in English. Then, getting people to abstain from thread hijacking and do quoting right is a fight against windmills. It’d help tremendously if email clients stopped quoting the entire message on reply and just give a blank editing window. Why has this copy-on-reply ever been invented?

                                                            1. 4

                                                              Mailing list software hasn’t really kept up with the times. Ideally one should be able to create them in a few clicks - select a name/topic, add some addresses, and away you go. Each email can have a footer with links for the individual addressee to manage their “subscription”.

                                                              (This sort of subscription might already exist. The point is that most people can easily create a group (private or not) on Facebook. Doing the same with email is not as easy).

                                                              1. 2

                                                                I understand that your criticism is mostly targeting UX issues rather than the underlying bits and pieces. In that sense, I more or less agree. However, with regards to almost everything related to what’s happening underneath the hood, I would even go so far as calling email is the most fundamentally broken piece of critical infrastructure we have (… well, that I am more intimately familiar with anyway). And there is no reasonable escape. No open protocol will be able replace it. But it badly needs to be replaced.

                                                                In the late aughts, I was a college student, and our department begun transitioning away from mailing lists. I really liked the concept of mailing lists and loved using my mail clients for reading lists, but even then the experience to make lists was horrible. When I was a TA, managing the lists was also quite annoying. The web clients were also terrible.

                                                                (This sort of subscription might already exist. The point is that most people can easily create a group (private or not) on Facebook. Doing the same with email is not as easy).

                                                                Agree. I think there’s a lot of mileage to be had by the open source world by improving the experience of mailist lists or usenet sites.

                                                                1. 1

                                                                  What you describe is in fact possible on Google groups already. I don’t think that’s the issue though.

                                                              2. 2

                                                                Has email become little more than a gateway to subscription services now?

                                                                Yeah. I’m 26 and was interested in Hey, but even if it was $50/yr it wouldn’t be worth it to me. I really don’t receive many emails. If I do, they’re newsletters, receipts, or bank transfer confirmations. I still keep my email open all day but I wouldn’t blame someone my age if they didn’t check it.

                                                                You’re correct that people don’t want a second or third inbox. The problem for you is that email is no longer anyone’s first inbox.

                                                                1. 2

                                                                  Amazing how different lives a few people can lead. I’m in a similar age bracket and read about 300 emails on a quiet weekday (down to just a few dozen on weekends). It’s mostly not social, that stuff I keep in person as much as I can.

                                                                  And Facebook sucks as an inbox. Try keeping more than a few dozen notifications; they silently get dropped so you never have too much clamoring for your attention.

                                                                  1. 1

                                                                    The problem for you is that email is no longer anyone’s first inbox.

                                                                    What is then? It would seem there are quite a few popular “first inbox”, which would basically force me to have several outboxes (I can manage), and several inboxes so I can see the replies. Pity, really.

                                                                1. 1

                                                                  “I accepted the number of homogeneous blocks to be 5 on the average.”

                                                                  Fascinating examples, although that assumption seems arbitrary.

                                                                  1. 3

                                                                    The bit about Google AMP pages address confusing people is spot on. I’ve just had to tell my wife about it (she’s starting a blog/site on parenting and Google told her there are AMP errors - panic mode on!!).

                                                                    What happened to the effort out allowing AMP on own domains?

                                                                    1. 5

                                                                      What happened to the effort out allowing AMP on own domains?

                                                                      A PM for a new chat app jingled their keys and now AMP is dead.

                                                                      1. 2

                                                                        In the first year, we shipped mid 8 figures per year worth of cost savings as a result.

                                                                        One wonders if any of this made it back in the form of bonuses.

                                                                        1. 2

                                                                          Or raises or promotions or, if the employee doesn’t feel sufficiently appreciated, an easier time interviewing for their next position.

                                                                          1. 1

                                                                            It’s highly likely that the author was well rewarded for this, yes.