1. 2

    This is great, I had picked some of these tricks up from stack overflow answers but I appreciate the comprehensive post!

    One question: does anyone else ever do a “transplant” of old bash history into a fresh machine? I find it comforting to still be able to reverse search over old commands.

    1. 1

      I’m wondering the same thing. I’ve thought about uploading to s3 after every command or session too….

    1. 13

      One of the easiest ways to get more out of our database is by introducing a new component to the system: the cache layer.

      In my experience, caching should only be used to reduce latency NOT to reduce database load. If your database can’t operate without an external caching layer (redis/memcached) you’re in big trouble when your cache hit ratio drops (restarting instances, cache key version update, etc.) It’s very easy to fall into this trap, and requires heavy architecture changes to get out of it.

      1. 1

        this is a really interesting (and I’m guessing hard-won) perspective.

        I’ve never been bitten by this but also never hosted something where a bit of jank during warm-up was a problem

      1. 1

        Very cool. Suggestion: I think I’d appreciate a time speed/scale slider. With all the organisms moving around so frenetically, it can sometimes feel stressful that one is not doing the right thing fast enough. :) Also, playing at a much slower speed could be enjoyable, too. Something where one day-night cycle is, say, a whole hour (wall clock time), or maybe several hours. So you could leave the browser tab, go live life, and come back and check on your little biome.

        1. 2

          this is a good idea, thanks for mentioning it.

          One way I might approach it is slow the game to half or 1/3 speed when it’s inactive (in the faded out UI state)

        1. 3

          This is great stuff. Reminds me of some “artificial life” software toys that were briefly popular in the 90s. (I remember a bunch of little web simulations of various sorts, some using the technique of the endlessly loading gif, and some DOS shareware(?) I ran for a while that had a bug of little bugs running around and propagating different traits.)

          1. 3

            Darwin pond was my favorite one!

            I had never heard of someone using the endlessly loading gif trick for something cool like that, would love to see if any of these have been preserved:)

            1. 1

              One of these days I’ll have to go spelunking on archive.org and see if I can turn anything up. I have impressions of the appearance of some of them, but sadly don’t remember any names or URLs.

          1. 26

            I recently did the “Glass Jar of Lake Water” experiment, which is this but in real actuality .. just take a clean glass jar, go to the nearest body of fresh, still water, gloop up a blob of algae and plants and water, seal the jar - then watch it grow for a few weeks.

            It has been a wonderful daily treat to observe life adjusting to the new environment - I have watched fleets of Hydra form on the glass surface, hundreds of daphnia and other water-fleas, and larvae galore. The hydra have stabilised after a few weeks and grown quite big - and the rowboat beetle has been making its home of oxygen bubbles in a garden of algae that it tends to, daily.

            I don’t give it too much direct sunlight - this raises the temperature of the water and can kill everything - but rather keep it in light, and give it an hour or so of real sunlight if things get too murky - some sort of balance has been attained, however, and now after 5 weeks, things seem to have gotten into a bit of equilibrium - every few days there will be more water fleas/daphnia, and then the hydra will grow, and a day or so later the water beetle will swim around knocking everything around.

            Its a real treat to just look at a bit of pond water and see what is growing within. I encourage anyone with an interest in complex systems to try this experiment and see for yourself how nature will find its own equilibrium if you give it the right inputs …

            1. 5

              Sounds like a nice experiment with my kids. Unfortunately, i don’t know anything about daphnia and stuff. Any suggestions how to learn that?

              1. 6

                there are a few good instagram accounts and youtube channels who educate in a fun way. one i love is “life in jars”

                1. 4

                  As maxbittker mentioned, the “Life In Jars” channel on youtube is pretty neat - I learned my version of this experiment before the advent of the Internet though ..

                  https://www.youtube.com/channel/UC0XNssyypOLiq4vVgXm9NtQ

                  Basically you take a clean jar, with no residues or anything, but nice and shiny and clean - and you swoop up a small amount of water and algae - a rough guide for how much algae is around 1/5th of the volume of the jar should have a bit of algae in it - don’t worry if you don’t get this right, just don’t fill the entire jar, for example.

                  Then, seal the jar and put it in an environment where the temperature will be stable, but not dark - and also not in direct sunlight. If you put it in direct sunlight, you will literally cook the contents - the only time you expose it to sunlight is if the algae doesn’t look like its getting enough sun, in which case you give it 30-minute bursts in the sunshine, then back to shade/cover - in order to put more energy into the jar.

                  As far as identifying daphnia and other critters - check Youtube as mentioned, and also this is pretty handy:

                  https://www.greatscopes.com/activity.htm

                  EDIT: Posted elsewhere, but including here for the fun:

                  Early days of the experiment:

                  https://imgur.com/gallery/tjy1nLw

                  6 weeks later:

                  https://imgur.com/gallery/PX3Vfxm

                  1. 1

                    After looking around a little, I got these links for further investigations. Now i have a few keywords like “ecosphere” to search for.

                  2. 2

                    This sounds like great fun! Can you show us a picture?

                    1. 7

                      Here you go, early days of the experiment:

                      https://imgur.com/gallery/tjy1nLw

                      6 weeks later:

                      https://imgur.com/gallery/PX3Vfxm

                      Definitely a fun thing to check on every day .. Let me know if you try the same experiment, its always fascinating to see what develops.

                      1. 2

                        Thanks for uploading those. Really interesting.

                        That hydra is an odd looking thing!

                        1. 4

                          Hydra are amazing creatures .. they can reassemble themselves if you blend them. ;) Their mouths seal shut after every meal and then tear open for food when its caught. They poop from their mouths! They have Chloroplasts living in their cells which give them sugar from photosynthesis when there isn’t much else to eat. They can reproduce by budding new versions of themselves, anywhere on their own bodies. A single hydra cell can grow a complete new hydra, and they never grow old - but rather are pretty resilient, invincible little life forms. They are very much one of my favourite creatures alongside octopus and tardigrades ..

                          1. 1

                            Having read the Wikipedia page for Hydra, can confirm amazingly odd creatures!

                            1. 1

                              Truly inspiring! I love the fact that they can re-assemble if you scramble their cells ..

                  1. 1

                    I use manjaro because it can come pre-installed with a nice i3 setup, and because the AUR usually has what I need, close at hand.

                    1. 2

                      I’m working on a livecoding environment for building face filters (in a way that’s not owned by facebook)

                      feeling very very depressed and anxious, coding is a nice escape.

                      1. 11

                        On the youtube section, I would add self hosting the videos. The <video> tag exists, you can treat videos the same as images and keep them on your server.

                        1. 3

                          youtube has a variety of performance improvements over a stock video tag, like choosing an appropriate bitrate, generating a thumbnail, and other things.

                          Do you know what i would read to learn what all these differences are and which ones can be implemented easily on a self hosted server?

                          1. 3

                            Thanks for sharing. Self hosting video is really difficult for most people. Many of the normal hosting providers that people use for their sites don’t even allow videos (or have severe restrictions). This is mostly why I didn’t mention it. I really like PeerTube as a better solution for people who don’t want to upload to YouTube.

                            1. 4

                              This is really a though one. For one, <video> tag might be too little, and having a peertube instance might be too much.

                              But I do think that e.g. video.js and a local file (or S3, Storage or similar) should be enough for a lot of the use cases. If you need a more complex solution, PeerTube might still be too much, but so is youtube or vimeo - so I think you don’t lose anything with self-hosting or having a local peertube. Things not covered by those two cases are probably quite specific and would benefit from custom solution anyway.

                              However, that’s what I think, and I’ve been known to hold opinions that simply suck.

                            2. 2

                              It can be done but posting videos from third parties on your webpage will surely get more copyright requests than just images. Also, you lose some features related to multi codec support and adaptative bitrate which require more advanced players (can also be self-hosted, but not just a plain tag)

                            1. 1

                              Can the expr editor perform expression evaluation in the browser?

                              1. 1

                                Currently no, bit now with rewriting parser it’s possible to do, as code can be compiled to js.

                              1. 4

                                I really appreciate i3 and it’s ecosystem of helper-tools, variants, and guides. I hope that I don’t need to learn a new window manager this decade :D

                                1. 2

                                  great post! thanks for writing this up. People really identified a ton of topics that are useful and interesting.

                                  Unfortunately, I think the resulting curriculum has metastasized into about 4 semesters worth of material!

                                  Does anyone have opinions on what they would chose from this menu if they were designing a course bounded by time and space?

                                  1. 3

                                    I’d teach regular and context-free languages, introduce the classes of languages that are recognised by recursive descent parsers, and teach regular expressions and recursive descent parsing. I’d highlight the importance of error recovery and error messages in parsers and emphasise that despite all the theory, most production compilers are recursive descent because it makes error handling much easier.

                                    I’d avoid the single pass compiler meme, because while they’re simple, in practice intermediate representations are important. I’d teach in a way that made sure students had assignments where they implement every stage of a compiler: scanning manually and with regular expressions (so they can see how much easier regex makes scanning), parsing with parser generators and with recursive descent (so they can see how much easier recursive descent makes parsing), desugaring and type checking on an abstract syntax tree, optimisations on an AST, SSA generation from an AST, optimisations on an SSA, code generation from an SSA. Probably in reverse order from that for reasons I explained in another comment.

                                    You can cover a lot more advanced material in lectures than you can expect students to implement in their assignments. I think you can cover things like Given timetable freedom, I would probably structure the course so that there would be two lectures per week of material early in the week, then a couple of days for them to do the assignment based on that material (they could of course start it earlier), then a “lecture” after the assignment is due where I would go over the standard implementation of the feature they were asked to do in the assignment, how it could be improved with different algorithm, etc. that’s ‘interactive’ in the sense that I would take a lot of questions, would actually write code in front of students, etc. Those ‘lectures’ where the lecturer wrote code live in the lecture theatre were some of the most effective lectures for me while I was at university, because you learn a lot about how to program by seeing a good programmer programming.

                                    So the basic course would be one where you go in a straight line: at the beginning you have nothing, each week you have one more stage of the process. Code generation, SSA Optimisations, SSA generation, AST optimisations, AST type checking, AST desugaring, Parser generators, Recursive descent parsing, Scanning + 1 more week of something else could be the roughly 10 weeks of teaching you get in a semester when you take into account public holidays, introductory lectures, summary lectures, etc. I’d do a week somewhere in there about error recovery or something, or an extra week on type checking maybe?

                                    An advanced course would be a series of relatively independent digressions at each step, I would suggest. You can do a couple of weeks on garbage collection, a couple of weeks on control flow analysis, a week on indentation-sensitive syntax, a couple of weeks on macros/staging. You could do an entire semester on JIT compilation if you cover how it interacts with monomorphisation, method vs tracing JIT, garbage collection, etc. You obviously could do years worth of courses on type system stuff, that’s an infinite rabbit hole.

                                  1. 1

                                    maxbittker.com

                                    maximalist & net-art inspired, but i tried to keep it usable

                                    1. 9

                                      on the broad view, this is cool and i’m glad to have informed people in congress.

                                      on the other hand, this guy is behaving one of those sorry people who ask super-technical details after a talk in order to show how smart they are, and l don’t think he added anything to the discussion at all.

                                      1. 22

                                        He added one bloody important point: they are using Rust nightly. This is a fairly unstable dependency, and more importantly, this is a dependency that accepts outside contributions in a way that may not be as controlled as the core Libra codebase.

                                        It would be a freaking security risk if they pushed that to production. They are using Rust nightly now, but they probably should move to stable before they go to production, or at least freeze to a particular commit until the Libra association actually reviews the newer commits.

                                        I was actually disappointed that the Libra guy didn’t have an answer.

                                        1. 8

                                          I strongly disagree that nightly rust in and of itself is a security risk. Using a conservative set of #![feature] flags and pinned version of nightly I think it’s honestly more stable than many languages.

                                          All stable is is a mutually agreed upon pinned version of nightly with no feature flags.

                                          1. 5

                                            I strongly disagree that nightly rust in and of itself is a security risk.

                                            Understand where the congressman is coming from: Libra is (will be) a currency. Depending on adoption, it can get quite strategic, on par with weapons and the electric grid. The possibility of malicious contributions to the code base itself or its dependencies should indeed be investigated. Such malicious contribution have happened before: remember that NPM package that was stealing wallet private keys from the projects that uses it? (Or maybe it was mining?) It went unnoticed for months.

                                            Rust is one such dependency. Rust nightly is the least reviewed part of Rust (besides experimental branches). Of course this makes some people nervous. Even if nightly Rust is as you say not a security risk, how it is not a security risk should be explained to those nervous people.

                                            1. 1
                                              • “No feature flags” is a big deal, since un-flagged operations are supposed to be frozen.

                                              • Stable also goes through the Beta period, when new features aren’t allowed in but regression fixes are.

                                              1. 1

                                                Not quite. Bug fix point releases are not uncommon.

                                              2. 4

                                                He was also concerned that one of the top committers for one of the Libra projects was from Nigeria. I think we assume good intent and don’t discriminate based on where one lives. But since he’s coming from a “national security” background, I can understand why he might be more suspicious (justified or not) of “foreign” contributors.

                                              3. 2

                                                I think its good to remember that Congresspeople’s offices do followup for written details of what couldn’t be covered in these televised hearings. I hope his office does followup and gets the answers he seeks.

                                              1. 2

                                                if this process was repeated a second time, would it be largely idempotent?

                                                1. 3

                                                  In this instance I believe it would. The Relative colorimetric rendering intent only alters colors that are out of gamut for CMYK, snapping them to the closest printable color, all the while keeping printable colors largely unchanged. Conversely, the Perceptual rendering intent will “shrink” the sRGB gamut to fit in the CMYK gamut, so all colors are shifted towards less saturated versions, and repeating the process would cause them to converge to… grey?

                                                  (Based on my limited understanding of the process, and without having tested my assumptions)

                                                  1. 1

                                                    In theory it should be idempotent. In practice, the roundtrip is going to induce some errors. Doing it repeatedly is a way to gain more insight into the structure of those errors. Could be a fun project to explore!

                                                    1. 1

                                                      In general exploring repeated function application for fixed points is nearly always a good idea.

                                                  2. 1

                                                    I would think it would monotonically decrease in saturation and contrast but who knows.

                                                  1. 6

                                                    Last week, I built tools for reporting & moderating sandspiel.club posts. Being responsible for keeping up with the reported posts has not been fun or very sustainable, so I’m not sure what the next step is- probably biting the bullet and forcing people to register accounts in order to post.

                                                    …feeling very acutely the difficulties of solving social problems with code

                                                    1. 4

                                                      Compared to react native, I think that it has a few strengths that people underestimate - It’s (from a developer’s perspective) significantly simpler than many other UI stacks.

                                                      The layout model and widget model are quick to learn and wrap your head around - especially compared to web- everything is tidy and built from a small set small of primitives.

                                                      Dart isn’t a stellar language but you can learn it in one day like you can learn Go.

                                                      The #1 weakness is ecosystem. Flutter requires a good deal of elbow grease to re-build common patterns because the libraries aren’t fully there.

                                                      Source: I write flutter at work

                                                      1. 1

                                                        Are you able to share where you work? Or what kind of apps you write? I’ve been doing some Flutter professionally as well but haven’t seen it out in the wild much yet.

                                                      1. 3

                                                        I assumed that Sandspiel sprang fully-formed from your first attempt. I like it even more knowing how you arrived at the current version. The idea of user-defined elements is really interesting. Are you still pursuing this?

                                                        1. 2

                                                          only passively - I don’t yet know what wasm toolchain i would use that i can compile & link in the browser.

                                                          another problem is interactions between elements built by different people- how do you mark your alien wood as flammable?

                                                          1. 1

                                                            I was thinking that the spirit of your idea could be followed by creating a “compounding” mechanism within the game. This would enable a user to create new elements by combining existing elements. The compound is some union of the behaviors of the combined elements. Or you could define a set of even more elemental behaviors that are combined to create a new element. This is more constrained than the original idea, but it eliminates the need to support user-defined code and avoids impedance mismatches between user-defined elements.

                                                            1. 1

                                                              I hadn’t considered this - I think that some sort of constrained declarative “production rule” language might be another option - I’ve seen those used effectively in similar simulations, like in dave ackley’s research

                                                              1. 1

                                                                Sounds interesting! I’ll check it out.

                                                        1. 10

                                                          I really like your blog and it saved me from thinking that you’d just landed on this really top of the line sand game on your first try. Thanks for humanizing your amazing work.

                                                          1. 3

                                                            I’m really glad that came across! To be honest, after writing this, I felt that I had been dishonest in leaving out many negative aspects of the process because those are more difficult to write about. It’s an easy trap to fall into.

                                                            Two things I didn’t write about:

                                                            I was kind of fried when I finished, because I didn’t make time to take breaks and formed a false narrative in my mind that I had to finish the game as fast as possible or I would never release it. I didn’t feel very healthy or happy for the month or so afterward (doesn’t help that it was the middle of the winter and I wasn’t getting any sunlight)

                                                            The upload board community has a lot of spam, bullying, and nastiness. I’ve tried to do things to make it nicer through the design (no tracking of points, no comments), but solving a social problem with a UI design is not working well. This bums me out but I’m not sure what to try next. I wish I had made the element set a little bit more nature oriented and less pyrotechnic - no idea if that would have helped.

                                                          1. 7

                                                            I played with this for almost 90 minutes last night. It’s addictive.

                                                            1. 3

                                                              I’ve heard someone categorize the two main pleasures of a video game as creating a mess and cleaning one up - I think that falling sand games capture both of these pleasures : )

                                                            1. 1

                                                              I think it might be the anti-aliasing and tasteful lighting, but these renders are so much cleaner and smoother than the ones I’ve built. (and the log-spherical mapping is really interesting!) Thanks for sharing!