1. 5

    In general, this was an interesting read but because I think it blames the technology a bit too much I’d like to point out, that:

    Before extracting Quiz Engine MySQL queries from our Rails service, we first needed to know where those queries were being made. As we discussed above this wasn’t obvious from reading the code.

    This is probably not what most people would do here as any kind of APM tool would clearly show you which query is executed where (among other things). For deeper investigations, there are things like rack-mini-profiler, etc.

    To find the MySQL queries themself, we built some tooling: we monkey-patched ActiveRecord to warn whenever an unknown read or write was made against one of the tables containing Quiz Engine data.

    Even if you don’t use APM, there is no need for any monkey-patching, you can simply subscribe to the feed with instrumentation API https://guides.rubyonrails.org/active_support_instrumentation.html#active-record

    Quiz Engine data from single-database MySQL to something horizontally scalable

    You don’t provide the numbers, so I can’t say anything to the scale you are dealing with, but in general, I wouldn’t want anyone reading this to think MySQL isn’t very scalable because it is horizontally and in other directions as well.

    1. 7

      For anyone else wondering, APM is an abbreviation for Application Performance Monitoring (or Management). It’s a generic term, not Ruby-specific.

      1. 6

        I think some of the “blame the tooling” comes from how starkly different it feels for us to use Haskell versus Ruby + Rails.

        With Rails, it’s particularly easy to get off the ground – batteries included! But as the codebase grows in complexity, it becomes harder and harder to feel confident that our changes all fit together correctly. Refactors are something we do very carefully. We need information from tests, APM, etc. Which means legacy code becomes increasingly hard to maintain.

        With Haskell, it’s particularly hard to get off the ground (we had to make so many decisions! which libraries should we use? How do we fit them together? How do we deploy them? Etc). But as our codebase has grown, it’s remained relatively straightforward and safe to do refactors, even in code where we have less familiarity. We have a high degree of confidence, before we deploy, that the code does the thing we want it to do. As the project grows in complexity, the APIs tend to be easier to massage into the direction we want, rather than avoiding improvements because of some kind of brittleness / fear of regressions / fighting with the test suite.

        For those that haven’t written a lot of code in statically-typed ml languages like elm, f#, or haskell, the experience of, “if it compiles it works” feels unreal. My experience with compiled languages before Elm was with C++ and Java, neither of whose compilers felt helpful. It’s been a learning experience adopting & embracing Elm, then Haskell.

        1. 2

          This is probably not what most people would do here as any kind of APM tool would clearly show you which query is executed where (among other things). For deeper investigations, there are things like rack-mini-profiler, etc.

          I agree this information can also be found while monitoring, and we did rely on our APM quite a bit through the work (though this is not mentioned in the blog post), for example to see whether certain code paths were dead.

          A benefit of the monkey patch approach I think, was that it was maybe easier to interact with programmatically. For example: We made our test suite fail if it ran a query against a Quiz Engine table, and send a warning to our bug tracker (Bugsnag) if such a query ran in staging and production (later we would throw in that case too).

          Didn’t know about the AR feed. That looks like it would have been a great alternative to the monkey-patch.

          Regardless, our criticism here isn’t really related to Rails tooling available to dig for information, rather that we would have liked not needing to dig so much to know where queries were happening, i.e. that being clearer from reading code.

          1. 2

            What APM tools did you use to give what info/data and what didn’t they provide that you needed to use other tools to fill the gap for?

            1. 1

              What APM tools did you use to give what info/data and what didn’t they provide that you needed to use other tools to fill the gap for?

              We primarily use NewRelic in our Rails codebase and Honeycomb in our haskell codebase.

              NewRelic is a huge product, and I bet we could have gotten more use from NQRL to find liveness / deadness, but we didn’t.

              We used NewRelic extensively to find dead code paths by instrumenting code paths we thought was dead and seeing if we saw any usage in production.

              For finding every last query, we wanted some clear documentation in the code of where queries were and where queries weren’t. NewRelic likely could have provided the “where” but our ruby tooling let us track progress of slicing out queries. The workflow looked like this:

              • Disable queries in a part of the site in dev (this would usually be at the root of an Action)
              • Ensure a test hits the part of the site with the disabled queries
              • Decorate all the allowed/known queries to get the test passing
              • Deploy, and see if we saw any queries being run in the disabled scope
                • if we do, write another test to ensure we hit the un-covered code path. Decorate some more.

              It looked something like this:

              SideEffects.deny do
                # queries here are denied
                 data = SideEffects.allow { Quiz.find(id) # this query is allowed }
        1. 1

          Anyone got a link to the running scoreboard of Intel vs AMD and others for mitigation compromises on performance due to CPU vulnerabilities?

          1. 8

            Really digging the new UI.

            1. 1

              I’ve been looking at the Asus PN50, comes with 4000 series AMD cpus (Ryzen7!) in a NUC-like format: https://www.asus.com/Displays-Desktops/Mini-PCs/All-series/Mini-PC-PN50/ One limitation is NVMe only up to 512gb, SSD to 1tb, but goes to 64gb ram.

              1. 1

                Is rsync still single threaded?

                1. 2

                  Yeah, just like scp.

                1. 1

                  I’d say a REPL is super useful…

                  1. 11

                    Agreed, though it definitely takes a bit of fiddling to set up to work well. It’s a bit of a project laptop. I’ve been meaning to write up my own experiences with it, but now that I have it working well I’m less motivated to do so, so I’ll summarize them here:

                    • The keyboard and robustness in general are way better than a $200 laptop deserves.
                    • The best windowing solution I’ve found for it so far is by far sway, faster and more responsive than even lightweight X11 WM’s like awesome.
                    • You can play Dosbox and emulated games up to the Playstation era just fine.
                    • Doesn’t handle it very gracefully if the battery runs out all the way – but then, no laptop really does.
                    • Mine has some audio issues but they might be hardware based, need to open it up and poke around.
                    1. 4

                      Sway brings up something I think is interesting. It’s made sense in the past if you wanted to do something lightweight under X to avoid variable-width fonts. After all, this is why i3, Sway’s inspiration, did just that. But with the likes of Sway, which is already using the likes of Freetype to render its fonts, using fixed-width fonts as a rather odd affectation.

                      1. 3

                        I didn’t know that, actually. As far as I care using Freetype or such is just fine, and don’t know why you’d want to avoid it – I rather like being able to see Unicode text without doing anything special. You can change the font that Sway uses quite easily, it’s a one-liner in the config file. Truetype fonts might have some cost in complexity and so on, but I’d rather have that than a X server clumsily trying to do its own font stuff.

                        1. 2

                          I think it’s just because Sway doesn’t want to stray too far away from i3 so that people with existing i3 configurations can get something very similar working with Sway without too much hassle. If you switch out the default Sway top bar for something like waybar, you’re already using your system default user interface font instead of Sway’s default monospace. If you hide the Sway window titles, Sway will hardly ever render any text in the first place.

                        2. 3

                          I’ve been happy with my kwin-tiling setup. I haven’t tried sway on it yet. It was definitely a bit of a project, but I got mine when it was still shipping with that funny debian install… now that it ships with manjaro plasma by default, it might be less of a project.

                          I’m seeing “all day” life out of the battery, and chromium is fine. It doesn’t behave if you run the battery to empty, but like you said, nothing really does.

                          1. 1

                            Yeah my battery experiences haven’t quite been all day, but it hits 5-6 hours pretty easily unless I’m watching videos or compiling code the whole time. I definitely expect it could be stretched out for longer if one tried. I should probably figure out how to turn the brightness down and see if I can hit the 12+ hours other people report, but so far I really haven’t needed to. I currently don’t even have a battery monitor widget in my config.

                          2. 1

                            How many Linux laptops don’t take at least a little bit of fiddling though? Would be keen to see your blog post on your experiences.