1. 46

    I was wondering whether there was a significant performance improvement of the table lookup glibc does over the naive implementation musl has.

    The answer is a pretty big yes. Glibc’s isalnum is about 6-7 times faster than musl’s.

    Benchmark code: https://paste.sr.ht/~minus/18b44cfe58789bc1fb69494130e859a1189d1772

    1. 28

      Yep, glibc didn’t add all of that weirdness for no reason…people tend to forget that.

      1. 1

        Wouldn’t a big boy switch() statement be faster then ? Something like

        switch(c) {
        case 'a':
        case 'b':
        case 'c':
        ...
        case 'Z':
            return 1;
        }
        return 0;
        

        Is it as fast as a lookup table ?

        1. 5

          It might be, if the compiler was smart enough to compile it into a lookup table.

          Think about how you would generate code for a switch: the naive solution is to make a bunch of conditional branches. If the values are contiguous (say, the 26 English letters plus case variants), you output some early-rejection code to handle values outside that range and then do conditional branches again or use a LUT.

          Maybe the cost of memory access for non-cached LUTs is high enough that a bunch of conditional branches is faster, maybe not. But, relying on the compiler switch codegen is iffy.

          1. 3

            Generating code for switches is a well understood area and the code generation can be pretty good.

            Here’s what an is_digit equivalent compiles into:

                add     eax, -48  
                xor     edx, edx  
                cmp     eax, 10  
            

            This is a standard trick for optimizing range checks.

                // The usual range check with two compares.  
                low_bound <= value <= high_bound  
                
                // An equivalent range check with a subtract and an unsigned compare.  
                value - low_bound u<= high_bound - low_bound  
            

            If you like this sort of bit-twiddling then I would recommend the book Hacker’s Delight by Henry Warren.

            Here’s a partial list of some other techniques that are used.

            • Lookup table
            • Profile driven layout of cases, e.g. testing hot case(s) first
            • Binary search
            • Perfect hashing
            • Tries
            • Combining multiple techniques, e.g. range checks for contiguous tests and binary search for spare tests.
          2. 1

            Wouldn’t a big boy switch() statement be faster then ?

            I wouldn’t assume either option is faster without profiling it first - a switch case in my response to the child post was optimized into 3 instructions.

            The benchmark in the grand parent’s post is pretty much the best possible case for a lookup table. Programs that call ctype functions rarely instead of often may get different results; there could be cache misses.

            This isn’t an issue with inline code such as @zx2c4’s ctype.h functions.

            The answer, as with everything in performance, is to profile your use case and come to a conclusion.

      1. 1

        I’m using a Yubikey in a similar manner to unlock my desktop (also sway!).

        I use udev to notify a daemon running in my user session. This daemon then asks the Yubikey for a challenge-response. So essentially the same minus PAM and not using it to lock.

        I was actually hoping to see NFC being used here, because plugging the thing in is kind of annoying.

        1. 4

          If you force “smooth scrolling” on your users you’re a monster

          1. 1

            On one hand, I kind of agree.

            On the other hand, it seems like for a good proportion of users, smooth-scrolling, at least from the “back to top” button, is less confusing?

          1. 4

            Notably from the changelog:

            It is now possible[1] to perform chosen-prefix attacks against the SHA-1 algorithm for less than USD$50K. For this reason, we will be disabling the “ssh-rsa” public key signature algorithm by default in a near-future release.

            1. 5

              Also noteworthy:

              A future release of OpenSSH will enable UpdateHostKeys by default to allow the client to automatically migrate to better algorithms.

              And FIDO/U2F support for keys

            1. 3

              If you’re putting binary files into git you’re doing it wrong. One could argue about small files, but compiled code/executables, photos or “gifs for the readme” are definitely misplaced in a git repository.

              1. 12

                I do find that having image files in a resources/ directory for something like a website is often simpler than separating the two. Even then making sure that images are compressed and generally not bloating repo size / git history is essential.

                1. 18

                  I do find that having image files in a resources/ directory for something like a website is often simpler than separating the two.

                  Yeah, the is exactly the use case here. Mercurial (and git) aren’t designed for handling large binary files, but if you’re checking in static assets/resources that rarely change it still tends to work fine. This repo was fine on Bitbucket for many years, and is working fine on an hgweb instance I’ve spun up in the mean time.

                  I specifically asked about limits because if it’s just the size of the repo being a technical problem for their infrastructure, I can understand. But they would not specify any limits, but just reiterated several times that Mercurial wasn’t designed for this. So I don’t know which of these was the actual problem:

                  1. The repo is so taxing on their infrastructure it’s causing issues for other users.
                  2. The repo is so large it’s costing more to store than some portion of the $100/year account price can cover.
                  3. They are morally opposed to me using Mercurial in a way that it wasn’t designed for (but which still works fine in practice).

                  Cases 1 and 2 are understandable. Setting some kind of limit would prevent those problems (you can still choose to “look the other way” for certain repos, or if it’s only code that’s being stored). Case 3 is something no limit would solve.

                  1. 3

                    If you want to store large files and you want to pay an amount proportional to the file sizes, perhaps AWS S3 or Backblaze B2 would be more appropriate than a code hosting website? I don’t mean to be obtuse, but the site is literally called source hut. Playing rules lawyer on it read like saying “Am I under arrest? So I’m free to go? Am I under arrest? So I’m free to go?” to a police officer.

                    1. 5

                      B2 or S3 would make things more complicated than necessary for this simple repo. I’ve spun up a $5/month Linode to run hgweb and it’s been working great. I’m all set.

                2. 6

                  This case was hg, but the same limitations are present. Hg has a special extension for supporting this:

                  https://www.mercurial-scm.org/wiki/LargefilesExtension

                  And it’s considered “a feature of last resort”. It’s not designed to deal with these use-cases.

                  LFS support requires dedicated engineering and operations efforts, which SourceHut has planned, but is not ready yet.

                  1. 5

                    I have a repository with mostly PNG files. Each PNG file is source code; a chunk of data inside each PNG file is machine-readable code for the graph visually encoded in that PNG’s pixels. What would you have me do?

                    I suspect that you would rather see my repository as a tree of text files. While this would be just as machine-readable, it would be less person-readable, and a motivating goal for this project is to have source files be visually readable in the way that they currently are, if not more so.

                    git would not support binary files if its authors did not think that binary-file support were not useful; that is the kind of people that they are and the kind of attitude that they have towards software design.

                    With all that said, I know how git works, and I deliberately attempt to avoid checking in PNGs which I think that I will have to change in a later revision. It would be quite nice if git were able to bridge this gap itself, and allow me to check in plaintext files which are automatically presented as PNGs, but this is not what git was designed to do, and we all can imagine the Makefile which I’d end up writing instead.

                    1. 1

                      I like the project, but pardon my ignorance - aren’t the PNG files still binary assets produced by the “real” source code, which is the textual expression parsed to generate both the embedded bitstring and the dot graph? If they’re machine readable, that places them in the same category as compiled object files.

                      1. 3

                        The real source code is non-textual; it is the diagram (WP, nLab) which is being given as a poset (WP, nLab). To achieve optimal space usage, each poset is stored as a single integer which codes for the adjacency matrix. However, this compressed format is completely unreadable. There are several layers around it, but each layer is meant to do one thing and add a minimum of overhead; JSON (in the future, BSON or Capn) for versioning and framing, and PNG for display and transport. There isn’t really source code; there’s just a couple Python and Monte scripts that I use to do data entry, and I want them eventually automated away in favor of API-driven development.

                        For example, the raw integer for this “big” poset is (at the time of writing) 11905710401280198804461645206862582864032733280538002552643783587742343463875542982826632679979531781130345962690055869140174557805164079451664493830119908249546448900393600362536375098236826502527472287219502587641866446344027189639396008435614121342172595257280100349850262710460607552082781379116891641029966906257269941782203148347435446319452110650150437819888183568953801710556668517927269819049826069754639635218001519121790080070299124681381391073905663214918834228377170513865681335718039072014942925734763447177695704726505508232677565207907808847361088533519190628768503935101450078436440078883570667613621377399190615990138641789867825632738232993306524474475686731263045976640892172841112492236837826524936991273493174493252277794719194724624788800854540425157965678492179958293592443502481921718293759598648627823849117026007852748145536301969541329010559576556167345793274146464743707377623052614506411610303673538441500857028082327094252838525283361694107747501060452083296779071329108952096981932329154808658134461352836962965680782547027111676034212381463001532108035024267617377788040931430694669554305150416269935699250945296649497910288856160812977577782420875349655110824367467382338222637344309284881261936350479660159974669827300003335652340304220699450056411068025062209368014080962770221004626200169073615123558458480350116668115018680372480286949148129488817476018620025866304409104277550106790930739825843129557280931640581742580657243659197320774352481739310337300453334832766294683618032459315377206656069384474626488794123815830298230349250261308484422476802951799392281959397902761456273759806713157666108792675886634397141328888098305747354465103699243937608547404520480305831393405718705181942963222123463560268031790155109126115213866048693391516959219000560878337219324622230146226960346469769371525338127604307953786112516810509019551617885907067412613823285538493443834790453576561810785102306389953804151473860800342221969666874213156376831068606096772785272984102609049257833898258081466729520326827598704376424140779421965233471588921765110820238036094910936640446304632443760482611408445010230964335747094869968021425396439555206085281953007985784739643408074475440039274314217788647485602069097474262381690379456154426900896918268563062231294937080146199930562645748389040251871291840481739518244706752426504146889097315360662429293711705265772337748378759001582638301784557163848933046038798381667545043026975297902178839764134784634179453671000024868722179355800776002690855305662785522771116635997791339179517016284742206819482196944663461005128697584753594559406283638837841370287286682993990297923202976404261911087739188860505577427942276773287168600954693735964671046522557013031834557159173262849132567983767216098382093390056878765856939614383049277441.

                        1. 1

                          Ah, okay, I see. Makes sense, thank you for explaining!

                    2. 4

                      I’ve seen this argument quite a number of times, and almost always without a coherent explanation of why is that wrong. What’s the rationale behind this argument?

                      1. 4

                        Shameless plug, I contributed heavily to this help topic back when I was the PM for Microsoft’s Git server: https://docs.microsoft.com/en-us/azure/devops/repos/git/manage-large-files?view=azure-devops

                        FWIW I disagree with the comment up-thread which says that GIFs for READMEs don’t belong. If you’re going to check in a logo or meme or whatever, that’s perfectly fine. Just don’t do 1000 of them and don’t churn it every week.

                        1. 2

                          I think a big part is also “are my tools there for me or am I slave to my tools?”

                          If I have a website and most content is under version control, it’s annoying and complicated to have (big) assets outside. Most people simply want one repo with everything inside, and it’s mostly additive, often once per week - it simply doesn’t matter if it’s the wrong tool.

                    1. 7

                      Seeing how the sendy author fails to understand and how basic* anti spam mechanisms work, even after it’s explained, I would move away from it immediately. It’s not even rate limiting per IP. Considering they’re asking $59 for it, that can’t be simply put off as “yeah, but it’s someone’s toy project”

                      1. 3

                        I wonder how many of the expired certs are due to Let’s Encrypt cron jobs not doing what they should

                        1. 4

                          I’m gonna hook up my newly acquired Behringer X-Touch Mini MIDI controller to PulseAudio for per-application volume control (and maybe other things). I don’t really have a clue about MIDI yet, so that’s gonna be fun.

                          1. 1

                            Those are pretty versatile devices. Some people use them with Lightroom/Photoshop.

                          1. 1

                            I wonder if they still run Kyoto Tycoon as backend DB and they use it for that

                            1. 1

                              If numeric type in PostgreSQL can go up to 16383 digits after the decimal point, how many comments can I store this way?

                              1. 4

                                If we’re taking Postgres, might as well have parents INTEGER[] and save yourself a lot of messing about.

                                1. 2

                                  Or just use a recursive CTE

                              1. 4

                                I was expecting to see the kernel being bypassed and directory structures being read directly from disk in bulk (obviously with limited fs support)

                                1. 4

                                  Ha-ha :-) Maybe next time. The main bottleneck in gitstatusd is fstatat, that’s why directory listing optimizations are in a separate document. If I could optimize that, it would yield serious gains in performance.

                                  Edit: By the way, the improvements in the article look pretty small between different versions because CPU time is dominated by system calls that are out of my control. If you look at userspace CPU only, the speedup between v1 and v5 is 15x. In other words, the final version uses 15 times less CPU time in userspace. It also uses less CPU in the kernel, but the improvement there isn’t as dramatic (just the openat vs open optimization).

                                1. 9

                                  Oh, I’ve read this blog post before. Everyone dealing with timestamps/dates in Postgres should read it. Or at least the tl;dr: Always use TIMESTAMP WITH TIME ZONE (or short timestamptz) unless you have a very good reason not to.

                                  1. 4

                                    Login-walled. Cannot access without a google account.

                                    EDIT: Scratch that, lobster’s provides a ‘cached’ link below the article link. Works ;)

                                    1. 5

                                      That happens if you’ve ever logged into Google before; clear your cookies

                                      1. 1

                                        This works, thankyou :) I had a work account in there. Not making that mistake again.

                                      2. 4

                                        Weird, I can access it without being logged in. I just tried in a not-logged-in-to-Google private browsing session and it worked fine (and it has the “Sign in” link at the bottom, where it usually displays the email address of my signed-in Google account).

                                        1. 1

                                          what browser

                                          1. 1

                                            I’m using Firefox 62.

                                        2. 1

                                          It’s just a JS-wall, not login-wall.

                                          1. 1

                                            Not a JS wall.

                                            If I go there without javascript then I get a page without CSS containing a hyperlink to https://accounts.google.com/AccountChooser?continue=https://groups.google.com/forum/m/&amp;hl=el&amp;service=groups2

                                            The page source is otherwise devoid of any groups/message content.

                                            EDIT: This is the same both before and after clearing cookies. If I then enable JS (and therefore go through a redirect) I end up either at the login page or the actual groups page, depending on my cookies (as per posts above).

                                        1. 3

                                          I may be alone with this, but I prefer gevent over asyncio, except for the part where gevent is a giant hack patching the standard lib.

                                          1. 2

                                            With sysctl paragraph you meant to say you do not want packets on the bridge to be piped through itpables, etc, right?

                                            I was very surprised to see that Linux throw frames on bridges into iptables; I was assuming that a bridge is just a layer 2 switch

                                            1. 1

                                              Notebooks make it easy to retry parts of the code you’re writing, which is very helpful if you have a bunch of steps that each take some time to compute. You can pretty easily write helpers to dump the state, and maybe pick up where you left off, but that still requires you to load the data again and define an appropriate entry point. Is there a good solution for that outside of notebooks?

                                              1. 6

                                                Repls?

                                                1. 1

                                                  Those don’t allow you to edit & rerun blocks of code, so that makes them less useful

                                              1. 2

                                                We built something very similar last year. It’s a bit simpler though: Raspberry Pi near the door, two wires to the door and a piece of aluminium foil on the door. The Pi runs a webserver that makes the current status available as JSON document and via WebSockets. We just embed that as iframe into our dashboard in the hallway (and in my status bar on my PC) and now have real-time information. It’s been quite useful so far.

                                                1. 4

                                                  I was expecting Python-AST wizardry and got disappointed. But not as much as by the fact that this is Python-2-style code. Otherwise it’s a thought-inspiring article that boils down to a “use a syntax tree built from nested dicts instead of writing your own DSL parser”.

                                                  Edit: technically correct is the best kind of correct

                                                  1. 2

                                                    What makes you say it’s Python 2?

                                                    1. 3

                                                      Usage of u"" and inheriting from object. Former was just added to Python 3.3+ for backwards compatibility; strs are unicode by default in Python 3 anyway. Making new-style classes doesn’t require inheriting from object in Python 3 (even in 2.7 I think) because old-style classes have been removed altogether.

                                                      Imo, code written today should not aim for Python 2 compatibility unless it’s something that has to work with a Python 2 code base that’s not migrated yet.

                                                      1. 3

                                                        Ok. I don’t think you should write “this is Python 2” when the code is (indeed) compatible with both Python 2 and 3.

                                                        1. 4

                                                          You’re right, technically it’s Python-3-compatible. It’s still Python-2-style code, which I think is bad because it encourages a soon-to-be-EOL’d Python version. Newly written code should just be Python-3-only.

                                                  1. 3

                                                    Nice that this is being pushed forward now. I’ve written a C implementation of (for now) v2.local with bindings to Java and C# (latter by a colleague). I’ve been wanting to publish it, but had my hands full. I hope I manage to get it out on the web in the next two weeks, hopefully including Python bindings!

                                                    1. 2

                                                      That sounds amazing :O

                                                      Can you please link them here when they’re released? https://github.com/paragonie/paseto/issues/45

                                                    1. 4

                                                      I was hoping for something combinding fzf and your shell’s directory history (e.g. ~/.zdirs), so you’d just have to hit a shortcut, enter relevant parts of the path you want and get there; without having to set up any aliases before.