1. 2

    I think this article misses the rather important section which could be relevant to the title saying “small CLI programs”: how can you transform that whole Common Lisp codebase into single, statically linked, standalone executable binary which could work on remote hosts being copied over SSH without gazillions of uneeded dependencies?

    1. 7

      how can you transform that whole Common Lisp codebase into single, statically linked, standalone executable binary which could work on remote hosts being copied over SSH without gazillions of uneeded dependencies?

      That’s… pretty much what it’s doing?

      sjl at alephnull in ~/src/dotfiles/lisp on default!?
      ><((°> touch batchcolor.lisp
      
      sjl at alephnull in ~/src/dotfiles/lisp on default!?
      ><((°> make
      mkdir -p bin
      ./build-binary batchcolor.lisp
      mv batchcolor bin/
      mkdir -p man/man1
      ./build-manual batchcolor.lisp
      mv batchcolor.1 man/man1/
      
      sjl at alephnull in ~/src/dotfiles/lisp on default!?
      ><((°> scp bin/batchcolor vm:batchcolor
      batchcolor                                   100%   42MB  59.4MB/s   00:00
      
      sjl at alephnull in ~/src/dotfiles/lisp on default!?
      ><((°> ssh vm
      
      vagrant@vm:~$ echo 'test 1234 test 1234 test 5678' | ./batchcolor '[0-9]+'
      test 1234 test 1234 test 5678
      
      vagrant@vm:~$ ldd batchcolor
              linux-vdso.so.1 (0x00007ffe7f369000)
              libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff4822da000)
              libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff4822b9000)
              libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff482136000)
              libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff481f75000)
              /lib64/ld-linux-x86-64.so.2 (0x00007ff482542000)
      

      It does depend on a few C libraries – if you want to remove even those you’ll need something like this.

      1. 3

        This article is not missing anything, it is what it is. If you did not find information you were looking for, then asking a question would be a polite way to get your answer.

        Also, if people cared so much about “single, statically linked, standalone executables” I would not have neither Python nor Perl installed on my system, and on the systems I’m connecting to over SSH. Or any other “dynamically linked” utilities, including SSH.

        1. 3

          Also, if people cared so much about “single, statically linked, standalone executables” I would not have neither Python nor Perl installed on my system, and on the systems I’m connecting to over SSH.

          I think the requirement to have a bajillion Python or Ruby libraries installed is why people have started to care about this sort of thing. In my experience, Go has been adopted for CLI tools partly as a rejection of this pattern since the Go compiler generates statically linked binaries (the fact that it handles cross-compilation fairly well is another benefit).

          1. 1

            Yes, “small python scripts” are basically useless if they require anything not in the stdlib, because then they’re not small anymore if you need dependencies (in this particular scenario) and that’s why most people still use shell scripts, or Go for statically compiled binaries.

      1. 2

        This article uses the original author’s Vlime repository at http://github.com/l04m33/vlime which hasn’t been updated in a few years (of course, it still works okay, since Common Lisp and Vim are generally pretty good about backwards compatibility, unlike most languages/environments).

        A while back I contacted the author and they said they had no plans to continue work, and were fine with other people taking over maintenance. A couple of months ago, Philipp Marek, Eitaro Fukamachi, and I set up https://github.com/vlime/vlime to track the changes we’ve been making. So if you encounter bugs or want more features out of Vlime, you might consider trying that new repository. I don’t think any further development will be happening at the old one.

        1. 1

          I have a twitter bot that generates images every five hours: https://twitter.com/bit_loom

          1. 1

            Wow that’s pretty cool what did you use to generate the art By the way i had the similar idea

            1. 1

              The code is linked in the Twitter profile. It’s written in Common Lisp. I made a little drawing protocol so I can output the same result as PNG (for twitter) or SVG (to plot with an AxiDraw) using various libraries.

          1. 7

            I’m very happy with my UHK (in spite of its awful name). I have three of them:

            • One with blues at home.
            • One with browns home.
            • One in the office I visit every couple of months (I work remotely) with the brown switches + rubber O rings to make it quieter. It ends up being no louder than the Microsoft keyboard IT gave me by default.

            It took a little while to get used to the split, but now that I have I won’t go back.

            The build quality is really solid, and the wooden wrist rests look beautiful. Their keymap configuration utility is pretty flexible and easy to use. I like being able to flip between separate keyboard layouts for Linux and Windows with a keystroke.

            One note to be aware of: the “modules” listed on the page haven’t actually shipped yet. They were working through a backlog of keyboard construction for a long time and are only now getting close to shipping the modules. But the keyboard is perfectly usable without them.

            1. 1

              Agreed with everything you said here. It’s a pretty compact keyboard. The one thing I haven’t figured out yet, I want the ‘mouse’ key to be an esc when pressed alone, but a mouse key when pressed with other keys. I haven’t spend much time trying to figure it out, but it’s not obvious to me that it’s possible.

            1. 18

              Another example of inadvertent algorithmic cruelty.

              1. 4

                http://www.nhplace.com/kent/PS/EQUAL.html is another Common Lisp focused discussion of this.

                1. 17

                  You can go even further and remove the need for if statements and booleans:

                  function cons(a, b) {
                      return fn => fn(a, b);
                  }
                  
                  function getFirst(pair) {
                      return pair((a, b) => a);
                  }
                  
                  function getSecond(pair) {
                      return pair((a, b) => b);
                  }
                  
                  1. 3

                    Beautiful. Truly data out of nothing but procedures.

                      1. 4

                        I’d encourage you to implement getFirst as written here, with two arguments (a, b) for the lambda passed to pair. I know it’s safe to omit the extra parameters in JavaScript, but it made me hesitate for a few seconds while mentally parsing the code.

                        1. 2

                          Good call. Done

                      2. 2

                        I like this approach better because it’s more “code-y” than the second approach mentioned in the post.

                      1. 4

                        I confess to always putting the following in every run.sh bash script. Despite the obvious simplification available, it’s always in there as-is.

                        # From https://stackoverflow.com/a/246128/3858681
                        DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
                        cd "${DIR}"
                        

                        From this StackOverflow answer

                        1. 5

                          BTW this reminded me that I added readlink to Oil specifically to reduce this boilerplate. But I honestly forgot about it until now!

                          https://github.com/oilshell/oil/issues/96

                          The shorter version with readlink is:

                          DIR=$(dirname $(readlink -f $0))
                          cd "$DIR"
                          

                          But readlink -f isn’t on OS X, which is the reason for all the cd / BASH_SOURCE stuff. And I used to use that myself too.

                          So maybe Oil needs a shortcut for $(dirname $(readlink -f $0)). It already has all the logic for it.

                          Filed issue here: https://github.com/oilshell/oil/issues/587

                          1. 1

                            GNU coreutils also have realpath which I use on my Mac all the time when I’m at the command-line and I need to get an absolute path for something else (like a COPY table FROM into DataGrip for my local postgresql).

                            ~ ➜ /tmp/example/of/cows/script.sh
                            /private/tmp/example/of/cows
                            ~ ➜ cd /tmp/example
                            example ➜ ./of/cows/script.sh
                            /private/tmp/example/of/cows
                            example ➜ cat ./of/cows/script.sh
                            #!/usr/bin/env bash
                            
                            dirname $(realpath $0)
                            

                            It is a bit overweight for this because it canonicalizes and all that which you really don’t need to get to the directory.

                            1. 2

                              If you want to handle pathological cases like commands with spaces in the name or commands named --help you’ll need something more quotes and --, something like DIR="$(dirname "$(realpath -- "$0")")".

                              Example (indentation added for readability): https://paste.stevelosh.com/5517e2b981a88dee24b7ae99dfca0c3974c38d15

                              1. 2

                                If you know you always have coreutils, then you can simplify all the BASH_SOURCE stuff to just

                                DIR=$(dirname $(readlink -f $0))

                                It does the same thing as the longer incantation.

                                I’m not sure if realpath is the same as readlink -f. If it is, then it can be even simpler with $(dirname $(realpath $0)). But I forget.

                                I got used to use the cd trick` because I didn’t have coreutils on OS X.


                                edit: or maybe it’s DIR=$(dirname $(readlink -f ${BASH_SOURCE[0]})). All this stuff is gross and hard to remember :)

                                1. 2

                                  Note that readlink isn’t POSIX, and I believe that -f isn’t supported on all platforms with readlink either (I forgot which, I just remember running in to problems or having them reported and avoiding it since).

                                  1. 1

                                    It’s like you said, I’m on a Mac (which doesn’t have -f) and deploy to Linux on Docker so I just keep it the way it works everywhere out of mental simplicity. It is true that I install coreutils everywhere though like some sort of slob throwing his clothes around the bedroom but the seduction of just copying standard incantation guaranteed to work everywhere (since I use bash everywhere) is just overpowering.

                                2. 2

                                  realpath is also on BSD, but it’s not on all Linux systems. I believe it’s missing by default on either Ubuntu, CentOS, or both (you can still install it, of course, and some Linux systems do have it by default).

                            1. 3

                              If you’re putting binary files into git you’re doing it wrong. One could argue about small files, but compiled code/executables, photos or “gifs for the readme” are definitely misplaced in a git repository.

                              1. 12

                                I do find that having image files in a resources/ directory for something like a website is often simpler than separating the two. Even then making sure that images are compressed and generally not bloating repo size / git history is essential.

                                1. 18

                                  I do find that having image files in a resources/ directory for something like a website is often simpler than separating the two.

                                  Yeah, the is exactly the use case here. Mercurial (and git) aren’t designed for handling large binary files, but if you’re checking in static assets/resources that rarely change it still tends to work fine. This repo was fine on Bitbucket for many years, and is working fine on an hgweb instance I’ve spun up in the mean time.

                                  I specifically asked about limits because if it’s just the size of the repo being a technical problem for their infrastructure, I can understand. But they would not specify any limits, but just reiterated several times that Mercurial wasn’t designed for this. So I don’t know which of these was the actual problem:

                                  1. The repo is so taxing on their infrastructure it’s causing issues for other users.
                                  2. The repo is so large it’s costing more to store than some portion of the $100/year account price can cover.
                                  3. They are morally opposed to me using Mercurial in a way that it wasn’t designed for (but which still works fine in practice).

                                  Cases 1 and 2 are understandable. Setting some kind of limit would prevent those problems (you can still choose to “look the other way” for certain repos, or if it’s only code that’s being stored). Case 3 is something no limit would solve.

                                  1. 3

                                    If you want to store large files and you want to pay an amount proportional to the file sizes, perhaps AWS S3 or Backblaze B2 would be more appropriate than a code hosting website? I don’t mean to be obtuse, but the site is literally called source hut. Playing rules lawyer on it read like saying “Am I under arrest? So I’m free to go? Am I under arrest? So I’m free to go?” to a police officer.

                                    1. 5

                                      B2 or S3 would make things more complicated than necessary for this simple repo. I’ve spun up a $5/month Linode to run hgweb and it’s been working great. I’m all set.

                                2. 6

                                  This case was hg, but the same limitations are present. Hg has a special extension for supporting this:

                                  https://www.mercurial-scm.org/wiki/LargefilesExtension

                                  And it’s considered “a feature of last resort”. It’s not designed to deal with these use-cases.

                                  LFS support requires dedicated engineering and operations efforts, which SourceHut has planned, but is not ready yet.

                                  1. 5

                                    I have a repository with mostly PNG files. Each PNG file is source code; a chunk of data inside each PNG file is machine-readable code for the graph visually encoded in that PNG’s pixels. What would you have me do?

                                    I suspect that you would rather see my repository as a tree of text files. While this would be just as machine-readable, it would be less person-readable, and a motivating goal for this project is to have source files be visually readable in the way that they currently are, if not more so.

                                    git would not support binary files if its authors did not think that binary-file support were not useful; that is the kind of people that they are and the kind of attitude that they have towards software design.

                                    With all that said, I know how git works, and I deliberately attempt to avoid checking in PNGs which I think that I will have to change in a later revision. It would be quite nice if git were able to bridge this gap itself, and allow me to check in plaintext files which are automatically presented as PNGs, but this is not what git was designed to do, and we all can imagine the Makefile which I’d end up writing instead.

                                    1. 1

                                      I like the project, but pardon my ignorance - aren’t the PNG files still binary assets produced by the “real” source code, which is the textual expression parsed to generate both the embedded bitstring and the dot graph? If they’re machine readable, that places them in the same category as compiled object files.

                                      1. 3

                                        The real source code is non-textual; it is the diagram (WP, nLab) which is being given as a poset (WP, nLab). To achieve optimal space usage, each poset is stored as a single integer which codes for the adjacency matrix. However, this compressed format is completely unreadable. There are several layers around it, but each layer is meant to do one thing and add a minimum of overhead; JSON (in the future, BSON or Capn) for versioning and framing, and PNG for display and transport. There isn’t really source code; there’s just a couple Python and Monte scripts that I use to do data entry, and I want them eventually automated away in favor of API-driven development.

                                        For example, the raw integer for this “big” poset is (at the time of writing) 11905710401280198804461645206862582864032733280538002552643783587742343463875542982826632679979531781130345962690055869140174557805164079451664493830119908249546448900393600362536375098236826502527472287219502587641866446344027189639396008435614121342172595257280100349850262710460607552082781379116891641029966906257269941782203148347435446319452110650150437819888183568953801710556668517927269819049826069754639635218001519121790080070299124681381391073905663214918834228377170513865681335718039072014942925734763447177695704726505508232677565207907808847361088533519190628768503935101450078436440078883570667613621377399190615990138641789867825632738232993306524474475686731263045976640892172841112492236837826524936991273493174493252277794719194724624788800854540425157965678492179958293592443502481921718293759598648627823849117026007852748145536301969541329010559576556167345793274146464743707377623052614506411610303673538441500857028082327094252838525283361694107747501060452083296779071329108952096981932329154808658134461352836962965680782547027111676034212381463001532108035024267617377788040931430694669554305150416269935699250945296649497910288856160812977577782420875349655110824367467382338222637344309284881261936350479660159974669827300003335652340304220699450056411068025062209368014080962770221004626200169073615123558458480350116668115018680372480286949148129488817476018620025866304409104277550106790930739825843129557280931640581742580657243659197320774352481739310337300453334832766294683618032459315377206656069384474626488794123815830298230349250261308484422476802951799392281959397902761456273759806713157666108792675886634397141328888098305747354465103699243937608547404520480305831393405718705181942963222123463560268031790155109126115213866048693391516959219000560878337219324622230146226960346469769371525338127604307953786112516810509019551617885907067412613823285538493443834790453576561810785102306389953804151473860800342221969666874213156376831068606096772785272984102609049257833898258081466729520326827598704376424140779421965233471588921765110820238036094910936640446304632443760482611408445010230964335747094869968021425396439555206085281953007985784739643408074475440039274314217788647485602069097474262381690379456154426900896918268563062231294937080146199930562645748389040251871291840481739518244706752426504146889097315360662429293711705265772337748378759001582638301784557163848933046038798381667545043026975297902178839764134784634179453671000024868722179355800776002690855305662785522771116635997791339179517016284742206819482196944663461005128697584753594559406283638837841370287286682993990297923202976404261911087739188860505577427942276773287168600954693735964671046522557013031834557159173262849132567983767216098382093390056878765856939614383049277441.

                                        1. 1

                                          Ah, okay, I see. Makes sense, thank you for explaining!

                                    2. 4

                                      I’ve seen this argument quite a number of times, and almost always without a coherent explanation of why is that wrong. What’s the rationale behind this argument?

                                      1. 4

                                        Shameless plug, I contributed heavily to this help topic back when I was the PM for Microsoft’s Git server: https://docs.microsoft.com/en-us/azure/devops/repos/git/manage-large-files?view=azure-devops

                                        FWIW I disagree with the comment up-thread which says that GIFs for READMEs don’t belong. If you’re going to check in a logo or meme or whatever, that’s perfectly fine. Just don’t do 1000 of them and don’t churn it every week.

                                        1. 2

                                          I think a big part is also “are my tools there for me or am I slave to my tools?”

                                          If I have a website and most content is under version control, it’s annoying and complicated to have (big) assets outside. Most people simply want one repo with everything inside, and it’s mostly additive, often once per week - it simply doesn’t matter if it’s the wrong tool.

                                    1. 10

                                      @ddevault Would it be possible to get a clear “Terms of Service” clarifying these sorts of use cases? 1.1 Gb seems like an excessive file size, but having a crystal clear & mutually agreed upon set of rules for platform use is essential for trust (more so for a paid service), and right now users don’t know what does and does not constitute as a reasonable use of the service .

                                      1. 37

                                        No, they’re intentionally vague so that we can exercise discretion. There are some large repositories which we overlook, such as Linux trees, pkgsrc, nixpkgs, even mozbase is overlooked despite being huge and expensive to host.

                                        In this guy’s case, he had uploaded gigabytes of high-resolution personal photos (>1.1 Gb - it takes up more space and CPU time on our server than on your workstation because we generate clonebundles for large repos). It was the second largest repository on all of SourceHut. SourceHut is a code forge, not Instagram.

                                        1. 40

                                          No, they’re intentionally vague so that we can exercise discretion.

                                          I like to call this “mystery meat TOS”. You never know what you’ll get until you take a bite!

                                          1. 24

                                            I mean, honestly, a small fraction of our users hit problems. I’ve had to talk to <10 people, and this guy is the only one who felt slighted. It’s an alpha-quality service, maybe it’ll be easier to publish objective limits once things settle down and the limitations are well defined. On the whole, I think more users benefit from having a human being making judgement calls in the process than not, because usually we err on the side of letting things slide.

                                            Generally we also are less strict on paid accounts, but the conversation with this guy got hostile quick so there wasn’t really an opportunity to exercise discretion in his case.

                                            1. 30

                                              the conversation with this guy got hostile quick

                                              Here’s the conversation, for folks who want to know what “the conversation got hostile” means to Source Hut: https://paste.stevelosh.com/18ddf23cb15679ac1ddca458b4f26c48b6a53f11

                                              1. 32

                                                i’m not a native speaker, but have the feeling that you got defensive quickly:

                                                Okay. I guess I assumed a single 1.1 gigabyte repository wouldn’t be an unreasonable use of a $100/year service. I certainly didn’t see any mention of a ban on large binary files during the sign up or billing process, but I admit I may have missed it. I’ve deleted the repository. Feel free to delete any backups you’ve made of it to reclaim the space, I’ve backed it up myself.

                                                it’s a pay-what-you-like alpha service, not backed by venture capital. you got a rather friendly mail, noticing you that you please shouldn’t put large files into hg, not requesting that you delete it immediately.

                                                ddevaults reply was explaining the reasoning, not knowing that you are a mercurial contributor:

                                                Hg was not designed to store large blobs, and it puts an unreasonable strain on our servers that most users don’t burden us with. I’m sorry, but hg is not suitable for large blobs. Neither is git. It’s just not the right place to put these kinds of files.

                                                i’m not sure i’d label this as condescending. again I’m no native speaker, so maybe i’m missing nuances.

                                                after that you’ve cancelled your account.

                                                1. 13

                                                  As a native speaker, your analysis aligns with how I interpreted it.

                                                  1. 9

                                                    Native speaker here, I actually felt the conversation was fairly polite right up until the very end (Steve’s last message).

                                                  2. 28

                                                    On the whole, I think more users benefit from having a human being making judgement calls in the process than not, because usually we err on the side of letting things slide.

                                                    Judgement calls are great if you have a documented soft limit (X GB max repo size / Y MB max inner repo file size) and say “contact me about limit increases”. Your customers can decide ahead of time if they will meet the criteria, and you get the wiggle room you are interested in.

                                                    Judgement calls suck if they allow users to successfully use your platform until you decide it isn’t proper/valid.

                                                    1. 12

                                                      That’s a fair compromise, and I’ll eventually have something like this. But it’s important to remember that SourceHut is an alpha service. I don’t think these kinds of details are a reasonable expectation to place on the service at this point. Right now we just have to monitor things and try to preempt any issues that come up. This informal process also helps to identify good limits for formalizing later. But, even then, it’ll still be important that we have an escape hatch to deal with outliers - the following is already in our terms of use:

                                                      You must not deliberately use the services for the purpose of:

                                                      • impacting service availability for other users

                                                      It’s important that we make sure that any single user isn’t affecting service availability for everyone else.

                                                      Edit: did a brief survey of competitor’s terms of service. They’re all equally vague, presumably for the same reasons

                                                      GitHub:

                                                      [under no circumstances will you] use our servers for any form of excessive automated bulk activity (for example, spamming or cryptocurrency mining), to place undue burden on our servers through automated means, or to relay any form of unsolicited advertising or solicitation through our servers, such as get-rich-quick schemes;

                                                      The Service’s bandwidth limitations vary based on the features you use. If we determine your bandwidth usage to be significantly excessive in relation to other users of similar features, we reserve the right to suspend your Account, throttle your file hosting, or otherwise limit your activity until you can reduce your bandwidth consumption

                                                      GitLab:

                                                      [you agree not to use] your account in a way that is harmful to others [such as] taxing resources with activities such as cryptocurrency mining.

                                                      At best they give examples, but always leave it open-ended. It would be irresponsible not to.

                                                      1. 17

                                                        The terms of service pages don’t mention the limits, but the limits are documented elsewhere.

                                                        GitHub:

                                                        We recommend repositories be kept under 1GB each. Repositories have a hard limit of 100GB. If you reach 75GB you’ll receive a warning from Git in your terminal when you push. This limit is easy to stay within if large files are kept out of the repository. If your repository exceeds 1GB, you might receive a polite email from GitHub Support requesting that you reduce the size of the repository to bring it back down.

                                                        In addition, we place a strict limit of files exceeding 100 MB in size. For more information, see “Working with large files.”

                                                        GitLab (unfortunately all I can find is a blog post):

                                                        we’ve permanently raised our storage limit per repository on GitLab.com from 5GB to 10GB

                                                        Bitbucket:

                                                        The repository size limit is 2GB for all plans, Free, Standard, or Premium.

                                                        1. 9

                                                          I see. This would be a nice model for a future SourceHut to implement, but it requries engineering effort and prioritization like everything else. Right now the procedure is:

                                                          1. High disk use alarm goes off
                                                          2. Manually do an audit for large repos
                                                          3. Send emails to their owners if they seem to qualify as excessive use

                                                          Then discuss the matter with each affected user. If there are no repos which constitute excessive use, then more hardware is provisioned.

                                                          1. 11

                                                            Maybe this is something you should put on your TOS/FAQ somewhere.

                                                        2. 8

                                                          This informal process also helps to identify good limits for formalizing later.

                                                          Sounds like you have some already:

                                                          • Gigabyte-scale repos get special attention
                                                          • Giant collections of source code, such as personal forks of large projects (Linux source, nix pkgtree) are usually okay
                                                          • Giant collections of non-source-code are usually not okay, especially binary/media files
                                                          • These guidelines are subject to judgement calls
                                                          • These guidelines may be changed or refined in the future

                                                          All you have to do is say this, then next time someone tries to do this (because there WILL be a next time) you can just point at the docs instead of having to take the time to explain the policy. That’s what the terms of service is for.

                                                      2. 8

                                                        Regardless of what this specific user was trying to do, I would exercise caution. There are valid use cases for large files in a code repository. For example: Game development, where you might have large textures, audio files, or 3D models. Or a repository for a static website that contains high-res images, audio, and perhaps video. The use of things like git-lfs as a way to solve these problems is common but not universal.

                                                        To say something like, “SourceHut is a code forge, not Instagram” is to pretend these use cases are invalid, or don’t exist, or that they’re not “code”, or something.

                                                        I’ve personally used competing services like GitHub for both the examples above and this whole discussion has completely put me off ever using Sourcehut despite my preference for Mercurial over Git.

                                                        1. 4

                                                          I agree that some use-cases like that are valid, but they require special consideration and engineering work that hg.sr.ht hasn’t received yet (namely largefiles, and in git’s case annex or git-lfs). For an alpha-quality service, sometimes we just can’t support those use-cases yet.

                                                          The instragram comparison doesn’t generalize, in this case this specific repo was just full of a bunch of personal photos, not assets necessary for some software to work. Our systems aren’t well equipped to handle game assets either, but the analogy doesn’t carry over.

                                                    2. 4

                                                      I don’t think the way you’re working is impossible to describe, I think it’s just hard and I think most people don’t understand the way you’re doing and building business. This means your clients may have an expectation that you will give a ToS or customer service level that you can not or will not provide

                                                      To strive towards a fair description that honours how you are actually defining things for yourself and tries to make that more transparent without having to have specific use cases, perhaps there is a direction with wording such as:

                                                      • To make a sustainable system we expect the distribution of computing resource usage and human work to follow a normal distribution. To preserve quality of service for all clients and to honour the sustainability of the business and wellbeing of our stuff and to attempt to provide a reasonably uniform and undestandable pricing model, we reserve the right to remove outliers who use an unusually large amount of any computing and/or human resource. If a client is identified as using a disproportionate amount of service, we will follow this process: (Describe fair process with notification, opportunity for communication/negotiation, fair time for resolution, clear actions if resolution is met or not).
                                                      • This system is provided for the purposes of XYZ and in order to be able to design/optimise/support this system well we expect all users to use it predominatly for this purpose. It may be the case that using our system for other things is possible, however in the case we detect this we reserve the right to (cancel service) to ensure that we do not arrive at a situation where an established client is using our service for another prupose which may perform poorly for them in the future because it is not supported, or may become disproportionately hard for us to provide computing resource or human time for because it is not part of XYZ. This will be decided at our discretion and the process we will follow if we identify a case like this is (1,2,3)
                                                      1. 2

                                                        Would it be possible to get a clear “Terms of Service” clarifying these sorts of use cases?

                                                        No, they’re intentionally vague so that we can exercise discretion. There

                                                        May I suggest, perhaps: “ToS: regular repositories have a maximum file size X and repository size Y. We provide extra space to some projects that we consider important.”

                                                        1. 1

                                                          No, they’re intentionally vague so that we can exercise discretion.

                                                          Funny way to say “so I can do whatever I want without having to explain myself”

                                                          1. 15

                                                            I think that’s unfair. He did in fact explain himself to the customer and it was the customer who decided to cancel the service. I’d agree if the data was deleted without sufficient warning, but that is not the case here.

                                                      1. 4

                                                        i’d always like to hear what provoked the “condescending replies”. sounds like a fun guy :)

                                                          1. 27

                                                            I don’t read anything condescending, he does not and know anything about you, he has to explain things clearly and can’t assume anything about what the end user does or does not understand.

                                                            1. 12

                                                              Yeah, me neither.

                                                              “I’m a contributor to Mercurial, but thanks for explaining how it’s designed to me.” is a pointlessly aggressive response unless you expect him to somehow know (and remember) that.

                                                              1. 12

                                                                On the other hand, stating that “hg is not suitable for your use case” strikes me as rather patronizing. It’s demonstrably false as evidenced by the fact that this repo exists, and has been working like this for a while on BitBucket. So clearly it works and Drew’s un-nuanced assertion is false.

                                                                Drew’s case would have been much better if he had just stated “sorry, we don’t support this particular use case” instead of saying that “you’re doing it wrong”.

                                                                I’m not trying to defend Steve here, but no one is exactly smelling like roses in this conversation. Both parties could have done better.

                                                                1. 2

                                                                  That’s fair.

                                                              2. 7

                                                                Ditto, the only one being condescending as far as I can tell was @sjl. @ddevault acted in a professional manner.

                                                                1. 4

                                                                  Professional in terms of tone. Not sure I would describe his decisions as professional, but that’s clearly more subject to debate, based on the number of comments here.

                                                                  1. 2

                                                                    My interpretation of the interaction was that @ddevault’s initial email was giving @sjl a heads up and wasn’t specifically ordering him to take down his files. @sjl took down his files voluntarily, but I think @ddevault’s initial email left open the possibility of a discussion / negotiation, which seems courteous and professional to me. I.e. there was no explicit decision made on @ddevault’s part, outside of the initial warning. Maybe I’m wrong in my interpretation

                                                          1. 3

                                                            Maybe there’s more details not included in this article but it seems like the bigger problem is numerical instability in whatever algorithms they are dealing with, not the unpredictability of sort.

                                                            1. 6

                                                              From the paper:

                                                              In the end, the inconsistency was traced to differences in the default file-sorting algorithm in Python across platforms, as shown in Figure 2, and the fact that, as written, the script “nmr-data_compilation” assumes that the frequency and NMR files are sorted in the same order.

                                                              So it doesn’t have anything to do with numerical instability.

                                                              1. 1

                                                                That makes a lot more sense.

                                                              2. 2

                                                                bigger problem is numerical instability in whatever algorithms they are dealing with, not the unpredictability of sort.

                                                                The rules of glob are well defined by POSIX. However, it’s likely the case that since Python runs on non-POSIX systems, it implements its own version entirely, leading to a mismatch in expectations, and therefore a problem. (Though, I don’t recall if glob specifies the order, or just the special characters and how they match.)

                                                                So, given that chemists aren’t known for the most elegant of programs, and he mentions “When I wrote the scripts 6 years ago, the OS was able to handle the sorting,” (i.e. he doesn’t fully understand the underlying implementation, but expects a certain behavior – not an uncommon thing from even experienced, professional programmers, mind you) my guess is that there’s something more sinister going on like:

                                                                part1 = process1(files[0:2]) 
                                                                part2 = process2(files[2:7])
                                                                part3 = process(3(files[7:])
                                                                result = part1 - (part2 / part3)
                                                                

                                                                Regardless of the actual root cause, the fact that 150-160 studies were cargo culted based on the original research is the real story here.

                                                                1. 4

                                                                  From the glob specification:

                                                                  The pathnames are in sort order as defined by the current setting of the LC_COLLATE category, see the XBD specification, LC_COLLATE.

                                                                  If I understand correctly, two computers running the same OS can sort in different ways and still be POSIX compliant.

                                                                  (That might be out of date)

                                                                  1. 4

                                                                    my guess is that there’s something more sinister going on like

                                                                    You don’t have to guess. The code for both is available in the “supplementary materials” zip files for the original Nature paper and the recent paper:

                                                                    The script reads all the *.out files into one big list, which was previously not explicitly sorted:

                                                                    # https://gist.github.com/sjl/a675c449a5452cb96a9fd8ce49741888#file-foo-py-L362-L370
                                                                    def read_gaussian_outputfiles():
                                                                        list_of_files = []
                                                                        for file in glob.glob('*.out'):
                                                                            list_of_files.append(file)
                                                                        if (len(list_of_files) == 0):
                                                                           for file in glob.glob('*.log'):
                                                                               list_of_files.append(file)
                                                                        list_of_files.sort()
                                                                        return list_of_files
                                                                    

                                                                    The data for each molecule is contained in two separate files: nmr-….out and freq-….out. So then the script splits the big list of files into two separate lists of nmr and freq files by iterating over the list and skipping files that don’t contain a particular word:

                                                                    # https://gist.github.com/sjl/a675c449a5452cb96a9fd8ce49741888#file-foo-py-L346-L360
                                                                    
                                                                    def read_gaussian_nmr_outfiles(list_of_files):
                                                                        list_of_nmr_outfiles = []
                                                                        for file in list_of_files:
                                                                            if file.find('nmr-') !=-1:
                                                                                list_of_nmr_outfiles.append([file,int(get_conf_number(file)),open(file,"r").readlines()])
                                                                    
                                                                        return list_of_nmr_outfiles
                                                                    
                                                                    def read_gaussian_freq_outfiles(list_of_files):
                                                                        list_of_freq_outfiles = []
                                                                        for file in list_of_files:
                                                                            if file.find('freq-') !=-1:
                                                                                list_of_freq_outfiles.append([file,int(get_conf_number(file)),open(file,"r").readlines()])
                                                                    
                                                                        return list_of_freq_outfiles
                                                                    

                                                                    These two lists are processed individually for a while and then are passed to get_chemical_shifts, which iterates through the nmr list and retrieves the corresponding freq entry by indexing into the freq list (comments mine):

                                                                    # https://gist.github.com/sjl/a675c449a5452cb96a9fd8ce49741888#file-foo-py-L285-L301
                                                                    
                                                                    def get_chemical_shifts(lofc_nmr, lofe):
                                                                        ATOM_NUMBER = 0; ATOM_SYMBOL = 1; ISOTROPIC_VALUE = 4
                                                                        counter = 0
                                                                        #           ITERATING THROUGH FIRST LIST
                                                                        for file in lofc_nmr:
                                                                            proton_chemicalshift_table = []
                                                                            carbon_chemicalshift_table = []
                                                                            for line in file[2]:
                                                                                if "Isotropic" in line:
                                                                                    linesplit = line.split()
                                                                                    if linesplit[ATOM_SYMBOL] == "C":
                                                                                        carbon_chemicalshift_table.append([linesplit[ATOM_NUMBER],linesplit[ISOTROPIC_VALUE]])
                                                                                    if linesplit[ATOM_SYMBOL] == "H":
                                                                                        proton_chemicalshift_table.append([linesplit[ATOM_NUMBER],linesplit[ISOTROPIC_VALUE]])
                                                                            # INDEXING INTO SECOND LIST
                                                                            lofe[counter].append(carbon_chemicalshift_table)
                                                                            lofe[counter].append(proton_chemicalshift_table)
                                                                            counter += 1
                                                                        return lofe
                                                                    

                                                                    If the original list is sorted, everything works, because it looks like:

                                                                    [freq1, freq2, freq3, nmr1, nmr2, nmr3]
                                                                    

                                                                    and then gets split into:

                                                                    [freq1, freq2, freq3]
                                                                    [nmr1, nmr2, nmr3]
                                                                    

                                                                    and then the iteration and indexing pairs the correct files with each other. But if the original list isn’t sorted:

                                                                    [nmr3, freq1, freq3, nmr1, freq2, nmr2]
                                                                    

                                                                    then the splitting produces lists like:

                                                                    [nmr3, nmr1, nmr2]
                                                                    [freq1, freq3, freq2]
                                                                    

                                                                    and the iteration/indexing pairs up the wrong files with each other.

                                                                    1. 2

                                                                      You don’t have to guess. The code for both is available in the “supplementary materials” zip files for the original Nature paper and the recent paper:

                                                                      I didn’t have time to research the issue. Perhaps that should have meant I keep my mouth shut, but the obvious conclusion (which happened to be in the same ballpark, but out in left field) is a list gets built and split up in some way for different processes.

                                                                      Thanks for doing the research and analysis of the root cause. It was a really interesting read!

                                                                    2. 1

                                                                      Python’s glob.glob() indeed doesn’t use the platform libc’s glob() at all. Instead it uses functions from Python’s os module to get a list of filenames in the searched directory/directories, and does matching of the pattern against the filenames in Python. You can find the Python 3.8.0 implementation of glob.glob() here, for example.

                                                                      The cross-platform variation comes from the first part of that: Python’s os.scandir()/os.listdir() are implemented in C and call the appropriate low-level directory-listing functions for the operating system you’re using. And that’s not guaranteed to order the same way, or at all, on every platform/filesystem.

                                                                      And if anyone’s wondering why Python does it this way: I don’t know for certain, but my guess from reading the implementation of the Python os.scandir() is mostly for normalization of the different platforms’ directory-listing results.

                                                                  1. 6

                                                                    Common Lisp

                                                                    Backquote doesn’t guarantee the resulting list structure is freshly consed, so using it as a quick way for “templating” lists isn’t completely safe.

                                                                    There’s no reliable way to determine whether a stream is a binary or character stream, i.e. no way to answer “can I call read-byte on this thing without it erroring?”. There’s no binary-stream-p or character-stream-p, and stream-element-type doesn’t work for bivalent streams:

                                                                    (subtypep (stream-element-type *standard-output*) 'character)
                                                                    T
                                                                    
                                                                    (write-byte 64 *standard-output*)
                                                                    @
                                                                    

                                                                    You just have to try calling read-byte or write-byte and see whether it errors, but of course that has side effects. Welp.

                                                                    read-line always returns a string, even if you’re reading from a stream with :element-type 'base-char. So if you want to read lines and get back simple-base-strings for performance (e.g. when reading data from 20gb FASTQ files), you have to implement that yourself. (I’m not 100% sure the standard requires this — maybe this is something an implementation could actually do.)

                                                                    peek-char and unread-char are nice, but there’s no peek-byte and unread-byte.

                                                                    Pathnames are bonkers.

                                                                    If you want to write a nicely-behaved iteration macro in pure CL you really have to use do as a base, not loop (and do is annoying to use). Otherwise if a user uses your iteration macro inside a loop and calls loop-finish it won’t do what they expect.

                                                                    Allowing defmethod without a corresponding defgeneric makes it way too easy to typo the method name in various ways and spend 15 minutes wondering why the hell your method isn’t getting called.

                                                                    (subtypep 'real 'complex) ;=> NIL

                                                                    Package local nicknames aren’t part of the standard. This is improving though — most of the major implementations support them and there’s a trivial-package-local-nicknames library now.

                                                                    There’s no custom hash functions for hash tables in the standard (there’s a portability library for it though).

                                                                    There’s a lot of fancy features on arrays that not very many people use (e.g. displacement) that make it harder for implementations to optimize them unless you do everything with simple-arrays. Removing some of the lesser used features would let implementations do more.

                                                                    Performance profiling is not great. SBCL’s statistical profiler is alright, but not as nice as something like the profiler in jvisualvm.

                                                                    There’s probably more, but that’s all I can think of off the top of my head.

                                                                    1. 9
                                                                      [alias]
                                                                              tags = tag -l
                                                                              branches = branch -a
                                                                              remotes = remote -v
                                                                      
                                                                      1. 3

                                                                        Thanks! I recall those (or equivalents) are built into mercurial. Anybody know any other band-aids over git’s cli?

                                                                        I should mention, to the git teams credit, git has been slowly becoming more sane and consistent about flags and behaviours

                                                                        1. 3
                                                                          [alias]
                                                                              root = rev-parse --show-toplevel
                                                                              discard-merge = reset --hard HEAD
                                                                              current-branch = rev-parse --abbrev-ref HEAD
                                                                              addremove = !git add . && git add -u
                                                                          
                                                                          1. 2

                                                                            Of course you’d comment on it. During the brief project where I did use mercurial, your blog’s posts on mercurial got a lot of reading.

                                                                            Thanks!

                                                                      1. 2

                                                                        Don’t loop, iterate

                                                                        It’s much more in keeping with the ethos of Lisp.

                                                                        1. 1

                                                                          Very interesting, the article links to this passage that I have to agree with:

                                                                          First, many dyed-in-the-wool Lisp hackers simply find it ugly. Second, it requires learning the syntax of a whole new sublanguage. Third, the absence of parens makes it hard to parse, both by machine and, more importantly, by human. Fourth, one often has to consult the manual to recall lesser-used aspects of the strange syntax. Fifth, there is no good interface with the rest of Lisp, so loop clauses cannot appear inside Lisp forms and macros cannot expand to pieces of loop. And Sixth, pretty-printers and indenters that don’t know about loop will invariably display it wrongly.

                                                                          I was just playing around with some CL and it wasn’t as pleasant as Scheme (why, for example, is and a macro? look me half an hour to even realize it). I’m guessing anyone is prone to over-abuse loop when they’re starting to learn CL, especially when I’m not too familiar with other functions and macros. Sadly iterate doesn’t seem to be part of my standard setup, and installing packages seems to me to be a mess.

                                                                          1. 2

                                                                            I think ‘and’ is a macro so it can do short-circuiting, but yeah, it is very annoying.

                                                                            1. 2

                                                                              and and or are macros in Scheme too.

                                                                              1. 1

                                                                                Other people designing languages or macros might want fewest number of constructs they can have balancing simplicity, performance, maintainability, and so. LOOP tried to do it all. The resulting article covers a lot of ground on use cases and techniques for loops. It could be a nice, starting point for considering better designs without the quoted problems. I’m bookmarking this thread for that reason.

                                                                                1. 1

                                                                                  What’s the mess around package installation? (ql:quickload :iterate) is all it should take.

                                                                                  1. 1
                                                                                    1. It doesn’t work for me
                                                                                    2. I’ve seen multiple (or I assume they were different) package instalation systems, and have no idea which to use or what to use which for.
                                                                                    3. I can’t find a good overview or introduction to package managment.

                                                                                    I installed it using apt in the end

                                                                                    1. 3

                                                                                      There are no package installation systems in Common Lisp because you don’t install packages, you install systems. This is (yet another) case where CL uses strange names that confuse newcomers for various historical reasons.

                                                                                      I wrote this a while ago that might help you understand the differences between packages/systems/asdf/quicklisp/etc, if you’re interested: http://stevelosh.com/blog/2018/08/a-road-to-common-lisp/#structure

                                                                                      1. 2

                                                                                        It doesn’t work for me

                                                                                        Well, that’s not very helpful for debugging the problem. Is Quicklisp installed correctly? What error did you get?

                                                                                        I’ve seen multiple (or I assume they were different) package instalation systems, and have no idea which to use or what to use which for.

                                                                                        I don’t know of any other package installations systems, but Quicklisp has been the de-facto standard for a while.

                                                                                        I can’t find a good overview or introduction to package managment.

                                                                                        The short answer is “Use Quicklisp”, but the book Common Lisp Recipes covers package and system management and also covers using Quicklisp.

                                                                                        1. 1

                                                                                          I’ll just put a point on what others of said: Quicklisp is the thing for common lisp…

                                                                                          …Just as pip is the thing in Python and Maven is the thing in Java and CPAN is the thing in perl and list-packages is the thing in emacs (for this last one, you want to add 3rd party repos!).

                                                                                          Personally, I make heavy use of the quicklisp local-projects directory, which allows me to make local modifications to “libraries” (“systems”), which I seem to need to do more often in CL than other ecosystems.

                                                                                          1. 2

                                                                                            …Just as pip is the thing in Python and Maven is the thing in Java and CPAN is the thing in perl and list-packages is the thing in emacs (for this last one, you want to add 3rd party repos!).

                                                                                            Maybe I just had a bad introduction or something, but it just wasn’t that easy for me to see that quicklisp takes the role of pip, maven, cpan, etc. in the common lisp sphere. I also couldn’t really rate the maturity. Qi seems newer, while ASDF seems to be more official, and I still can’t say if there are more. All of them seem rather intransparent from outside…

                                                                                            (And regarding Emacs, I’m quite familiar with it’s situation)

                                                                                            1. 1

                                                                                              Qi

                                                                                              Ok, never heard of that. It seems to live here: https://github.com/CodyReichert/qi/ …I’ll form opinions about it momentarily. :)

                                                                                              1. 1

                                                                                                Looks like it visits https://github.com/CodyReichert/qi-manifest/blob/master/manifest.lisp when you ask it to update the list of “packages”. Ok, there are 1314 packages defined in that file.

                                                                                                Quicklisp has 1646 listed on https://www.quicklisp.org/beta/releases.html.

                                                                                                Qi’s Manifest was initially seeded by Quicklisp’s projects which means that any project you can find in Quicklisp can be found in Qi.

                                                                                                Oh, so, Qi needs to refresh.

                                                                                                Who is this Qi developer anyway? https://github.com/CodyReichert?tab=followers I don’t recognize any of these people, but I recognize the lisp mascot and some cons cells in the profile images… Maybe these folks don’t hang out in #emacs or #lisp? (Pardon me if they do and actually I’m the one out of touch!)

                                                                                      2. 1

                                                                                        Actually and is a macro in Scheme too.

                                                                                        And with good reason. The evaluation rule for a function call is to first evaluate all of its arguments and then apply the function to the results. You don’t want and to do that. You want it to evaluate its arguments in turn but stop when one of them is false (nil, whatever).

                                                                                        And it’s not at all hard to write a functional and that first evaluates all of its arguments and then does conjunction.

                                                                                        1. 1

                                                                                          You’re right, I just never realized it. I just tried it out in scheme too, and at least with MIT Scheme, I can’t pass and as a argument to a higher order function (such as reduce, which is what I tried to do with CL since I couldn’t find a any).

                                                                                          Thank god it’s easy to write anything you need easily. I just appreciate it when certain common concepts are already offered be default.

                                                                                          1. 3

                                                                                            CL offers what you need by default, but names it something you don’t expect:

                                                                                            • (reduce #'and ...) is (every #'identity ...)
                                                                                            • (reduce #'or ...) is (some #'identity ...)

                                                                                            Note that every and some can take multiple sequences:

                                                                                            (every #'< '(1  2 3) '(10 20 30)) ; t
                                                                                            (every #'< '(1 99 3) '(10 20 30)) ; nil
                                                                                            
                                                                                            (let ((lower-bounds '(  0   0 10 10))
                                                                                                  (upper-bounds '(100 100 90 80))
                                                                                                  (values (loop :repeat 4 :collect (random 100))))
                                                                                              (every #'< lower-bounds values upper-bounds))
                                                                                            
                                                                                            1. 1

                                                                                              I ended up using that in the end, but as you say, it’s not what I excepted.

                                                                                        2. 1

                                                                                          I was just playing around with some CL and it wasn’t as pleasant as Scheme (why, for example, is and a macro? look me half an hour to even realize it).

                                                                                          and is a macro for the same reason that && isn’t a function in any programming language without lazy evaluation: short-circuiting. You want (and nil ...) to not evaluate the ....

                                                                                          1. 2

                                                                                            for the same reason that && isn’t a function in any programming language without lazy evaluation

                                                                                            That was my mistake, I was thinking in a haskell-ish way.

                                                                                        3. 1

                                                                                          Iterate is kind of unpretty in its own way. It’s neither LOOP nor SERIES.

                                                                                        1. 13

                                                                                          Thanks @sjl! I keep thinking I should try again at Lisp and this post will make a handy roadmap to refer to.

                                                                                          But my number 1 question, why aren’t there more significant projects written in lisp?

                                                                                          I would think that it would become a dependency for some important piece of software but it never is (aside from Emacs).

                                                                                          Git is so important that it will require perl and bash on my local machine (even on windows!), mercurial will include python, firefox is starting to sneak Rust in piece by piece, but is lisp just so happy being on its own island that it never gets used by any other project? Clojure flipped this by buildling the lisp on top of the other language, which I think contributes to its success.

                                                                                          Or does it’s flexibility make it difficult to use in large organizations (with varying levels of experience)? Or tsomething else?

                                                                                          Thanks

                                                                                          1. 13

                                                                                            First, there are at least some popular non-trivial projects that use Common Lisp – pgloader is an example. But you’re right that it’s certainly rare.

                                                                                            The glib answer to “why aren’t there a lot of Common Lisp projects” is”because there aren’t a lot of Common Lisp programmers”. I think there are a couple of reasons for that.

                                                                                            One reason is that the barrier to entry for Common Lisp is really high. I’ve had people tell me I could learn Go in a month. I don’t know if that’s true (though I’ll be finding out when I start a new job in Go in October) but I can tell you that you definitely cannot really learn Common Lisp in a month. I think even six months of hard work would be pushing it. It’s been three years for me, with two of them being working in CL almost full time (thanks, grad school), and I’m comfortable now but still feel like I have a lot to learn. Most people aren’t lucky enough to have a few years of their life to dedicate to learning a language, so the pool of Common Lisp programmers is always going to be smaller than languages with less of a barrier to entry.

                                                                                            Another reason is that there are some really common misconceptions about Common Lisp, the primary one being “it’s a functional programming language”. It’s not. It’s a language you can do functional programming in if you want, but is really a procedural language at heart. The misconception is bad for two reasons: people who start learning it because they want a functional language get disillusioned and quit, and people who would otherwise want a procedural Lisp never consider it because of the reputation.

                                                                                            Your example of Python is a good one because it illustrates another aspect of the chicken/egg problem: Common Lisp isn’t installed by default anywhere. If I write a small script in Python I can be pretty sure I can just run it on a server somewhere. If a newbie wants to get started writing some Python on MacOS or Linux they can just dive in and worry about all the virtualenv shit later. But CL isn’t included by default on any distro that I know of, so that’s another barrier to entry to overcome.

                                                                                            Those are just a few reasons off the top of my head. To sum it up: I think there’s a bunch of reasons that all feed back on each other and result in a high barrier to entry for Common Lisp. This means there are fewer Common Lisp programmers overall, and that results in fewer Common Lisp projects overall.

                                                                                            1. 6

                                                                                              Why is the barrier to entry for Common Lisp so high? Is it just the absurdly high number of things that can be done in it? It kinda sounds like the C++ of parenthesis-languages, if the learning curve is that long.

                                                                                              1. 10

                                                                                                A couple of reasons:

                                                                                                • It’s a big language (though not C++ big, I think). CLtL2 is one book, but it’s a thousand-page book.
                                                                                                • A lot of things are structured and/or named oddly for historical reasons. You need to unlearn and relearn a bunch of stuff, and just have to memorize other things.
                                                                                                • There’s more high and low-level power than you might be used to in a single language. If you really want to learn CL you’ll want to eventually get comfortable writing super high-level metaprogramming macros and also reading x86 assembly spit out by DISASSEMBLE.
                                                                                                • CL tends to err on the side of giving the programmer freedom instead of saying “there is exactly one way to do it” (e.g. the package system’s orthogonality with files).

                                                                                                In short: the barrier to entry is so high for the same reasons it took me ten thousand words to describe how to learn it :)

                                                                                                1. 8

                                                                                                  Shorter: Common Lisp is the combined second-system effect of multiple existing Lisp implementations, all implemented by people who worked at places like the MIT AI Lab and Symbolics and who were, therefore, accustomed to complicated and powerful systems. I mean, when your OS’s command line interpreter is a machine code debugger (HACTRN on ITS), you obviously don’t have much respect for an argument that a given feature gives programmers “too much” power.

                                                                                                  1. 4

                                                                                                    That’s exactly what I read happened. Each one was a powerful toolset people were using. Common LISP tried to be the one language to rule them all. It was a huge mess that achieved its goal. A programmer wanting LISP-like power without compatibility with older, merged LISP’s might want something else entirely. Hence, the success of some Scheme’s, Clojure, and even non-LISP’s with macro systems. There’s still people that like what Common LISP has, too.

                                                                                                    If CL is too much, there’s other options that get one many benefits.

                                                                                                  2. 3

                                                                                                    It’s a big language (though not C++ big, I think)

                                                                                                    It looks like CL standard is quite close in size to C++.

                                                                                                2. 6

                                                                                                  You don’t write about Clojure much lately, is that because of a preference for CL / distaste for Clojure? Would be interested in your current thoughts on Clojure.

                                                                                                  1. 2

                                                                                                    Having learned Go (and written the majority of a decent sized system in production), a solid programmer can be productive in 1 month. I taught it to myself (mostly) with a greenfield project and if you were working in an environment with knowledgable co-workers or some code to reference/build on, that shouldn’t be too much of a stretch.

                                                                                                    1. 3

                                                                                                      You can be a productive programmer in a couple of weeks in CL as well. IIRC Dan Weinreb said they gave new hires PCL and to weeks. But being able to contribute to a code base doesn’t require learning the whole language. For example you can be productive without learning how to customize the reader.

                                                                                                      1. 2

                                                                                                        Respectfully, I’m going to say Dan Weinreb had a sampling bias. Not all of us live near schools that have students (much less faculty) that are bastions of Lisp (or scheme) programmers. Not to mention that most MIT engineers don’t typically work next to programmers from $SMALL_STATE_SCHOOL.

                                                                                                        1. 6

                                                                                                          Lisp, and Common Lisp, are not complicated languages to learn. Quite the opposite in fact and that is the reason I like them.

                                                                                                          There is a myth that they are complicated. When I tell others Common Lisp is my favorite language one of the first things said (besides the obvious parenthesis jokes) is: “Ooh, that’s such a hard language!” I don’t know where that comes from.

                                                                                                          C++ is hard, Common Lisp is not.

                                                                                                          1. 2

                                                                                                            Not all of us live near schools that have students (much less faculty) that are bastions of Lisp (or scheme) programmers

                                                                                                            They were talking about people that didn’t knew any Lisp before joining the company. So I’m not sure why would the availability of Lisp programmers would be of any relevance to the question how long does it takes a programmer to go from I don’t know Lisp to I can contribute to a code base.

                                                                                                      2. 2

                                                                                                        I can tell you that you definitely cannot really learn Common Lisp in a month

                                                                                                        I think you can be productive in a month or two. (In any language, of course, it takes much longer to become expert.) A lot of people compare their experience learning Common Lisp on their own to learning other languages on the job, or in some other collaborative environment where it’s easy to ask questions of more experienced people, or where you are working on an existing codebase, written by people skilled in the language, with patterns you can emulate. It’s much easier to learn a language in that kind of environment, but there are few opportunities to learn Common Lisp that way. I think it’s these circumstantial factors, not anything intrinsic, that give Lisp its reputation as a hard language to learn;

                                                                                                        1. 1

                                                                                                          Is the barrier to Common Lisp really high?

                                                                                                          I’m not sure I agree. It’s pretty easy (except perhaps on Windows) to just start up SBCL and fiddle around. There’s no huge swathes of things you need to install and IDE’s to tweak to do a simple “Hello, World!” or to do some excercises from a book.

                                                                                                          Also, it’s a very practical language that does not get in the way. Lisp itself is syntactically very simple.

                                                                                                          I do admit that for certain projects (f.e. GUIs) things get a little harder and the standard library is huge but that’s something you pick up along the way.

                                                                                                        2. 10

                                                                                                          Lisp provokes an instinctive hostility from others, much like Haskell.

                                                                                                          The Common Lisp community is a crew of mostly loners.

                                                                                                          Writing Lisp well takes, in my experience, an above average programmer, and in a group, a programmer sensitive to varying capabilities.

                                                                                                          Many groups reject tools that demand anything above average.

                                                                                                          GUI integration is, as usual for most open source, terrible.

                                                                                                          Web service software tends to be messy.

                                                                                                          No company or Glamorous Person really is championing Common Lisp. Unlike Haskell, the academy has rejected Common Lisp (but tolerates Scheme).

                                                                                                          Companies absolutely loathe risk and extra investment, so the above keeps Lisp out of the corporate environment.

                                                                                                          All of the above keep Common Lisp roughly around OCaml in capability and mindshare. It’s a shame.

                                                                                                          1. 3

                                                                                                            Writing Lisp well takes, in my experience, an above average programmer, and in a group, a programmer sensitive to varying capabilities.

                                                                                                            I disagree with this assessment. The main attraction of CL to me is that it gets out of my way when I want to do something. I don’t have to jump through hoops of the arbitrary restrictions of a particular programming language. All programers may benefit from this quality. I would concede that it may take an above average programmer to exploit said freedom.

                                                                                                            1. 7

                                                                                                              A lot of programmers are simply uninterested in expressive power. And, a lot of engineering leads prefer to limit expressive power to limit cleverness. Hence Java and Go.

                                                                                                              I understand the power of Common Lisp - I maintainish two Common Lisp assistance sites and used it for 8 years nearly continuously at home; I still use it myself for my own site. But how typical industrial programming work goes is simply opposed to the Common Lisp ideals, in my experience and discussion with others.

                                                                                                          2. 3

                                                                                                            The main reason why you won’t see a Lisp ‘killer app’ as part of the user’s OS is that Lisp’s world view of the world is incompatible with Unix. Unix accommodates to different languages by providing a ‘common interface’ in processes. To some degree, one can draw the analogy with processes being function calls , the standard input being the parameters of the function and the standard output its return value. By contrast, in Lisp (and Smalltalk) you start with a base world (your image) and you add your program into the world little by little, by incremental compilation. Your compiler, debugger, and the rest of the environment is part of your image. To accommodate to the Unix world view would mean to recompile the whole program each time, which destroys the advantages of incremental development offered by Lisp.

                                                                                                            Or does it’s flexibility make it difficult to use in large organizations (with varying levels of experience)? Or something else?

                                                                                                            We know its been used in large organization (e.j. ITA now Google Flights) so its not a question of capability.

                                                                                                            1. 1

                                                                                                              I would say the mind set of not wanting to deal with the outside world seems problematic. You wouldn’t want to write a program that solves a problem for the rest of the world, because the other problems are solved by something else?

                                                                                                              I knew of ITA/Google Flights, but didn’t think it qualified as a large organization (though I’ll happily be corrected).
                                                                                                              I know that the designers of Java and much later Go and even later Dart all specifically tailored aspects of the language towards large groups with varying skill levels, but I’ve never come across anything that mentioned that was a design goal of Common Lisp - again, happy to be corrected.

                                                                                                              1. 2

                                                                                                                I would say the mind set of not wanting to deal with the outside world seems problematic.

                                                                                                                I didn’t say that Lisp doesn’t want to deal with the outside world. I said that its view of the world is different from Unix. In Lisp a ‘program’ is not different than a function. How else are you going to edit your compiler while its running? Your debugger while your debugging?

                                                                                                                Mind you that you could a Lisp implementation that performs the edit/compile/run cycle from the CLI, mocl does AoT compilation and there is wcl (which is mostly a PoC) but that would destroy any semblance of interactive development which is one of the main differentiating features of Lips. It would be no more interactive than the other batch oriented PLs, say Racket.

                                                                                                                I knew of ITA/Google Flights, but didn’t think it qualified as a large organization

                                                                                                                So what number qualifies? 50+ (Or 70 I’m not sure) developers working on a +2M LoC is a large organization in my books.

                                                                                                                1. 1

                                                                                                                  Thanks, I was unsure of the size of ITA (pre-acquisition) or Google Flights (post-acquisition).

                                                                                                          1. 3

                                                                                                            Realforce 104U. The variable-weighting has ruined me – trying to use a normal keyboard now means either the main keys are too light or the pinky keys are too heavy.

                                                                                                            1. 1

                                                                                                              Related to anaphoric macros.

                                                                                                              I like the improved style vs. Graham’s implicit bindings, but for whatever reason binding conditionals in Lisp never gained traction. Perhaps they are just harder to read.

                                                                                                              1. 1

                                                                                                                These are(were?) super common in Clojure, fwiw. I can’t fathom a reason they ‘d be popular in Clojure, but not in Common Lisp, other than one having a Lisp-1 and a community that hates typing more characters than necessary…but in that case, the implied ‘it Would be better….

                                                                                                                1. 1

                                                                                                                  I can’t fathom a reason they ‘d be popular in Clojure, but not in Common Lisp

                                                                                                                  They’re included in the Clojure standard library, but you have to write them yourself in CL (or use a library). That’s probably why.

                                                                                                                  1. 1

                                                                                                                    Right. The barrier to entry is far less in Clojure, but the idea of binding conditionals was not new in Clojure. So there was lots of opportunity for CL users to adopt it, and for it to end up in SBCL’s standard library, or be a quicklisp library away. That seems to have not happened.

                                                                                                              1. 1

                                                                                                                How hard would it be to make these without the Alexandria dependency? (I’d try it myself but just don’t have the time and energy currently.)

                                                                                                                1. 1

                                                                                                                  with-gensyms is trivial to replace. parse-body would be tougher, but still isn’t too bad.