1. 1

    Pterosaur used vims clientserver mode to embed vim directly into any text field (and wysiwyg-editor) – sadly, it stopped working years ago :(

    1. 2

      I wanted to parse <h1>, <h2>, … headers in the HTML output in order to generate a table of contents, like the one at the top of this post.

      Since I could’t find your script, here’s the one I’m using on my homepage. This perl script expects headings in the style <h2 id=foo>.

      #!/usr/bin/env perl
      
      print "<ul>\n";
      $depth = 2;
      while(<>) {
      	if (/<h([2-3]) +id=["']?(.+?)["']?>(.+?)<\/h\1>/) {
      		print " "x($1-2)."<ul style=\"padding:0 0 0 1em\">\n" if ($1 > $depth);
      		print " "x($1-2)."</ul>\n" if ($1 < $depth);
      		print " "x($1-2)."<li><a href=\"#$2\">$3</a>\n";
      		$depth = $1;
      	} elsif (/<h([2-3])>/) {
      		print STDERR "! No id attribute on line $.: $_";
      	}
      }
      print "</ul>\n";
      
      1. 5

        I plan to work on these things […] A “carrot” for Oil, i.e. a feature that existing shells don’t have. Ideas: [static analysis, app bundles, and crash reports]

        If you’re looking for ideas, here’s another one that might interest you: nestable string literals, a.k.a. tagged string delimiters, a.k.a. I don’t think this construct has an official name. It’s a rare language feature AFAICT, but solves an annoying and error-prone task that is especially common in shell: quoting, escaping, and multiple escaping.

        Examples of tagged string delimiters in other languages (I am aware of only two):

        • Lua’s long brackets let you write string literals (and multiline comments) as [[...]], [=[...]=], [==[…]==]`, and so forth – the number of equals signs in the opening delimiter is counted, and the string ends when the matching closing delimiter is found.

            --[[ This comments out an assignment to my_lua_string
            my_lua_string = [==[one [=[inner]=] two]==]
            ]]
          
            -- This is a string delimited with long brackets, that contains several other closing delimiters that are ignored without needing escaping
            [=[one ]==]'" two]=]
            --> 'one ]==]'" two'
          
            -- Using long brackets with loadstring (Lua's `eval`):
            f = loadstring([[i = i + "x"]])
            i = 'a'
            f()  -- i = 'ix'
            f()  -- i = 'ixx'
          
        • PostgreSQL’s dollar quoting:

            $$Dianne's horse$$
            $SomeTag$Dianne's horse$SomeTag$
          
            CREATE OR REPLACE
                FUNCTION increment(i integer) RETURNS integer AS $myAddOne$
                    BEGIN
                        RETURN i + 1;
                    END;
                $myAddOne$ LANGUAGE plpgsql;
          
        • Heredocs don’t count, because you can’t write them inline.

        Ways tagged string delimiters would improve the OSH/shell experience:

        • Tagged string delimiters make it trivial to write any string literal without escaping the contents: you simply choose a delimiter that doesn’t occur in the string.
        • Tagged delimiters make it easy to nest commands with quoted string arguments inside the quoted string argument of a higher-level command: for example, cp 'my file with.spaces' /etc within sudo sh -c 'multiple; commands; cp "..."' within ssh 'multiple; commands; sudo sh -c "..."'.
        • Shell’s string-centric nature and interactive usage means is the context in which I frequently quote strings, or pass quoted shell code that contains its own quotes that need escaping, and so on. The example in the previous item is one I have encountered in real life, albeit only once. It would be good to get away from escaping, and wonderful to get away from double escaping.

        Sorry to dump such a long post on you (although turnabout is fair play :-P), but I thought you might be interested. For my part, I find your OSH project fascinating, and avidly read every post. Thank you for writing such good writeups!

        1. 5

          Perl has nestable strings with q{} / qq{} / qx{} as well.

          print q{foo q{bar}};
          
          1. 3

            Yes I totally agree that shell’s string-centric nature means that this is one of the most important features! I often have HTML snippets in my shell scripts, and I see tons of config file snippets in shell scripts, like /etc/resolv.conf, etc. Not to mention Awk, Perl, and Python snippets.

            I think of this feature as “multiline strings”. It will subsume here docs, and as you mention you can use one as an argument to a command.

            It will replace all the variants of here docs: << EOF , << 'EOF' (quoted), <<-EOF (indented), etc.

            I’m thinking using a variant on Python’s syntax:

            # operator << takes a string literal, not a filename, like <<< in shell
            cat << 'normal string' 
            
            cat << '''
            $var is NOT expanded in 3 single quotes, equivalent to quoting 'EOF'
            '''
            
            cat << """
            $var is expanded in 3 double quotes, like unquoted EOF
            """
            

            I have thought about the “tag” problem too. I originally had a proposal to have some sort of tag, but someone pointed out to me that it’s not necessary for multiple here docs on a line. You can just use the order of the here docs.

            I guess I have never really had an issue with Python’s multiline strings – e.g. embedding a multiline string in a multiline string! The fact that there are two different types of quotes helps. But I’m open to adding that later if it’s a problem.

            I plan to write a preview of Oil syntax, as requested on lobste.rs. So I’ll make sure to highlight this part. Unfortunately I have at least 2-3 posts to write before that, so it might not come for awhile.

            There is also the issue of C escapes, e.g. $'\n' in shell.

            Thanks for the feedback!

            1. 1

              multiline strings with interpolation and also trimming of leading indentation is super handy, the way nix does it is pretty fun to use.

              1. 1

                OK interesting, I didn’t know Nix did that. That’s pretty much what I expect Oil to have – the indentation of the closing quote is what you strip off of every line.

                But ‘’ already means something in both shell and Oil, so it will be ‘’’ and “”” like Python, except that single and double mean what they already do in shell.

                https://learnxinyminutes.com/docs/nix/

          1. 7

            In short, for those who can’t listen to a video right now: They punctured the floppy at a precise location, which causes writes to that location to fail. Then, in a few lines of assembly, check if the write succeeds and start messing with the pirates.

            Quite neat, actually.

            1. 1

              The one thing I hate about programming language specific package managers is that they usually don’t set any standard for what qualifies. Anyone can upload packages and they aren’t vetted. There have been many types of weaknesses – form typo squatting attacks (from our /u/hanno IIRC), to reusing deleted package names just this week.

              They also aren’t playing along with your distro’s package manager and often encourage fixed-version dependencies (and many dependencies), leading to windows-dll-hell type situations with many versions of the same library installed.

              “But what if dependencies update and break my stuff?”

              anyone seen fefe’s talk at 34c3 (german, but simultaneous translation available)? he talks about exactly these kind of antipatterns.

              1. 2

                man, sometimes I just feel like wanting to write a much-used npm package, just so that I can delete it and cause chaos. As a non-js dev, and a minimal user JavaScript in general, seeming this always gives me Schadenfreude.

                1. 1

                  After left-pad it’s not possible to just remove a popular package, such request needs to go through the support first.

                  https://docs.npmjs.com/cli/unpublish

                  1. 4

                    I don’t understand why it’s possible to delete it in the first place. If you release a piece of code with a permissive enough license you’re accepting the fact that others can keep using that code forever and without any further permission from you. The worst you can do is to stop improving that code. So, why doesn’t npm exercise its right to keep using that piece of code?

                    1. 3

                      This is the stance on Rust’s crates.io:

                      Take care when publishing a crate, because a publish is permanent. The version can never be overwritten, and the code cannot be deleted.

                      I’m sure crates/cargo will encounter its own issues over time but it’s nice that this at least shouldn’t be one of them.

                    2. 4
                      1. write a package and spice it up with non-free and/or patent-encumbered code without license
                      2. publish on npm
                      3. wait (or help it) ’till your package becomes similarly important to left-pad
                      4. tip off / file a DMCA / etc. with npm
                      5. ???
                      6. Profit!1
                  1. 5

                    very reminiscent of Linus Neumann’s Trolldrossel (German for ‘troll throttle’): Depending on the amount of bad words in a given comment, the captcha fails regardless of the answer given. It was installed at a site providing comments for controversial german blogger Fefe when it was overwhelmed with racist, misogynistic and otherwise awful comments.

                    Here’s the 2013 talk about it, if you understand German.

                    1. 2

                      Hmmm, previous existing works make a patent attackable, no?

                      1. 4

                        Yep, it’d be existing prior art. In NZ you wouldn’t be able to patent this concept at all. They’ve banned software patents. The US really needs to do the same.

                        1. 2

                          Australia needs to do so also.

                        2. 1

                          Only if the existing work is implemented the same as each claim in the patent.

                      1. 2

                        You never make a case for what is the standard. Is your contention that there should be no such thing as a “standard” and that one should consider all probable browsers? What does that look like in practice? In my annual roadmap for 2018 major development projects, do I ignore PWA because it’s what Chrome is pushing or do I find a way of giving it 59% consideration?

                        1. 20

                          Web standards are a thing that have existed for years.

                          Unfortunately web developers jumping on the band wagon for “we only support X” is also a thing that has existed for years.

                          1. 5

                            I would contend that browser implementations have driven web innovation a lot more effectively than web standards agencies. Standards agencies gave us the wrong box model, the never-implemented CSS2, the dead end of XHTML. The web only started innovating again with the WhatWG takeover, which was effectively a coup where browser makers displaced the standards agency.

                            Browsers implement features, websites use them, they get standardised once they’ve proven themselves in practice rather than before. That’s the model that works, and using new features that chrome (or anyone else) has implemented, as and when those new features are useful to you as a web designer, is part of that.

                            1. 6

                              Such questions/statements are weird. Do you, for example, give Firefox the same consideration as Chrome for German customers? http://gs.statcounter.com/browser-market-share/desktop/germany

                              Note: NetMarketShare only gives you global statistics unpaid and hides others behind a paywall. I assume most companies don’t pay for that and don’t do proper research of their actual target audience.

                              The question is a rather broad one: Do we, as an industry, want to support one of the biggest and most nosy software companies in taking over one of the crown jewels of the free web? The users client?

                              Yes, that’s a hard question to answer day to day, when features have to be implemented and budgets are thin. It still has to be answered.

                              We have more control over the situation then it might seem. This is how Firefox won the browser war back then: users recommending other users not to use the monopoly browser. Yes, you can totally ignore what Chrome is pushing for and deliver a great product.

                              1. 0

                                I don’t think Firefox ever won the browser war; at its peak it still had significantly lower marketshare than IE.

                                1. 11

                                  Firefox never aimed for dominance, but for breaking dominance. Winning is not “getting to the highest market share”.

                                  Firefox also had multiple target markets where it was the dominant browser for a couple of years.

                                  This whole idea that you have to be on slot 1 in a market with multiple billions of users to be winning is absurd.

                                  1. -2

                                    Sounds like you are trying to redefine win to mean succeed.

                                    1. 5

                                      No.

                            1. 1

                              In the heat of this discussion I’ve also made a comment that was uncalled for. When you take a step back from the outrage (regarding a org close to many of us) you’ll see that no harm was intentioned (well meant != well done), no harm was done and we’ll get better processes out of that situation.

                              To the Mozilla devs in here, shall any of you see this: I’m sorry for stirring the outrage and thereby also attacking your work.

                              1. 26

                                Another item onto the list of stupid, self-sabotaging ideas from Mozilla.

                                • Pocket
                                • Cliqz
                                • Looking Glass
                                • (Anything else I missed?)

                                That said, I’m still a Firefox user, because after all, I still trust the Mozilla Foundation and Community more than the makers of the other browser vendors.

                                1. 10

                                  Mozilla has it’s missteps, on the other hand, they are still better than the other Browser Vendors out there and I haven’t seen a viable Firefox Fork out there that works for me. Plus it seems the Looking Glass addon was inert unless specifically enabled by the user, so I don’t see the harm tbh.

                                  “Atleast [they are] the prettiest pile of shit.” ~ Some quote I heard somewhere

                                  1. 3

                                    I would add Mozilla Persona to this list, which was a great idea, but was mismanaged and shut down by Mozilla before it could do anything good.

                                    I pretty much lost my faith in Mozilla having any idea what it is doing at that point.

                                    1. 5

                                      Original Pocket introduction was mishandled, but since Mozilla owns and operates it now, integration with Firefox makes sense.

                                      1. 7

                                        is it open source now?

                                        1. 6

                                          My understanding is, it’s not yet. It’s being worked on. I have no idea what kind of work it takes, but the intention is that it will be fully open sourced.

                                      2. 4

                                        You missed ‘Quantum.’ (The one where they broke their extension API for the sake of alleged performance).

                                        1. 45

                                          That one I actually like; the performance is much better, and the memory leaks much fewer. Pre-quantum I was on the verge of switching to Chrome because of the performance gap and leaks.

                                          1. 11

                                            I agree. The browser engine is noticeably better - if only the software around it were also on the same level. Some lightweight browser like surf or midori should adopt it, instead of WebKit.

                                            1. 1

                                              WebKit is easy to adopt because WebKitGTK and QtWebKit (or whatever it’s called) are well supported and easy to use. And Chromium has CEF. (IIRC Servo is also implementing CEF.)

                                              I don’t think current Gecko is easily embeddable into whatever.

                                              Back in the day Camino on Mac OS was a Gecko browser with a custom Cocoa UI, but doing that today would be way too hard.

                                              1. 2

                                                I should clarify, I was talking about Servo. I don’t really thing there would be a point in using Gecko, since it will probably turn into a legacy project.

                                                1. 2

                                                  It seems the other way to me? What they’re doing instead is slowly retrofitting pieces of Servo into Gecko piecemeal. (or at least, some kind of Rust equivalent to the C/C++/JS code being replaced) Servo would then be dead or explicitly turned into some staging ground for Gecko.

                                          2. 20

                                            I will go beyond alleging a performance improvement, I will attest to it. Surprisingly enough, the improvement includes Google properties such as Gmail and YouTube. They are both more responsive in Firefox now than Chromium or Chrome.
                                            On the extension side, I do not use a large number. Those which I do, however, still function.

                                            I freely admit that the plural of anecdote is not “data”, but I would feel remiss not to share how impressed I am with Quantum. Pocket has always annoyed me, so I certainly do not see Mozilla’s actions as unimpeachable and am only giving them credit for Quantum because I feel they deserve it.

                                            1. 8

                                              Based on this, Quantum was a balanced update where the team had do sacrifice the old extension API. Also, it’s not that they’ve removed extensions completely. (And no, I’m not talking about you Looking Glass)

                                            2. 8

                                              Quantum is great. uBlock Origin and uMatrix now work on Firefox for Android just as well as on desktop.

                                              1. 3

                                                ublock origin worked on firefox android before firefox quantum no ?

                                                1. 1

                                                  IIRC it worked but the UI was somewhat different. Now uMatrix is also available, and both extensions have UI that looks practically identical to the desktop versions.

                                          1. 8

                                            The BBS that I built for it was not open source, but did interact with much of the website components with telnet, SSH, and Javascript/WebSockets interfaces. (source)

                                            Quite a few people asked me how the whole system worked, and since I am not currently planning on releasing the source code (as it was largely a creative effort) (blog post)

                                            1. 1

                                              (as it was largely a creative effort)

                                              i know this is heresy to some of you, but if jcs is still attached to this code, maybe some copyleft license would be appropriate for it?

                                              1. 2

                                                Or maybe we could just collectively pay for it?

                                            1. 3

                                              With some vim trickery I’ve extracted the total sum of money mentioned in the article:

                                              US Senate: 3,538,971
                                              US House of Representatives: 5,617,841
                                              Total: 9,156,812

                                              1. 3

                                                A few gems here. I was surprised at the #2 entry but I guess that’s dependent on the audience to whom the survey was aimed at.

                                                I’m pretty sure the command to tweet via curl isn’t supported anymore as Twitter now is Oauth only. For command-line tweeting and automation, consider oysttyer.

                                                1. 2

                                                  I’ve found the Python HTTP server useful for getting files into a VM. (And yet I still never can remember what IPs QEMU uses…)

                                                1. 8

                                                  Next up: a complete flight simulator inside cat

                                                  1. 24

                                                    Or a text editor within emacs!

                                                    1. 16

                                                      Or a text editor within emacs!

                                                      It’s been done: https://www.emacswiki.org/emacs/Evil

                                                    2. 8

                                                      You jest, but I only realised in the last year that cat can read from unix sockets as well as files. (I was reimplementing it to learn a new language and read the manpage carefully.) Never realised/knew that before.

                                                      1. 4

                                                        well, bash can read from a tcp socket too:

                                                        exec 3<>/dev/tcp/lobste.rs/80
                                                        echo -e "GET /\r\n" >&3
                                                        cat <&3
                                                        
                                                        1. 4

                                                          A similar feature is also implemented in GNU awk - and someone wrote a web server for it.

                                                          1. 2

                                                            That’s fascinating. That’s so new to me, I don’t even know how to look that up. Could you give me some more info about what’s going on there? Also, it interestingly doesn’t work on the Mac (it just crashes my terminal ¯_(ツ)_/¯). I suppose it could be because of an outdated version of bash, but, again, I can’t even look that up.

                                                          2. 2

                                                            That is cool, I never knew that and kept using socat. Will try cat the next time.

                                                          3. 1

                                                            I read that as “a complete flight simulator inside a cat”, I was worried you were some kind of mad scientist.

                                                            1. 1

                                                              I thought I’d created something great for the world, a flight simulator inside a cat! But little did I know, I’d created a flying man-eating MONSTER!!

                                                          1. 8

                                                            Too bad, lobste.rs-BBS isn’t running any longer – that was gorgeous on my VT220

                                                            (hint, hint sysop team :P)

                                                            1. 2

                                                              Probably not worth fixing given limited use. It as a beautiful and neat feature, though. I used it for a few days. I could also imagine a native client done that way being more useful, efficient, and secure.

                                                              1. 4

                                                                I tried it for a bit but the fact you got disconnected after like 5 minutes and I had to input my 24 char randomly generated password every time put a damper on my enthusiasm.

                                                            1. 4

                                                              do file extensions even have meaning in unix-like systems at all? i know of graphical file managers using them (sometimes) to set the program to open the file with, but other than that? same with .foo hidden files - the file system doesn’t care (if i understand correctly), only tools like ls et al (but not find for some reason)

                                                              u/rbn said it well: its the world surrounding them, not paths itself.

                                                              1. 5

                                                                File extensions don’t really matter. Atleast not under the unix systems I’ve used (including linux)

                                                                Mac does care to some extend I believe but not on a kernel level.

                                                                Having less exceptions in your code makes the code easier to maintain, this is especially true for kernel code.

                                                              1. 2

                                                                neat – too bad they are osx only! I’ll have to stay with bouncing cow then ;-)

                                                                1. 1

                                                                  It’s missing NetSurf - a HTML4.1 compliant browser with its own engine, that is still being developed. (It’s not too bad either, modulo some rendering issues on too complicated sites)

                                                                  1. 4

                                                                    I’ve started to learn perl in 2016 and I’m in love. Where else can you make a script that executes shell code that executes awk and process that in perl again that easy?

                                                                    I’ve also used it to implement some simple CGI web-“apps” like my teletext-to-html converter. It’s no beauty, but i bodged it together so fast and got up-and-running in no time at all.

                                                                    1. [Comment removed by author]

                                                                      1. 3
                                                                        my $foo = `ls -l | sort -R`;
                                                                        

                                                                        looks a lot easier to me than

                                                                        import subprocess
                                                                        tmp = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE)
                                                                        foo = subprocess.check_output(['sort', '-R'], stdin=tmp.stdout)
                                                                        
                                                                        1. 5
                                                                          foo = `ls -l | sort -R`
                                                                          

                                                                          Shorter in Ruby.

                                                                          Though I do appreciate that you’ve gone back to CGI. Web frameworks have gotten so big, we forget how simple and easy things can be. I was going to go that route for something quick that definitely didn’t need Rails, but I found out that the built in http server was even easier than that:

                                                                          require ‘webrick’
                                                                          
                                                                          server = WEBrick::HTTPServer.new(Port: 8080) # takes config for TLS too!
                                                                          
                                                                          server.mount_proc(‘/‘) do |req, res|
                                                                            res.status = 200
                                                                            res.content_type = ‘text/plain’
                                                                            res.body = ‘Hello, web!’
                                                                          end
                                                                          
                                                                          trap ‘INT’ { server.shutdown } # graceful shutdown on ctrl-C
                                                                          server.start
                                                                          

                                                                          Typed on my phone, so it probably uses invalid quote characters, oh well.

                                                                          I run most of my personal projects this way, plain Ruby without any other web server like Apache or Nginx.

                                                                          1. 2

                                                                            Why shell out when you can call internal libraries? From C# (off the top of my head and I had a REPL open):

                                                                            var r = new Random();
                                                                            var foo = Directory.EnumerateFileSystemEntries(".").OrderBy(x => r.Next());
                                                                            
                                                                            1. 1

                                                                              That’s because shell languages are designed to make it easy to describe shell operations. One can do the same thing with DSL’s or functions in other high-level languages. When I left Perl etc, my solution was to have a pre-made library and script for that to automate the boilerplate with the language itself being a high-level BASIC variant. This autogenerated code for either an industrial BASIC I was using or usually C if I wanted an optimizing compile. I had a separate command/app to use on those that essentially preprocessed them this way. Pretty much just as easy to type as shell scripts, consistent with my usual language, and penalty being a brief pause for preprocessing and compile. The latter were ultra-fast since BASIC compilers were ultra-fast.

                                                                          2. 1

                                                                            Are you learning Perl 5 or 6?

                                                                          1. 3

                                                                            someone recently mentioned Delta Chat - an instant messenger based on Push-IMAP. That would eliminate even the “central servers can be hacked” commenters’ argument (if you deem your email server secure, which you probably do).