1. 6

      A little more convincing: if you correlate with just “fifa”, the peaks do align. (And there are “fifa” spikes in the last week of June that are 10x bigger, and don’t align with “web” anything). Good reminder as to what’s really popular outside our expanding tech bubble.

      1. 4

        Wow, it’s the kind of thing that puts our little web development bubble into perspective. Just searches for a single web app are enough to swamp the numbers for “web app” in general :|

        1. 3

          Why does FIFA popularity increase every September?

          1. 8

            Like most sports franchise games, a new iteration is released annually, and in FIFA’s case it is released around September: typically $59.99 gets you minor gameplay and graphics updates, maybe a new gameplay mode nobody really cares about, and (most importantly) you get new player and team adjustments.

            Why is it released in September? Players are free to move from club to club during transfer windows which are only open twice per season — and the leagues most people play and follow (England, Spain, most European leagues) close mid to late August. So this gives the developer time to handle any late transfers and set the rosters before release time.

            Whether or not this is why “web app” spikes I’m not sure. But that’s why FIFA spikes in September.

            1. 2

              Ah, thanks for explaining! I didn’t know that.

          2. 1

            Wow, I think that’s it!

          1. 9

            Current job (small team):

            • Commit and push changes
            • Concourse CI picks those changes up and
              1. if they’re not tagged, it runs the unit tests and stops
              2. if they’re tagged, it runs the unit tests and continues
            • It builds docker images and pushes them to ECR
            • It deploys those images to ECS in the staging environment
            • We monitor the changes in staging (metrics are scraped by Prometheus w/ graphing from Grafana)
            • If everything looks good in staging we push a button in Concourse to send the images to ECS in the prod environment

            Concourse build pipeline definition, tasks and scripts are defined in the repo and infra is managed with Terraform (which Concourse runs). It took me about three days to set everything up and it has been running smoothly ever since (~6 months).

            To roll back, we just re-run older deployment jobs.

            We don’t have a dedicated QA team. Everyone tests their own changes or asks someone else on the team for help and we have an extensive unit test suite.

            Side projects (just me):

            • ./scripts/deploy runs tests and deploys either to GAE or Heroku and that’s it.
            1. 1

              Same, but Jenkins and Datadog. It just works.

              1. 2

                Oh, using the tags to decide whether to deploy or not is a niiiice idea, I’ll steal it.

            1. 2

              Quote from Wikipedia:

              An enumeration is a complete, ordered listing of all the items in a collection.

              Could someone enlight me on this? What the Article describes doesn’t seem like “complete listing”.

              1. 3

                To enumerate can also mean “to build a list” which is closer to this usage, but I’d agree it was used imprecisely.

                I’d prefer calling this a username oracle attack!

                1. 4

                  A couple decades late I think. Guess and check attacks have been called enumeration for quite a while.

                  1. 2

                    it’s never too late to tilt at windmillsencourage precise speech!

                    Legitimately though - good to know this is common parlance in the security community.

                  2. 2

                    Given enough time (possibly heat death of the universe scales) this method could create a full enumeration.

                  3. 1

                    It could be seen as a complete listing, if the “collection of usernames” isn’t interpreted to be the collection of all usernames the server has, but rather all usernames the attacker cares about.

                  1. 3

                    Trivia related to computer graphics + early Linux: Bruce Perens was the leader of Debian for awhile, and also worked at Pixar for 12 years.

                    I assume that Pixar was an early adopter of Linux, because otherwise they would have to pay commercial OS licensing fees for the hundreds / thousands of machines they used to render movies.

                    Although I read Ed Catmull’s recent book and I don’t think he mentioned Linux? That book did mention the NYIT graphics lab.

                    https://en.wikipedia.org/wiki/Bruce_Perens

                    https://en.wikipedia.org/wiki/New_York_Institute_of_Technology_Computer_Graphics_Lab

                    1. 2

                      I vaguely recall some news around 2003 (I think it was) about Pixar switching from Sun to Intel hardware, and porting renderman.

                      1. 1

                        Sun? I know they bought a ton of SGI Octanes for Toy Story.

                        1. 3

                          found this: https://www.cnet.com/news/pixar-switches-from-sun-to-intel/
                          May have been what I was recalling.

                          Maybe they used SGI before that?

                          1. 1

                            Cool! I didn’t know that.

                            You are probably right - buying SGI in the 2000s isn’t likely a smart move ;)

                          2. 2

                            This story said they used SGI for desktops and Suns for rendering.

                            Also for @trousers.

                            1. 2

                              This story said they used SGI for desktops and Suns for rendering.

                              Also for @trousers.

                              They used Suns for trousers? Sparc64 pants? A novel usecase for sure. ;)

                              I kid, I kid. Thanks for the link. :)

                              1. 3

                                They were rendering them in the movie. Had to get accurate lighting, ruffling, and so on. Geek producing it spent so much on the hardware they couldnt afford all the actors. Show got cancelled.

                                Many investors now suspect the Trouser Tour documentary was a ruse devised so the producer could play with a bunch of SGI and Sun boxes. Stay tuned for updates.

                      1. 6

                        Like with voting, this is a scenario where adding technology, esp Internet-enabled, is just a bad idea. The less tech (and potential attacks) the better. The best ways to do espionage were those in the Cold War with people, drops, and ways of hiding stuff in other stuff. If distance is a problem, then burst radio was the best way to do it. There’s still spies being caught in the U.S. using radio. It’s probably a safe route for Chinese spies if the NSA and its wireless partners still haven’t clamped down on it domestically.

                        Additionally, they could just put the files encrypted in online storage from a random, hot spot. Then, send a coded version of the link via the shortwave, hidden message in mail, or drop.

                        1. 5

                          I couldn’t help but compare this to the Russian(?) operatives who were communicating via coded comments on a particular Britney Spears instagram post.

                          1. 4

                            This reminded me of the number stations, broadcast on short wave. Its reasonable for any civilian to have a radio and the broadcasts can be encoded with any book freely available from a library.

                            When it comes to keeping hidden, low tech is best tech.

                            1. 4

                              the broadcasts can be encoded with any book freely available from a library

                              Running key ciphers are bad tradecraft, use one time pads ;)

                            2. 2

                              Internet monitoring has gotten terrifyingly powerful - although it’s worth noting that the article doesn’t say that the Chinese found the communication channel, only that they escalated their access a lot once they’d found the channel in the first place - but radio monitoring has also advanced, with cheap and powerful software-/FPGA-defined radio and very powerful post-processing. How sure are you that radio is a good option?

                              1. 2

                                @c12 has the right idea. There’s both burst transmission and number stations being used by spies in the US. Goes back to Cold War at least. Watching the prosecutions, we rarely see anyone get caught with that method described despite NSA operating the largest array of SIGINT collection in existence. That means they’re letting spies they know about continue to operate (eg poisoned intel) or they can’t find them.

                                Im thinking it’s the latter. If it’s analog radio, they also can’t remotely hack it like they might try with a cellphone or computer.

                            1. 10

                              It’s all about the threat model, right?

                              If you’re worried about the NSA intercepting your DNS traffic, we don’t have any good solution today - I’m guessing that the only real difference between ISP DNS and DNS over Tor (if you’d be crazy enough to use it) is hiding in a big crowd and hiding in a very tiny, heavily watched crowd… your traffic will be monitored either way. I’m really hoping for this “Russian alternate DNS” that’s heavily be-FUDded to launch and allow some level of encrypted access for non-Russian citizens, as I think NickP’s antagonistic jurisdiction model is the only real chance we have.

                              If you’re worried about leaking data to big businesses, Cloudflare has a ton of it anyway as they host so many endpoints. But being able to pull them out of the loop sure would be an improvement.

                              This definitely solves the script kiddie in the coffee shop attack though, which seems to be the only traffic interception risk we can seem to get meaningful traction against, so hey - that’s a win.

                              1. 5

                                Seems to be some kind of viewport issue on iOS / Safari- text on the left and right is clipped off:

                                https://imgur.com/a/kl9P6PG

                                1. 4

                                  I have the same issue on mobile chrome.

                                  1. 1
                                  1. 3

                                    This was a very interesting post. I really enjoyed the take away around byte-seconds:

                                    how do you measure overall resource consumption? The answer is to multiply time by memory, which gets you a single dimension measured in byte-seconds. […]relatively it makes a lot of sense: If you use twice as much memory as your competitor, you can make up for it by being twice as fast.

                                    Embarrassingly, this never occurred to me as a way to think about multi-dimensional optimization problems. I’m definitely going to use this in the future.

                                    1. 3

                                      The nodes gossip periodically to ensure the leader is still there. If the leader ever dies, a new leader will be elected through a simple protocol that uses random sleeps and leader declarations. While this is simple and unsophisticated, it is easy to reason about and understand, and it works effectively at scale.

                                      The sound of three Byzantine generals cackling in the distance was heard right before the point of sale systems mysteriously crashed.

                                      1. 1

                                        It reads like a trimmed down version of how Raft elections work less the log shipping.

                                        1. 1

                                          Kubernetes uses etcd that implements raft IIRC

                                        1. 3

                                          Is there a comprehensive and/or up-to-date set of recommendations for simple, static HTTP servers anywhere?

                                          After years of trying to lock down Apache, PHP, CMSs, etc. and keep up to date on vulnerabilities and patches, I opted to switch to a static site and a simple HTTP server to reduce my attack surface and the possibility of misconfiguration.

                                          thttpd seems to be the classic option, but I’m a little wary of it due to past security issues apparent lack of maintainance (would be fine if it were “done”, but security issues make that less credible). I’m currently using darkhttpd after seeing it recommended on http://suckless.org/rocks

                                          Edit: I upvoted the third-party hosting suggestions (S3, CloudFlare, etc.) since that’s clearly the most practical; for personal stuff I still prefer self-hosted FOSS though :)

                                          1. 4

                                            If all you need is static http you don’t have to host it yourself. I host my blog in Amazon S3 (because I wanted to add SSL and GitHub didn’t support that last year) and for the last 13 months it’s costs me about $0.91 / month, and about two thirds of that is Route 53 :-)

                                            AWS gives you free SSL certificates, which was one of the main drivers for me to go with that approach.

                                            1. 3

                                              I use S3 / CloudFront for static HTTP content. It’s idiot proof (important for idiots like me!), highly reliable, and I spend less every year on it than I spend on a cup of coffee.

                                              The only real security risk I worried about was that someone could DDoS the site and run up my bill, but I deployed a CloudWatch alarm tied to a Lambda to monitor this. It’s never fired. I think at my worst month I used 3% of my outbound budget :)

                                              1. 1

                                                I’ve always wondered why AWS doesn’t provide a spending limit feature… it can’t be due to a technical reason, right? I know their service is supposed to be more complex, but even the cheapest VPS provider gives you this option, often enabled by default. I can only conclude they decided they don’t want that kind of customer.

                                                1. 1

                                                  I also worried about the risk of “DDoS causing unexpexted cost” when I was looking for a place to host my private DNS zones. To me it appeared that the free Cloudflare plan (https://www.cloudflare.com/plans/) was the best fit (basically free unmetered service).

                                                  Would using that same free plan be a safer choice than Cloudfront from a cost perspective?

                                                2. 3

                                                  You’d be hard pressed to go wrong with httpd from the OpenBSD project. It’s quite stable, it’s been in OpenBSD base for a while now. It’s lack of features definitely keeps it in the simple category. :)

                                                  There is also NGINX stable branch. it’s not as simple as OpenBSD’s option, but is stable, maintained and is well hardened by being very popular.

                                                  1. 3

                                                    In hurricane architecture, they used Nginx (dynamic caching) -> Varnish (static caching) -> HAProxy (crypto) -> optional Cloudfare for acceleration/DDOS. Looked like a nice default for something that needed a balance of flexibility, security, and performance. Depending on one’s needs, Nginx might get swapped for a simpler server but it gets lots of security review.

                                                    I’ll also note for OP both this list of web servers.

                                                  2. 1

                                                    Check out this.

                                                    1. 1

                                                      Yeah, I also like this similar list, but neither provide value judgements about e.g. whether it’s sane to leave such things exposed to the Internet unattended for many years (except for OS security updates).

                                                  1. 18

                                                    “Static sites, on the other hand, are impossible to hack: there is no code running, and thus no vulnerabilities to exploit!”

                                                    That’s overstating it. We were hacking static sites all the time before the invention of web applications. We did it via their web servers or other software running on the machine. Sometimes hit the boxes of people connecting to the machine with user or admin privileges. Vulnerabilities are still found in web servers. People still use buggy software in the trusted network. So, this claim should instead say they’re either more secure or harder to hack since they just depend on a web server without extra, bug-ridden code on top. Then, maybe a recommendation of using some specific ones that have good, track record both in number of vulnerabilities and how quickly they patch them. Maybe there should be a mention of Let’s Encrypt in that section, too. Kind of a combined recommendation.

                                                    I like your additional sections on ownership and portability. Those were either not covered or barely discussed in some prior write-ups on static sites.

                                                    1. 4

                                                      That’s true – thanks for calling it out. What I typically do is deploy my static sites into an S3 bucket (with restricted permissions, obviously), then throw it behind Cloudfront for speed.

                                                      Netlify is another awesome service that makes this stuff really easy/simple and mitigates a lot of the misconfigs for web servers/etc. that many people run into.

                                                      1. 4

                                                        I like your additional sections on ownership and portability.

                                                        Funny enough, I have a problem with one of those sections.

                                                        Take a look at any outsourced products, and compare them to in-house products: with very few exceptions, in-house projects are almost always better.

                                                        I’m currently doing a POC of a competitor of Okta for enterprise authn/authz (ha!). The only metric favoring our in-house solution was cost.

                                                        I’ve found that outsourced products are often more reliable, secure, and maintainable than in-house solutions. Where in-house solutions typically win are around matching the solution to the very specific business problem they’re designed to solve. When that’s enough, it’s perfect; when it isn’t, though, the rough edges really start to be noticed.

                                                        1. 4

                                                          Heh. I think my argument there was that you shouldn’t outsource your core product code. So if you’re a web company and your website is a main driver of your business, being able to customize/control it is pretty important. Really depends on the business goals though =)

                                                          1. 3

                                                            I think that’s fair. FWIW: after going through the static site/CMS debate internally, I can also agree 100% with your conclusion.

                                                            I guess after further consideration what I’m arguing with here is your definition of “outsourcing”. To me, I’d rather toss dollars at SaaS to solve something that isn’t my core competency. If I can’t do that, I’d rather come up with an extremely narrowly tailored solution that addresses my very specific business needs. The least palatable option is a very general-purpose tool (like Drupal or Wordpress) that I still have to operationalize, as it tends to come with nasty headaches elsewhere.

                                                            It’s an odd one… I’d chose to pay someone for operational certainty around a general purpose tool over operationalizing the general purpose tool myself. I wonder what that says about me.

                                                            1. 2

                                                              I agree. For what it’s worth, you should totally check out Netlify. They make managing static sites so easy. I started using it recently and absolutely fell in love with it <3 I’m not at all affiliated with them but it is useful.

                                                      1. 7

                                                        You have two different problems in your hands and I think they requires different solutions. Fast updates in case of events and faster (bulk) data upload. SMS and HAM radio are the cheapest possible form of fast update I can think of. It’s possible to do IP routing on radio equipment and I think you can establish a mesh service to provide internet to your user if they can bring some more equipment. Once they are online they can send you data but this leads to delays in update from a user. If your app can do peer to peer replicable data you can ameliorate this problem by sharing your data to others that will maybe come back online before you. Both problems can be solved using these kind of mesh services (hardware and software). A totally different approach is to become an ISP and negotiate peering with other providers, this can cut your bills and requires not that much infrastructure.

                                                        1. 4

                                                          SMS is hard to get up and running quickly, primarily because of trying get sims in bulk from whoever the local telecom is. What I was thinking was if we can get away with getting 4-5 sims to provide sort of connection points where we could set up a basic tower or something that broadcasts wifi over an extended area running off a solar battery. We used to have these wires for sat phones that we’d toss up a tree to get a signal for instance. Even something like amplifying the tether signal from an android phone then sending the message over 3g or SMS to the countries application servers. At least we’re then down to procuring a smaller batch of sims and we can spend less time haggling with the local telecom.

                                                          Another thing we thought about sort of pie in the sky was drones, basically getting drones to do “rounds” where during the week they travel between locations, sending a notification to local mobile apps that they’ll have access to internet for the next 2 hours, or to plug in the drone and let it charge up so it can continue on it’s rounds, but drones open up a whole other can of worms in terms of regulations and also potentially getting shot down or just downright freaking people out. Even potentially setting up relay drones, where we drop drone “stations” on the way out to remote localities, then use drones to leapfrog to the location to provide syncing, lab sample pick up and leapfrog back to a central location. But they’re cost prohibitive for the complexity of the drone as well as in terms of dealing with local regulations and would only really be viable in a longer-term health surveillance context, would be too hard to get up and running in an emergency ramp-up.

                                                          The HAM radio looks really interesting actually. I’m just digging into it, they don’t really need wider internet access, we just need a way to get data from them and in some cases send back some minimal data (for instance, the wait time for a lab result for these guys can be weeks, we can’t do much about the time it takes for a sample to get to a lab, but we can speed up the return time by pushing the lab result down to them).

                                                          We’re also working on the peer to peer syncing of data on mobile and the desktop application. We’ve made some prototypes using the zyre c libs that are built on 0mq but it’s finicky so we can’t deploy it yet. The idea being that we eventually get data as users running our apps come into contact with each other and then eventually the wider internet or the ministry of healths or an NGOS installation of our systems.

                                                          We’re sort of trying to build as many avenues as possible for them to get data in/out of the locality so there’s almost always an option because the context is so critical.

                                                          The HAM radio packet data looks interesting, i’m definitely going to dig into that more and see if I can figure out prototype test for that for shifting data. Thanks for pointing me to that!

                                                          1. 2

                                                            The HAM radio looks really interesting actually. I’m just digging into it, they don’t really need wider internet access, we just need a way to get data from them and in some cases send back some minimal data (for instance, the wait time for a lab result for these guys can be weeks, we can’t do much about the time it takes for a sample to get to a lab, but we can speed up the return time by pushing the lab result down to them).

                                                            AMPRNet will definitely be your friend here. Much of the foundation exists for you to build on. I’d wager it’s mostly an “amateur vs. non-amateur” question then.

                                                        1. 6

                                                          The system first registered radar and LIDAR observations of the pedestrian about 6 seconds before impact, when the vehicle was traveling at 43 mph

                                                          % units
                                                          You have: 43 miles per hour
                                                          You want: feet per second
                                                          	* 63.066667
                                                          

                                                          Uber’s software couldn’t decide what to do with 126 yards of processing time.

                                                          1. 1

                                                            The claim of hundreds of billions of lines in COBOL sounds exaggerated.

                                                            1. 2

                                                              “Police, today, have apprehended a cartel of COBOL programmers with a shipment of COBOL code with an estimated street value of 300 billion lines…”

                                                              I think articles like this always inflate/exaggerate by some orders of magnitude and can’t be taken literally, like when you read about police arresting a dealer with a dimebag worth $1,000 “on the street.” Unless there is an actual measurement (vs. “some guy estimates …”), they could be wildly off in either direction, but will tend to round up for effect.

                                                              1. 1

                                                                It might be high, I’m not sure, but I’ve been told (no hard evidence to back this up) that a lot of COBOL code starts as copy&paste of old code. So the number of lines produced per year is large but they are not new lines.

                                                                1. 1

                                                                  Has to be some, but still. Just was trying to ballpark it: a billion means a thousand of 1MLOC projects. This is a major institutional project size, even today. The article implies there were hundreds of thousands such projects in COBOL’s heyday, or untold millions of smaller projects. Given the installed mainframe base by late 1970s, it just doesn’t check out. The only explanation is it boomed in later years, which sounds contrary to common perception.

                                                                  1. 4

                                                                    COBOL is exceptionally verbose. A programmer I know who develops new features in it today estimates that a typical module developed years ago is likely around an order of magnitude larger than if it was written in a modern language.

                                                                    After seeing the code for a legacy ERP system up close, I’d wager it’s closer to 15x larger.

                                                              1. 2

                                                                Nobody likes typing Float::INFINITY or 1.fdiv(0) over and over.

                                                                uhhhhhhhhhhhhhhhhhh

                                                                this seems dangerous

                                                                1. 4

                                                                  .. does it?

                                                                  1. 2

                                                                    OHHHH, fdiv uses floating point semantics I guess. That’s less scary. (And also makes sense - guess I missed the F the first time :))

                                                                    1. 2

                                                                      :D Right!

                                                                1. 3

                                                                  Seems like this would be trivial to detect, as fonts have a very well known and fixed representation that is present on… er, every computer in use. OCR the pages, overlay the letters, calculate deviance.

                                                                  I’d personally rather get a printed photograph where color levels were monkeyed with or other classic steganographic techniques were used. At least that way I’ve got some level of plausible deniability when state security came knocking.

                                                                  1. 3

                                                                    What did your query plan look like? Did you consider a materialized view or any other technique that’d let you solve this in the database?

                                                                    1. 2

                                                                      I also was curious if they’d exhausted the SQL options. Appears from the brief Group Builder example, they could’ve generated raw SQL queries that hit all the right indexes, without doing the unions and group by.

                                                                      As a query builder, this Group Builder looks pretty slick. The technical discussion about their history and solutions is also interesting and instructive. The Go language propaganda is kind of distracting from an other wise interesting article. Not saying that Go didn’t provide the benefits described, it just seems a little irrelavant.

                                                                      1. 2

                                                                        It looks to me as if they are just constructing boolean expressions in the web interface, e.g., the first screenshot corresponding to (as far is I understand it):

                                                                        ...
                                                                        WHERE gender != 'male' AND ( age > 50 OR censor_rating_r18 < 0.2 )
                                                                        

                                                                        (it’s not clear to me how they actually store/process the censor rating, so i’m just guessing here)

                                                                        So it’s a matter of translating from the interface to a boolean expression with “Subgroup”s mapped to parentheses and comparisons based on exclude/include specification.

                                                                        I don’t really see the necessity for unions, or am I missing something?

                                                                    1. [Comment removed by author]

                                                                      1. 9

                                                                        If the forum allows it, anyone who can link an image in their signature is “tracking” users and has access to this information.

                                                                        The 600MB file, I’d agree with, though.

                                                                        1. 0

                                                                          By the way, it was pushcx himself who replaced the big image with an humerous remark. Might not have been the brightest idea to put it there in the first place.

                                                                        2. 4

                                                                          The lack of response or action from @pushcx is sad to say the least.

                                                                          1. 3

                                                                            He was there when it happened. They saw the picture, people joked on it, pushcx removed it, put his own comment on it into my signature, i liked it, other people liked it, i kept it. Some people had a good laugh. At this point, i was still assuming that most lobste.rs users were on desktop.

                                                                            After compiling the statistics, i felt like, “Oh shit”. Mistakes were made. I can’t turn that back now.

                                                                            You should have been there when it happened, then maybe you would have an different perspective on it. I dont want that pushcx now gets shit from people missing context. Mistakes were made.

                                                                            1. 1

                                                                              Just because @pushcx was “there” when it happened doesn’t mean that it’s OK. You abused the trust we all have in this website and I’m starting to feel like @pushcx is abusing my trust in him as the sysop to act fairly across the board. Not only did you pry into the privacy of users you wasted their time, money and energy doing so.

                                                                              1. 2

                                                                                users weren’t required to download his tracking pixel. they chose to run software that would download it by default. i consider this a lesson about the state of our software ecosystem.

                                                                                1. 5

                                                                                  This is a strawman. Every browser behaves this way. What is the lesson supposed to be? Do not trust lobste.rs and move to a better community?

                                                                                  1. 2

                                                                                    are you using the term strawman to refer to any argument you disagree with? or did i actually construct some sort of strawman?

                                                                                    lynx doesn’t behave this way. firefox doesn’t behave this way, with 3rd party images disabled in matrix. the tor browser would not leak data this way. the lesson is that the web is a hostile environment because we allow it to be. if we all used more secure browsers, sites that are broken by the security features would lose traffic. but we allow it to happen.

                                                                                    1. 0

                                                                                      No, the lesson should be do not trust the browser.

                                                                                      1. 3

                                                                                        so you have a whitelist of domains that you trust or how do you use the www?

                                                                                        1. 1

                                                                                          I try to use it as little as possible and when I use it, I consider it a hostile attacker that I don’t trust.

                                                                                          If at some point there will be a bitcoin miner on the site, I won’t consider myself betrayed by anyone, as nobody made any promise to me, nor I expected anything from anyone. I will simply move on with my life. If I am concerned about blowing through my data allowance, I won’t visit radom websites in the first place.

                                                                                          It seems that currently there aren’t any javascript bitcoin miners here on this site, but I have no expectations that there won’t be any tomorrow or some other day.

                                                                            2. 2

                                                                              Probably worth probation for a week or two.

                                                                              Hey, if we are doing the 2000s BB thing, let’s go all in! ;)

                                                                            1. 2

                                                                              Hells yes!

                                                                              I am still most comfortable by far doing my text manipulation scripting in ed. Why? Because I can sit there and interactively try things out and put the commands together. IMO that’s huge.

                                                                              1. 1

                                                                                I want to know more!

                                                                                Could you share an example of how you work with ed?

                                                                                1. 4

                                                                                  OK, let’s say you have a simple text file you need to edit under script control:

                                                                                  cpatti@dev-dsk-cpatti-1e-5d911563:03:02 PM:~
                                                                                  $ cat test.txt
                                                                                  How many cans can a canner can if a canner can can cans?  A canner can can as many cans as a canner can if a canner can can cans.
                                                                                  cpatti@dev-dsk-cpatti-1e-5d911563:03:02 PM:~
                                                                                  

                                                                                  Let’s say you want to write a Limerick transmogrification script :)

                                                                                  You can fire up ed, interactively twiddle until you find the right transformations (Note that ed uses the same basic commands as vi’s command mode, vi contains ex, which is a descendant of ed):

                                                                                  Here’s my session for this example:

                                                                                  cpatti@dev-dsk-cpatti-1e-5d911563:02:47 PM:~
                                                                                  $ ed test.txt
                                                                                  130
                                                                                  s/many/much/g
                                                                                  s/canner/woodchuck/g
                                                                                  .
                                                                                  How much cans can a woodchuck can if a woodchuck can can cans?  A woodchuck can can as much cans as a woodchuck can if a woodchuck can can cans.
                                                                                  s/cans/wood/g
                                                                                  .
                                                                                  How much wood can a woodchuck can if a woodchuck can can wood?  A woodchuck can can as much wood as a woodchuck can if a woodchuck can can wood.
                                                                                  s/woodchuck can/woodchuck chuck/g
                                                                                  .
                                                                                  How much wood can a woodchuck chuck if a woodchuck chuck can wood?  A woodchuck chuck can as much wood as a woodchuck chuck if a woodchuck chuck can wood.
                                                                                  s/woodchuck chuck can/woodchuck could chuck/g
                                                                                  .
                                                                                  How much wood can a woodchuck chuck if a woodchuck could chuck wood?  A woodchuck could chuck as much wood as a woodchuck chuck if a woodchuck could chuck wood.
                                                                                  

                                                                                  Now that I have the commands I want to run, I can embed them in a here document in my script, or as I’m doing tin this case, put them into a file so I can do multiple invocations without violating DRY:

                                                                                  cpatti@dev-dsk-cpatti-1e-5d911563:02:54 PM:~
                                                                                  $ cat ed
                                                                                  s/many/much/g
                                                                                  s/canner/woodchuck/g
                                                                                  s/cans/wood/g
                                                                                  s/woodchuck can/woodchuck chuck/g
                                                                                  s/woodchuck chuck can/woodchuck could chuck/g
                                                                                  s/as a woodchuck chuck/as a woodchuck could/g
                                                                                  

                                                                                  Now let’s run our script and examine our output:

                                                                                  cpatti@dev-dsk-cpatti-1e-5d911563:03:02 PM:~
                                                                                  $ cat ed | ed -v test.txt
                                                                                  130
                                                                                  161
                                                                                  cpatti@dev-dsk-cpatti-1e-5d911563:03:03 PM:~
                                                                                  $ cat test.txt
                                                                                  How much wood can a woodchuck chuck if a woodchuck could chuck wood?  A woodchuck could chuck as much wood as a woodchuck could if a woodchuck could chuck wood.
                                                                                  

                                                                                  Now, sure, you could do EXACTLY this with a sed invocation, or with awk, or myriad other ways, but being able to interactively build the edit is incredibly powerful and can lead to super simple development of some very powerful text transformations.

                                                                                  I feel a blog post coming on :)

                                                                                  1. 1

                                                                                    Oops. One error in that ed script I fixed but wasn’t there when I ‘cat’ted the file - You have to have a ‘w’ at the end or A) the changes won’t write and B) ed will whine with its ever helpful ‘?’ error :)

                                                                              1. 10

                                                                                My main editor is acme(1), sometimes I fire up sam(1) when I either do batch edits, or when I need to do more than quick edits on remote computers. However, for general system administration of remote computers, I use ed(1), the standard editor. Its main feature compared to vi(1) is that it doesn’t take over the screen, so I don’t lose context of all the history. I find that valuable.

                                                                                While I don’t think anybody should switch to ed(1), I think that knowing how to use it is essential, and the value of not losing context while editing text should be more appreciated.

                                                                                1. 4

                                                                                  Wow, I’ve never seen anyone actually using acme outside of the plan9 team and maybe the cat-v crowd. What do you use it for? Why do you use it?

                                                                                  I have it installed via the plan9ports on my laptop, and sometimes play with it, but since I’m so used to either vi (specifically nvi or vis) or emacs keybindings, I can’t really be productive in it. And I can’t even start using sam, although I do have experience with writing scripts with ed, and have used it over slow ssh connections.In fact I belive that it might have been exactly the linked article that pushed me to bother learning anything about ed. Would you know of any resources like that for acme or sam?

                                                                                  1. 10

                                                                                    I use acme for everything that involves text files, source code or otherwise. It’s the single most important program that I use. If acme were to somehow disappear from the face of the Earth tomorrow, I would re-write it. All my friends who use acme say that they would do the same. Fortunately, because acme is so small and so simple, that wouldn’t take very long.

                                                                                    I use it because it’s by far the best text editor. As for why it is the best text editor, that is rather difficult to explain. I could enumerate a list of features, but realistically that wouldn’t be very enlightening. It’s not any single feature, it’s the whole package, and you have to experience it.

                                                                                    The most important things to me though are the mouse-based interface, the fact that text is executable, and the fact that it doesn’t have any configuration options. I suggest watching the linked Russ Cox video, it does a good job demoing acme.

                                                                                    As for sam, I don’t know of any other resource rather than the sam paper. I don’t like the sam user interface, I’d much rather use acme, but some people disagree about that and prefer sam. However, sam has some features that acme lacks. The way the program is split into multiple processes means it works extremely well for editing remote files. Even better than ed. The user interface always runs on the local terminal, while the server side does the actual edits. The protocol is extremely efficient, not just in throughput, but in latency too. Editing files over satellite connections with seconds-long pings is doable.

                                                                                    However, what sam really excels at is editing multiple files at once. I routinely edit thousands of files at once in sam, e.g. when doing large scale refactoring. I know of no better tool for that than sam.

                                                                                    1. 1

                                                                                      What OS(s) do you use acme and Sam on?

                                                                                      1. 2

                                                                                        I use sam on everything that I use, macOS, Solaris, Linux, FreeBSD, OpenBSD and Plan 9.

                                                                                        I use acme on my workstation systems, macOS and Plan 9. Exceedingly rarely I get to use a Linux desktop system. Then I use acme there too.

                                                                                        1. 2

                                                                                          Sounds like you use Sam like I use vi keybinds: Everywhere it can possibly fit.

                                                                                          I think that for me, the lack of syntax highlighting is probably one thing that will keep me from using Sam/Acme for the indefinite future, as fascinating as they are.

                                                                                          1. 3

                                                                                            syntax highlighting

                                                                                            Try dropping it for a while. You might not miss it as much as you’d think!

                                                                                            I personally thought I’d never live without it, but nowadays the only time I notice its absence is when I forget to close a string literal. Otherwise, I’m far happier without it.

                                                                                            1. 3

                                                                                              I’ve dropped it on and off when I was experimenting with using greyscale on my screens. But there are places (like HTML templates with Angular), that really suffer if you don’t have syntax highlighting, at least they did for me.

                                                                                              1. 3

                                                                                                Greyscale screens!? Trip report needed!

                                                                                                  1. 1

                                                                                                    Basically using color space simulation to get a monochromatic screens. It doesn’t get rid of syntax highlighting entirely, but reduces it by a lot, and can be pleasant when code is well structured. HTML templates aren’t given to being structured well

                                                                                    2. 3

                                                                                      Have you checked out vis? It is a bare-bones vim w/ sam commands. Plus it be used as an interactive filter when piping commands.

                                                                                      1. 3

                                                                                        I know about vis, but I never had any interest in trying it out. In general, I stay away from curses programs if I can help it, and I also avoid installing software if I can help it, even though it might be useful. I’d rather use a real GUI program, or some command line program (non-curses). Preferably something that already comes with the system (acme fails the last requirement, sadly on some Linux distributions ed fails too). I use irssi regularly and it bothers me greatly that that is the state of the art.

                                                                                        1. 1

                                                                                          I use irssi regularly and it bothers me greatly that that is the state of the art.

                                                                                          irssi was state of the art in…. 1999? It’s certainly not in a world where WeeChat exists, let alone protocols that claim to obsolete IRC.

                                                                                          1. 2

                                                                                            I looked at weechat, and it was a curses program, exactly like irssi. To me, there’s no real difference. Except that it also had a web frontend, which made it worse (!). Plus weechat, in its default configuration used more screen estate than irssi, which to me again made it worse. But thanks for the suggestion, I am very interested in alternative IRC clients.

                                                                                            Other protocols have no value to me because of network effects.

                                                                                            1. 1

                                                                                              If you’re some kind of purist, ii works.

                                                                                              1. 1

                                                                                                Yeah, I need to try out ii and ircII.

                                                                                                1. 1

                                                                                                  Isn’t there an IRC client (or however one would want to call it) for Acme?

                                                                                                  1. 1

                                                                                                    There is, but in my opinion it is not very good, and I don’t really like programs that use acme as their interface. I prefer to use acme for editing text only.

                                                                                                    1. 1

                                                                                                      IIRC (lol), ii exposes all kinds of magic files for interacting with channels. I think that’d integrate nicely with Acme by default

                                                                                            2. 1

                                                                                              Something I do fairly often is SSH to a remote server to take a look at its logs. I could cat(1) the log file to read it, but if it is long (most are) then it will wipe out my terminal’s history (or at least force me to scroll back for miles) so I lose my context. less(1) is really handy for interactively taking a peek inside a log file, scrolling around, searching for the next occurrence of something I’ve found, etc. But it is a curses program. Do you ever do this sort of thing? What do you use?

                                                                                              1. 1

                                                                                                I use less (or more). It’s not ideal, but I use it.

                                                                                                1. 1

                                                                                                  Damn, I was hoping you had a different solution that would work with 9term :)

                                                                                          2. 2

                                                                                            Which version of Sam do you use?

                                                                                            1. 1

                                                                                              The one in plan9port, or the one in 9front.

                                                                                            2. 1

                                                                                              That’s an interesting way to preserve history! Never thought about it.

                                                                                              Any tmux users? I usually use it, so I split horizontally to preserve history. A bit of a newschool solution ;)

                                                                                              The -4d and -2,5t. stuff is imo the most valuable take-away. I use it in vimlikes, but I would appreciate it in any other editor as well!