Threads for st3fan

  1. 3

    But no periodic tasks with a scheduler that is part of the platform.

    1. 11

      Second place I’ve seen this brought up. I agree, we should do something like this.

      1. 4

        The Heroku one was a light rails app over a Postgres database. Worked great! A few people on my team tried to replace it. That didn’t go well! So, it’s probably still a rails app + postgres.

    1. 2

      Makes me think of the stamps in Magic Cap.

      1. 2

        Same artist. There is a great documentary about General Magic. Highly recommended to watch if you like that kind of history.

      1. 3

        Disabling retweets is amazing. Your feed turns from echo chamber to mostly original content.

        1. 1

          Yes, there are also a few more tricks to reduce spam, but I think mostly manual.

        1. 9

          Lots of bogus excuses for a lot of busy work. Sorry I know that is not constructive but I just could not find a single business reason in this post that would justify this migration.

          1. 3

            Well, if the business case was “we had problems with Linux and we’re (pretty) sure that how FreeBSD is done by one team those will not pop up” that’s a massive business reason.

            A few years ago there were lots of people running their webservers on FreeBSD. The difference is just very small (if your stack is available or easily buildable) and so the business case can be very small difference of speed, maintainability, knowledge of the team.

            But yeah, the article doesn’t really persuade me either.

            1. 8

              Learning how to manage Linux systems is probably a lot cheaper than migrating to another OS if the reasons are mostly opinionated and superficial.

            1. 6

              A quick web search turned up this as well, if it’s the same person….

              https://gist.github.com/lclarkmichalek/716164

              1. 1

                Given that the owner of the colors.js repository replied to that Gist with a thank you, I think we can call it confirmed it is the same person. I don’t know about the Reuters article but it looks conceivable it was the same. Even if the latter turns out to be a different chap of the same name it seems like the FOSS world will be better off without relying on code from such a loose cannon.

            1. 3

              This is probably the biggest RCE of the past decade. We’re going to see unpatched servers for years.

              1. 6

                I hope that is not your password on that sticky note

                1. 41

                  I doubt it is. It’s hunter2 base64 encoded.

                  ➜ echo "aHVudGVyMgo=" | base64 --decode
                  hunter2
                  
                  
                  1. 28

                    I only see *******, does lobste.rs hide passwords if you put them in posts? That’s a neat feature. Here’s mine to test: *******.

                    1. 8

                      <DavidDiamond> Here’s mine to test: *******.
                      thats what I see

                      1. 3

                        hunter3

                        1. 3

                          OH NO!

                1. 4

                  Looks fantastic. I love the addition of a real learning path and syllabus. The online code editor and tester also works great. I’ve been doing Elixir exercises and it is just so nice to do it all in a browser without needing any local setup. The code editor could be better / smarter but maybe that is for a next iteration.

                  1. 1

                    More Elixir and Phoenix! One of my personal goals for this year was to learn how to build web applications. I picked Elixir, Phoenix (with LiveView) and Tailwind for my “stack”. I’ve been following the http://pragmaticstudio.com course on LiveView which has been great.

                    1. 1

                      http://pragmaticstudio.com May I know what kind of personal project is? I love learning new technology, but sometimes not excited if build the same thing again (blog/crud/etc.. ) , especially when it doesn’t show the real intention of that particular technology.

                    1. 1

                      Working on shipping a long due update for Firefox Focus. And in my personal time I’m learning some more Elixir and Phoenix & LiveView.

                      1. 2

                        I’m exploring Elixir which seems like a really fun language.

                        1. 1

                          Your server is not private if it has a public IP.

                          I really wish infrastructure providers had the option to not assign a public IP at all at instance creation and only hook it up in a private network.

                          1. 2

                            Prime example of why you should never pipe curl output straight into bash.

                            I’m really surprised this is not a top story here or HN .. millions of projects are potentially compromised or have at least their CI secrets / API keys stolen.

                            1. 2

                              This was my first response as well, but in a way it represents a larger problem of auto-updating/live dependencies.

                              From what I’ve seen in the past, the standard “alternative” to piping curl to the shell is to download the file, “read” it, and then execute it. [1] [2] [3] That wouldn’t’ve helped here. Even if a thoughtful user had downloaded the file and run it in two separate steps, they’d still be vulnerable.

                              The “actual” solution would be to download, verify the checksum against something, and then execute the script. That an issue as well: where do you get the checksum from? For the codecov script, they post the checksums in the GitHub repo which might be a safe place. But if you’re downloading the file from a remote source it’s technically possible that file has also been compromised.

                              If you instead download the file once, “pinning” it to a version: what if you’ve downloaded the file during the period when it was malicious. Now you’ve pinned a malicious version.

                              There’s no easy solution here, and while I agree that curl | bash is a bad idea for many reasons, that isn’t the root issue here. This is a reminder that everything should be considered malicious, but little more than that.

                              1. 3

                                Even if a thoughtful user had downloaded the file and run it in two separate steps, they’d still be vulnerable.

                                This is only true if they had downloaded it from the compromise window of February-March. The smart thing to do is download it once and drop it in your repo; that makes it trivial to go back later and see whether any given build was compromised. If people had done this instead of following codecov’s advice, then only projects which added codecov during that window would have been compromised, rather than literally every project which used codecov during the window.

                                No one is saying that “avoiding curl|bash means you’re completely safe”; it’s all about limiting the blast radius of an intrusion.

                                1. 2

                                  The thing is that a lot of people focus on the “piping curl to bash”-part; which is what the top comment did as well (“you should never pipe curl output straight into bash”). There is little difference between piping directly to bash or saving it first, or between that bash script and some library you added from your language package tool.

                                  The problem in this specific case – which is actually not very common because most curl | bash things are one-time installers – is that you’re always directly loading the code from a remote source, possibly dozens of times per day.

                                  A good solution would be to limit the access of such things; all it needs to do is read a file and upload it; there’s no need for it to access anything else on the filesystem, or even access the environment beyond a few basics. All of this is possible already I think, but not very easy. Certainly not as easy as something like:

                                  run-limit --deny-all \
                                      --allow-fs coverage.profile --allow-http api.codecov.com \
                                      --allow-env PATH --allow-env CODECOV_KEY \
                                      'curl codecov.com/sh | bash'
                                  
                            1. 2

                              The cult of objects and abstractions and patterns everywhere.

                              All you need is an enum like type and a function to convert input to that type. Maybe in some insanely more complicated example you could justify the factories and objects and visitors and what not but this code the question was asked about can be made type safe fool proof and no repetitive with a basic type and a function.

                              He advises against an enum in the article. I don’t really understand why. Maybe it is too simple. Too pragmatic.

                              1. 1

                                He says why right in the article. The assumption is that you will be conditioning real behaviournon this, probably in many places in the system. If you’re not, you don’t even need the enum and a string will do, if you are then the enum will lead to repeated cases all over the system vs a set of objects encapsulsting that complexity.

                              1. 2

                                Does anyone here use straight? I never got the advantage, compared to just cloning the repositories yourself (or as submodules). Plus you have to run random code from a github repository.

                                1. 2

                                  You run random code anyway. Even without straight? Or do you audit all the packages and their dependencies before you use them?

                                  1. 2

                                    I minimize the number of packages I use and usually skim through the source code. Automatically downloading and evaluating code from a repository seems more dangerous, but I do get your point.

                                  2. 2

                                    No I haven’t felt the need.

                                    I (ab)used Cask for a long time because it was a nice declarative way of configuring packages in ~/.emacs.d. Since I’ve returned back to using package.el and the fact that it handles dependencies much better now, I just rely on it maintaining package-selected-packages and if needed I can just call (package-install-selected-packages) but I don’t have more sophisticated needs.

                                  1. 4

                                    You can simplify this with http.Handle("/", http.FileServer(content))

                                    1. 9

                                      I don’t really like Go, and have only done like …. 5 pages of it ever. But I feel like that goroutine example is pretty convincing that Go is easy? The amount of futzing about I have to do in Python to get a similar-working pipeline would lead to maybe 3x as much code for it to be that clean.

                                      I dunno, Go reintroduces C’s abstraction ceiling problem meaning it’s hard for someone to show up and offer nicer wrapper for common patterns. But if you’re operating at the “message passing” and task spinup level, you’re gonna have to do flow control, and that code looks very nice (and local, which is a bit of a godsend in this kind of stuff).

                                      Though I feel you for the list examples. Rust (which, granted, has more obstacles to doing this) also involves a lot of messing around when trying to do pretty simple things. At least it’s not C++ iterators I guess

                                      1. 18
                                        import asyncio
                                        
                                        async def do_work(semaphore, id):
                                            async with semaphore:
                                                # do work
                                                await asyncio.sleep(1)
                                                print(id)
                                        
                                        async def run():
                                            semaphore = asyncio.Semaphore(3)
                                            jobs = []
                                            for x in range(20):
                                                jobs.append(do_work(semaphore, x))
                                            await asyncio.gather(*jobs)
                                            print("done")
                                        
                                        asyncio.run(run())
                                        

                                        About the same length in lines, but IMO quite a bit easier to write.

                                        1. 4

                                          That is a good example but you can’t really compare asyncio and go routines. The latter are more like “mini threads” and don’t need to inherit all the “ceremony” that asyncio needs to prevent locking up the io loop.

                                          1. 12

                                            Goroutines are arguably worse IMO. They can be both run on the same and a different thread, which makes you think about the implications of both. But here’s an a bit more wordy solution with regular Python threads which can be more comparable:

                                            import threading
                                            import time
                                            
                                            def do_work(semaphore, id):
                                                with semaphore:
                                                    # do work
                                                    time.sleep(1)
                                                    print(id)
                                            
                                            def run():
                                                semaphore = threading.Semaphore(3)
                                                threads = []
                                                for x in range(20):
                                                    thread = threading.Thread(target=do_work, args=(semaphore, x))
                                                    thread.start()
                                                    threads.append(thread)
                                                for thread in threads:
                                                    thread.join()
                                                print("done")
                                            
                                            run()
                                            

                                            As you can see, not much has changed in the semantics, just the wording changed and I had to manually join the threads due to a lack of a helper in the standard library. I could probably easily modify this to work with the multiprocessing library as well, but I’m not gonna bother.

                                            Edit: I did bother. It was way too easy.

                                            import multiprocessing as mp
                                            import time
                                            
                                            def do_work(semaphore, id):
                                                with semaphore:
                                                    # do work
                                                    time.sleep(1)
                                                    print(id)
                                            
                                            def run():
                                                semaphore = mp.Semaphore(3)
                                                processes = []
                                                for x in range(20):
                                                    process = mp.Process(target=do_work, args=(semaphore, x))
                                                    process.start()
                                                    processes.append(process)
                                                for process in processes:
                                                    process.join()
                                                print("done")
                                            
                                            run()
                                            
                                            1. 10

                                              goroutines aren’t just good for async I/O. They also work well for parallelism.

                                              Python’s multiprocessing module only works well for parallelism is basic cases. I’ve written a lot of Python and a lot of Go. When it comes to writing parallel programs, Go and Python are in different categories.

                                              1. 4

                                                It’s best to decide which one do you actually want. If you’ll try to reap benefits of both event loops and thread parallelism, you’ll have to deal with the disadvantages of both. Generally, you should be able to reason about this and separate those concerns into separate tasks to be able to separate your concerns. Python has decent support for that, with asyncio supporting running functions in threadpool or processpool executors.

                                                I do agree though that Python isn’t the best at parallelism, because it carries quite a lot of historical baggage. When it’s threading was being designed in 1998, computers with multiple CPU’s where rare, and the first multi-core CPU was still 3 years away[1], with consumer multi-core CPU’s arriving 7 years later. The intention for it was to allow multiple tasks to run seemingly concurrently for the programmer on a single CPU and speed up IO operations. At the time, the common opinion was that most of the things will continue to have only a single core, so the threading module was designed appropriately for the language, with a GIL, giving it safety that it will not corrupt the memory. Sadly the things didn’t turn out how they initially thought they would, and now we have a GIL problem on our hands that is very difficult to solve. It’s not unlike the errno in C, which now requires macro hacks to correctly work between threads. Just that GIL touches things that are a bit harder to hack over.

                                                1. 7

                                                  I’m aware of the history. My point is that the Python code you’ve presented is not a great comparison point because it’s comparing apples and oranges in a substantial way. In the Go program, “do work” might be a CPU-bound task that utilizes shared mutable memory and synchronizes with other goroutines. If you try that in Python, it’s likely you’re going to have a bad time.

                                                  1. 3

                                                    The example with multiprocessing module works just fine for CPU tasks. asyncio works great for synchronization and sharing memory. You just mix and match depending on your problem. It is quite easy to deffer CPU heavy or blocking IO tasks to an appropriate executor with asyncio. It forces you to better separate your code. And in this way, you only need to deal with one type of concurrency at a time. Goroutines mashes them together, leaving you to deal with thread problems where coroutines would have worked just fine, and coroutine problems, where threads would have worked just fine. In go you only have a flathead scredriwer for everything betweem nails and crosshead screws. It surely works, sometimes even well. But you have to deal with warts of trying to do everything with one tool. On the other hand, Python tries to give you a tool for most situations.

                                                    1. 6

                                                      The example with multiprocessing module works just fine for CPU tasks.

                                                      But not when you want to add synchronization on shared mutable memory. That’s my only point. You keep trying to suck me into some larger abstract conversation about flat-head screwdrivers, but that’s not my point. My point is that your example comparison is a bad one.

                                                      1. 3

                                                        Give me an example of a task of that nature that cannot be solved using multiprocessing and asyncio and I’ll show you how to solve it. You shouldn’t try to use single tool for everything - every job has it’s tools, and you might need more than one to do it well.

                                                        1. 4

                                                          I did. Parallelism with synchronized shared writable memory is specifically problematic for multiprocessing. If you now also need to combine it with asyncio, then the simplicity of your code goes away. But Go code remains simple.

                                                          You shouldn’t try to use single tool for everything

                                                          If you think I need to hear this, then I think this conversation is probably over.

                                                          1. 2

                                                            Parallelism with synchronized shared writable memory

                                                            You describe a class of problems. But I cannot solve a class of problems without knowing at least one concrete problem from the class. And I do not.

                                                            1. 3

                                                              Here’s an example of something I was trying to do yesterday:

                                                              I wanted to use multiprocessing to have multiple workers pull (CPU-bound) tasks off a (shared) priority queue, process each task in a way that generates zero or more new tasks (with priorities) and put them back on the queue.

                                                              multiprocessing.Manager has a shared Queue class, but not a shared priority queue, and I couldn’t figure out a way to make it work, and eventually I gave up. (I tried using heapq with a shared multiprocessing.list and that didn’t work.)

                                                              If you can tell me how to solve this, I would actually be pretty grateful.

                                                              1. 1

                                                                I gave it a bit of time today, here’s the result. Works decently well, if you don’t do CPU expensive stuff (like printing big numbers) in the main process and your jobs aren’t very short.

                                      1. 2

                                        I think Cocoa Emacs may need a bigger rewrite to become future proof. It will probably survive macOS 11 but the Cocoa integration code really is a huge complicated 20? year old hair ball.

                                        1. 2

                                          Would be interesting to see if the new pgtk (pure gtk) integration works, since gtk3 does support macOS. It’s still WIP of course, and I’m not saying it should replace cocoa, but still…

                                          1. 2

                                            As a macos user, there is nothing about gtk3 or gtk4 that would make me want it to be used in gui emacs on macos.

                                            Why would I trust a work in progress over something that actually uses the platform? Why is macos special in this regard and windows being left out? Why not just update cocoa emacs instead of make emacs an even worse experience on macos? Will gtk improve the spawned process speed on macos?

                                        1. 13
                                          1. 7

                                            No, seems like it’s still on the homepage & in the source code as of 3.0.0.

                                            1. 6

                                              And thereby possibly violating the GDPR of those are in the EU. But any discussions about this are immediately censored in the Homebrew project.

                                              1. 2

                                                Can confirm, as a project they don’t like being told their approach isn’t fantastic.

                                                1. 2

                                                  As someone who has been on the receiving end of many discussions like that (specifically, Firefox for Android aka Fenix), I can say the following:

                                                  An open source project and their maintainers can make decisions about how they wish t run and implement the project.

                                                  People can file all the bugs and re-open all the discussions for things that they do not agree with, but at one point they will have to accept that a decision has been made by the people running the project.

                                                  What I see over and over again are the same responses:

                                                  • they do not listen
                                                  • they do care
                                                  • they censure me
                                                  • they don’t deal well with criticism
                                                  • i will leave if you don’t do what I want
                                                  • I will fork the project
                                                  • the project is doomed

                                                  As someone who has been on the receiving end of this many many times I can tell you that it is extremely tiring and unproductive. As a maintainer you simply cannot satisfy all requests from all users. For all kinds of reasons. Sometimes it is as simple as “we like the way this works”.

                                                  So dead horses will be beaten and discussion threads will be closed. Sometimes no is no. It is that simple. How do you reach consensus? Well often you do. Sometimes you don’t.

                                                  What makes it even more complicated is that the user base is usually also fully split. I can guarantee you that in this case, sending anonymous usage to google analytics, many people could not care less and probably don’t bother to disable it even after reading the note that is shown about it.

                                                  So I ask people here, as a maintainer of a project, what do you do? Do you submit yourself to a small group of very vocal and strongly opinionated users? Or do you just stick with what you and fellow maintainers think is the best way to go forward?

                                                  (What I find really interesting is that in communities like this, Lobsters, there is often a lot of agreement about how bad it is that people ask for time or free support from open source projects. It is often quickly labeled as unacceptable entitlement. How are issues like https://github.com/Homebrew/brew/issues/142 different I ask you?)

                                                  1. 3

                                                    As a maintainer you simply cannot satisfy all requests from all users.

                                                    Nobody asked them to satisfy all requests from all users.

                                                    I’d imagine I’m pretty safe to say that no single user anywhere asked them to add analytics, using one of the most privacy invasive companies on the planet.

                                                    Debian, as usual, shows how to do this right:

                                                    (a) prompt the user on-install/first-run, if they’re happy to provide anonymous usage information. Provide links to more information if necessary.

                                                    (b) don’t rely on a privacy abusing mega-corp to provide the backend for said usage tracking.

                                                    1. 1

                                                      … shows how to do this right

                                                      This is an opinion. And exactly why these discussions are so difficult.

                                                2. 1

                                                  Can you explain how it violates the GDPR?

                                                  1. 3

                                                    You need informed consent before you send any identifiable data, and most data points are identifiable.

                                                    1. 1

                                                      I am not a lawyer but I am pretty sure that you don’t need consent for anonymous analytics since there is no connection to an person.

                                                      If you think the data that Homebrew sends is indeed PII (Personally Identifiable Information) then you should definitely raise that with them. But I highly doubt this is the case since Homebrew does not know who you are.

                                                      1. 2

                                                        I’m not a lawyer either, but e.g. cookies require consent. Obviously a shell doesn’t have cookies, but Home-brew essentially re-creates them, by generating a UUID (and storing it, only generating a new one if it can’t read an existing one) and sending it with the calls to GA.

                                                        If you accept that a cookie is a little piece of text on your computer that can be used to uniquely identify you, I’m not sure how “a UUID strode on your computer that can be used to uniquely identify you” wouldn’t be considered the same thing.

                                                        1. 2

                                                          The nuance here is “identify you”. Again I am not a lawyer but a random session or install ID is not PII and not something that identifies you. If it is connected to for example an email address then it would be.

                                                          Home brew has no relationship with its users. It does not know who those are or how to identify them. If you would send a GDPR request to them to ask for your data then they cannot send you anything because they cannot uniquely identify you. Only when you give them that UUID and tell them, that’s me, they can. But at that point you have made the connection.

                                                          1. 2

                                                            As I understand it (also not a lawyer) - because the data can later be combined with another source to identify the customer (that is, ‘this uuid is that persons laptop’), it’s covered.

                                                            The GDPR is not written to allow clever hacks to get around it.

                                                            On the bright side - violators can expect formal letters of warning at no cost. If you persistently ignore those for a year or so you’re likely to get a really damn big fine, but it’s not like it’ll come as a surprise.

                                              1. 1

                                                Reeder.app on iOS and macOS. Wonderful app.