1. 1

    The rule of thumb I’ve usually used is “#num cpus * 2” for threads, to improve the chances that any thread which pauses allows other threads to fill in the CPU utilisation.

    Is that not necessary here because the go runtime is mapping goros to OS threads to explicitly soak up any pauses for I/O?

    1. 2

      Yes. When GOMAXPROCS=1 was the default, there was no parallelism but there was still plenty of concurrency.

      1. 1

        Sorry, I meant I was wondering why the new default was just “num cpus” not “num cpus * 2”, the latter being a rule of thumb to ensure that there are enough threads to hand to max the cpus.

        1. 4

          Rule of thumb where? In Java a lot of async code is really hiding the waiting in a thread somewhere. In Go and Erlang the runtime can actually do something else while waiting so more schedulars than cores can oversubscribe the system.

          1. 1

            Usually when this happens in java, it’s because you actually need to do work on a CPU. Presumably, for the same problem, you would still need to do that work on a CPU in Go or Erlang. I’m not sure why there would be a difference other than semantically (unless someone decided to have a thread just block on an async call, which just sounds like a bug to me).

            1. 2

              Usually when this happens in java, it’s because you actually need to do work on a CPU. Presumably, for the same problem, you would still need to do that work on a CPU in Go or Erlang.

              But, Go and Erlang are actually utilizing non-blocking I/O primitives, and so they can preempt a computation and switch back to the I/O task. Essentially, you should be able to peg the CPU completely with CPU bound computations while waiting for I/O.

              Java can utilize Non blocking I/O, but in my experience, code often doesn’t. It instead utilizes ThreadPools (Executors), does blocking I/O, and provides an async interface (Futures, Listeners, etc) to the work completing.

              1. 2

                I feel like most folks I’ve talked to who are doing serious work where they care about performance are using netty, which is designed from the ground up to be non-blocking.

                Perhaps I’m biased though, because I work on a library for the JVM that doesn’t do blocking I/O.

          2. 3

            My experience is that num CPUs + 2 results in better coverage. For more CPUs, *2 just results in a lot of thrashing.

        2. 1

          Have there been any tests on whether “#num cpus * 2” is better than “num cpus”? Is this from your personal experience? I admit it makes sense logically.

          1. 2

            I’ve seen similar results for one workload, but I’m not sure I agree with it as a rule of thumb in general. As with any configuration setting, you should run your own experiments to see what works for you.

        1. 3

          Comparable presentation: Go for Pythonistas

          The difficult thing for me with both of these is answering the question, when is the right time to choose Go or Rust or C++ instead of Python?

          1. 2

            I feel like choosing Rust or C++ instead of python should be done when you need more performance. Go if you need more performance and won’t mind generics missing and explicit (and very manual (but could be automated)) error handling. Go, Rust, or C++ if you need a compiled binary.

            I personally have pretty much replaced Python with Haskell, Scala, Clojure, and Racket.

          1. 5

            I really liked how this article explains the lazy evaluation step by step, first with graphical visualizations of the reduction graph and later by introducing the textual representation. I went through Real World Haskell which has as well an explication of lazy evaluation, but for me it stayed unclear until this article.

            Finally at the end of this article: “In fact, I would go so far as to say that with lazy evaluation, it is no longer feasible to trace evaluation in detail, except for very simple examples. Thus, analyzing the space usage of a Haskell program can be hard. My advice would be to only act when your program really does have a significant space leak, in which case I recommend profiling tools to find out what’s going on.”

            For me this is the ugly area of Haskell programming. On one hand Haskell is often promoted with “when it compiles it mostly works”, implying that you almost do not need testing, or at least not as extensively as in other languages, on the other hand it is really difficult to find out the space usage of Haskell programs without extensive testing. For me this is as crucial as space leaks in C, but is often dismissed, or should I say waived over, by the Haskell community. I really love Haskell, and I think we need better tools/methods to deterministically be able to predict the space usage of a Haskell program, without heavy profiling.

            1. 2

              Totally agree about deterministically predicting space usage. I’m very excited about Idris and it’s research towards being able to guarantee space usage via the type system.

              However, while Idris is still brewing and being experimented with I very much hope for tooling to be developed in the areas of space usage. If I can understand it myself, perhaps I’ll be able to contribute some tools!

            1. 7

              I wonder how applicable this is for Haskell. A nice quote I found in the “Yesod tutorial for newbies:”

              Weaknesses of Haskell certainly won’t be technical:

              • Hard to grasp Haskell
              • Hard to find a Haskell programmer
              • The Haskell community is smaller than the community for /.*/

              I would further say that another weakness is that way too many people outside the Haskell community don’t take it seriously. At some point in their career someone tried to explain what Haskell was all about and they interpreted that to mean that Haskell is esoteric and only for academic purposes, or at least it’s a language like Latin, where it improves you as a programmer, but you won’t find yourself writing any real Haskell code.

              So this rant ended up way longer than I wanted it to be… But now I have an idea for yet-another-blog-post-that-I-will-never-finish.

              TL;DR The point I wanted to make is that this article seems to try to point out that immutability and functional thinking is easier than mutable and procedural thinking. I’m curious as to how well you can point this out for Haskell, but I think too many people see the type system and everything else to do with it as scary and complex.

              1. 7

                Actually finding developers for Haskell teams isn’t super difficult, based on personal experience. In fact it might be easier than for most other ecosystems, and could be considered an advantage rather than a weakness. We’ve been hiring folks to work in our mix of Clojure + Haskell stack for the past half year and it’s been pretty reasonable.

                There are considerably fewer Haskell gigs out there than there are developers either familiar with Haskell or very willing to learn it asap. Many people would love to upgrade to working in the language, but cannot because their current full-time gigs will never move off of Java / Ruby / Python or whatever else they’re using. Most folks that apply for Haskell jobs are amateurs who haven’t used it in production before but would really want to, it’s again a reflection of the dearth of Haskell teams out there. Veeeeery rarely you get people with professional experience in it, but I’m confident this will slowly go up in the coming years.

                It’s clearly hard to find amazing Haskell developers, but that’s true for ecosystems out there.

                On an unrelated note: “Think less” is hardly applicable to Clojure. You can’t develop anything sizeable without having to use your brain as a 3rd party compiler to continuously keep track of what kind of types you’re passing around since they’re never explicitly called out in the code. Writing Clojure code at scale is actually quite high-effort compared to a few alternatives out there. Yes yes you can use schema / clojure.typed.

                1. 7

                  Haskell is a language I’d really like to work more with, but I always find myself struggling to get started with any project beyond fizzbuzz. I think that’s one of the things that pulled me to Clojure; I’d heard of and tried Haskell first, but jumping out of the gate with algebraic types and monads to make IO and state possible while enforcing purity, plus the symbol-heavy syntax, all while I was just learning to embrace immutable data structures and pure-functional programming for the first time, was just too much for me.

                  Clojure – as soon as I figured out to just use leiningen, that is – has the latter two, but the syntax and related concepts are markedly simpler. I built my first app without major problems, and here I am a few years later.

                  This doesn’t make Clojure better, by the way; in terms of writing stable, fast, “provably” correct software, I don’t think anything has Haskell and its ML brothers beat. Perhaps the difference can be illustrated by my first Clojure app: it was janky and did a lot of stateful and un-idiomatic things, but it still worked, dammit! Of course, my Clojure code is much better now, but if I hadn’t been able to write bad Clojure code and still make it go, Clojure probably would have ended up in the “someday” pile next to Haskell and I’d be writing Go or Rust or something (god forbid!).

                  1. 8

                    It’s true, for a lot of us Clojure was actually a gateway drug towards Haskell. I started there, @bitemyapp was also big in the community before moving on. My guys typically start on Clojure (in big part because we have a lot of legacy code written in it) and then eventually transition to Haskell, being able to reuse a huge subset of concepts, layering a lot of new ones on top. I haven’t tried putting someone straight in front of Haskell just yet, not for any particular reason, but that’d be an interesting experience. I’m getting closer to the point where I can decently teach someone how it works, whereas a few months ago I would have had a lot more trouble with it.

                    And yes, I think it’s reasonable to say that both Haskell the language and its current type system are more complex upfront than Clojure’s. You have to think much beyond maps, sets, lists and vecs, you have to deal with uncompromising purity and having to grok weaving monads before you can do anything really interesting. You save time and complexity over the long term, but it’s more to absorb when just starting out.

                    Btw Adam, we still heavily use your err->> monad macro at Front Row <3

                    1. 3

                      Btw Adam, we still heavily use your err->> monad macro at Front Row <3

                      Haha, I’m glad to hear it. My greatest contribution to open source!

                    2. 7

                      Hi! I came to Haskell from Clojure. I had trouble too!

                      This is my guide for learning Haskell. It can get you where to need to be in order to write real stuff in Haskell, but you have to do the exercises.

                      Ping me if you need help, there’s also an IRC channel devoted to people learning to become productive in Haskell on Freenode at #haskell-beginners.

                      Btw thanks for your posts and tutorials for Clojure, they helped me a lot back in the day. :)

                      1. 2

                        You’re super welcome!

                        I’m not actively learning Haskell atm, but I think it’s something I’ll feel the need to do someday. Thanks for the material!

                      2. 6

                        http://howistart.org/posts/haskell/1 (quick taste of a haskell project)

                        http://www.seas.upenn.edu/%7Ecis194/spring13/index.html (comprehensive in theory and application)

                        https://github.com/bitemyapp/learnhaskell (source of the above and where to go after)

                        http://yannesposito.com/Scratch/en/blog/Holy-Haskell-Starter/ (great intro to testing and nice tool to use)

                        http://www.amazon.com/Beginning-Haskell-Alejandro-Serrano-Mena/dp/1430262508/ (In depth about lazy evaluation, basics, exposure to popular libraries, and more… everything you could want)

                        After getting past those, hopefully you’ve written substantial code. My first big project was 1000 lines of Haskell code in continuation passing style. Despite that, it was very nice to work with because of Haskell’s other pluses.

                        I saw this video a short time ago: http://vimeo.com/73648150

                        After watching the above (and consulting the Haskell wikibook) replaced it with a 224 line State Monad Transformer and then the actual application was about 150 lines.

                        Have fun, and please provide feedback!

                  1. 2

                    “They all date from before 2013. That’s how we know the NSA wasn’t involved.”

                    This is quite a silly assertion that is 100% untrue. The NSA has existed pre-2013 and there is documentation of their activity. At the very least your closing sentence is disingenuous.

                    1. 12

                      He was obviously joking.

                      1. 2

                        I had just woken up, give me a break :)

                        1. 2

                          True, but he does come off as incredibly arrogant in the process.

                        2. 3

                          If we posit that the NSA introduced the Apple bug, then logic demands we look back at the past ten years' worth of bugs and assume they are sabotage as well. There are a lot of unburnt witches running free.

                          It’s a response to the people demanding Apple identify and fire the person responsible.

                          1. 1

                            I would argue that logic demands we look back at the past ten years' worth of bugs of companies who were as directly involved with the NSA as apple was, not all bugs. What good would Jill OSSdev’s bug data be?

                            1. 3

                              If anybody is being insufficiently paranoid, I think it’s the people who think the NSA restricted their activities to the set of companies whose logos fit on a PowerPoint slide.

                          2. 1

                            After reading the tarsnap note, I thought he was making a joke.

                          1. 1

                            Intriguing, if cryptic. But from where I sit, this is just a project named “Atom” hosted on Github. Is there any more information on it, something that connects it to Github besides the Github-employed members?

                            1. 2

                              Answering myself: https://github.com/atom/welcome/blob/master/lib/welcome.md

                              You are one of the first people to use GitHub’s new text editor!

                              1. 1

                                What is the non-mac equivalent of cmd-shift-P?

                                1. 1

                                  ctrl+shift+p

                                2. 1

                                  Wow, try clicking on the docs link. It takes you to a GitHub app auth page for your email.

                                3. 2

                                  Also worth noting:

                                  $ whois atom.io
                                  
                                  Domain : atom.io
                                  Status : Client Updt+Delt Lock
                                  Owner  : GitHub Hostmaster
                                  Owner  : GitHub, Inc.
                                  Owner  : 88 Colin P Kelly Jr St
                                  Owner  : San Francisco
                                  Owner  : CA
                                  Owner  : US
                                  
                                1. 5

                                  I’m writing a simple REST app in Haskell that is a test for a company I’m interviewing with.

                                  I’m also swapping out an in-house oauth integration system in python that worked with a small set of oauth services with python-social-auth that offers integration with a large set of oauth services. Other advantages include free quality testing from other users and free development from the open source community. As well as the awesome chance to give back if I find bugs.

                                  I’m also having a conversation about being a partner in a startup. I’m not sure how that will go though, since it’s an equity up front and get paid after we start making money type offer. I’m also continuing to learn Haskell and have been reading source code of GHC standard libraries as well as on the implementation of Snap.

                                  I was sent a review copy of an ebook on Snap and I’m dying to start reading/proofreading it. I believe Haskell has a great future in web development. I hope I can help that along by improving any and all learning resources.

                                  1. 1

                                    I love these thoughts, and I try to reason about my own code while keeping them in mind, but it is hard to argue with a very simple truth: It is much harder to write functional programming applications. Harder than mitigating the outages and scaling problems of imperative applications. In terms of building applications that people actually use, it is generally easier to find a junior developer than on that can adhere to no-local-state standards.

                                    1. 2

                                      I disagree that writing functional programming applications is inherently harder. I agree that it is easier to use your imperative programming base to fix buggy imperative programs than it is to convert them to functional programs.

                                      1. 2

                                        it is hard to argue with a very simple truth: It is much harder to write functional programming applications. Harder than mitigating the outages and scaling problems of imperative applications.

                                        Can you be more specific? In my experience, I have not found functional vs imperative to have be a huge distinction in regards to this. Problems are hard no matter what tool you use to solve them in. Mostly, I have just found FP providing a model for making dealing with bugs easier, but it depends on the developer to follow that model, which many don’t.

                                      1. 1

                                        I am definitely going to have to try this out. I’ve been using an older version of Haskell mode that doesn’t seem to be cabal sandbox aware. On the plus side I’ve gotten pretty quick at opening a new split, opening shell, typing “cabal repl” but this should be much more streamlined.

                                        Has structured-haskell-mode stablized? Last I looked there were warnings about data loss. That’s not a huge issue since I’m usually using version control, but it’s definitely something I’d want to be aware of.