1. 1

    Druid

    I think this deserve a word here. I worked with it around 2014-2016. Now the project is passed to apache.

    This was like a “secret weapon” for our team at that time. I worked on make a social media analytics (think a dashboard for twitter/FB etc…)

    With that we were able to provide analysis in real time about a ridiculously big amount of data. It was very hard to configure correctly. But once done, it was incredible. You could click on a button and hundreds of aggregations were processed about ten of millions big json object in less than 200ms. That was a really cool tool.

    Here is a short presentation about it: http://yogsototh.github.io/mkdocs/druid/druid.reveal.html#/

    1. 1

      Very cool I was thinking about building something similar

    1. 1

      I use org-publish + a few scripts to generate an RSS feed and optimize the size of the website by taking advantage of a full HTML+CSS minimizer. I previously used hakyll, and nanoc. I tried to have a CSS that look like a markdown in a terminal.

      my blog: https://her.esy.fun

      1. 2

        Having tried just about all kinds of static site generators under the sun—from the more mainstream ones like Jekyll and Hugo to more exotic ones like ssg, ox-hugo, org-page, Org publish, org-static-blog, Haunt, and even a custom one written in Haskell—I’m now back to hand-written HTML files + SSI rules (for simple templating), and love the simplicity. The only thing missing right now is an Atom feed. I wonder if I could use GNU M4 for that, like @technomancy does.

        Result at https://bandali.eu.org, “sources” at https://git.bandali.eu.org/site.

        1. 2

          If your page structure is more or less consistent (looks like it is), you can extract metadata from pages with an HTML parser and generate a feed from it. That approach allows some things that are impossible or unwieldy in the traditional paradigm, such as using an arbitrary paragraph for post excerpt, not the first one.

          My own generator automates that process: the blog index of soupault.neocities.org/blog is produced by this config. It can dump exported metadata to JSON, which isn’t hard to produce Atom from. I’m still to complete the JSONFeed/Atom generator script good for public release.

          That said, making a custom page to Atom script using an HTML parsing library that supports querying the data with CSS selectors (BeautifulSoup, lambdasoup etc.) isn’t that hard if making it work for anyone else’s site isn’t a goal.

          1. 1

            Indeed; that’s one of the approaches I’m considering. Also, soupalt seems quite interesting, thanks for the links; I’ll check it out!

          2. 1

            I also missed a tool to generate RSS from a tree of HTML files. This is how I generate my RSS with a very basic shell script using html-xml-utils:

            https://her.esy.fun/posts/rss-gen.html

            1. 1

              Nice, thanks for sharing!

            2. 1

              Any particular reason you switched back to handwriting html files?

              1. 2

                Yeah, a few, actually:

                • I’d like my site setup to be very lightweight, both on the server and on my machine when editing, and use tools that are nearly universally available on all GNU/Linux systems. This rules out things like my custom static site generator written Haskell, since Haskell and its ecosystem are arguably quite the opposite of my two criteria above.

                • I’d like to have convenient complete control over the generated HTML output, and this is rather hard to come by in most existing static site generators and rules out most markup formats, since almost none are as expressive and flexible as HTML.

                Aside from the repetitive pre/postamble bits in each file, I find writing plain HTML quite alright, actually. The annoyance of HTML’s lack of built-in templating and simple conditionals can be solved by using SSI directives, which are fairly widely supported across web servers. Alternatively, I’m considering using GNU M4 in place of SSI, if it results in a simpler and cleaner setup. And it fits the two criteria in my first point above too.

                1. 3

                  These are rather valid arguments. I used to write in plain html too, it’s perfectly fine, especially with an editor’s auto-completion like emmet. Nevertheless, I now mostly write in markdown and enjoy it. Whenever I need something more complex, I just embed html.

            1. 1

              My current main website is: https://her.esy.fun

              Interesting aspects:

              • org-mode instead of markdown (it’s way better IMHO)
              • use org-publish with a bit of magic but not too much (more infos here
              • different CSS choices (all support light/dark theme, I’m particularly fond of the sci dark one when there are images)
              • no tracker of any sort
              • no js (except for pages displaying formula with Mathjax)
              • RSS is generated via a self made shell script (I was quite surprised no tool like that existed before more info here
              • Optimize the size by using all CSS and HTML, surprisingly it was very efficient (~30% better than classics minimizers) even using quite naive approach more info here

              My older one was https://yannesposito.com. I used hakyll for this one.

              The hardest part is always to produce content, and not to lose too much time optimizing its blog tech. Still I love doing that time to time :).

                1. 36

                  I use fish on macOS. There’s occasional headaches due to its lack of bash compatibility, but I find the ergonomics of using it to be much nicer than bash, zsh, or any other bash-compatible shell I’ve tried.

                  1. 2

                    As you I use fish for my shell (with oh-my-fish).

                    For short/quick scripts I use zsh generally using a nix-shell bang pattern like this:

                    #!/usr/bin/env nix-shell
                    #!nix-shell -i zsh
                    #!nix-shell -I nixpkgs="https://github.com/NixOS/nixpkgs/archive/19.09.tar.gz"
                    #!nix-shell -p minify
                    
                    minify $1 > $2
                    

                    And when I’m serious about a script that I switch to turtle.

                    1. 2

                      I find autocomplete and highlighting in fish are amazing compared to any other shell I’ve used

                      1. 0

                        Sell me zsh. A few friends use it but they’ve not been able to clearly convey its advantages over ye olde bash that is just… everywhere.

                        1. 4

                          auto completion works even if you don’t type the string from the beginning. i.e. you have a series of folders

                          /backup_images

                          /backup_files

                          /backup_videos

                          you can do an ls and start typing videos and it will tab-autocomplete to the right folder. It was the one feature that signaled to me, you made a good decision to switch.

                          1. 2

                            If bash is good enough for you then you probably have no reason to switch.

                            In the past I used zsh for some of its fancier features: I could do more expressive expansions to make my prompt pretty or do clever directory chomping; I found its completion much faster than bash, but that’s only useful if you use flag or subcommand completion; menu completion can be useful. I think also the history settings in zsh were or are more featureful than bash, but I’m not entirely sure what bash’s current features are like.

                            I’ve been using bash just fine for the last ~10 years on personal systems but I do have zsh on some servers so that I can do more clever things with prompting.

                            1. 2

                              The main reason I used zsh was to handle correctly files with special chars in them ([ \t\n] for example). It also has “real” lists and associative arrays. Mainly it was often better to write scripts. After I also found that print in zsh is often better than echo. For example print -P "%Bsomething bold%b". Also things like ${fic:t} instead of basename $fic, ${fic:s/x/_/} instead of echo $fic | sed 's/x/_/' and a lot of small niceties.

                              I no longer use zsh as my main shell, I switched to fish. Still I always preferred zsh over bash. But it was a long time ago, perhaps bash is better now.

                              I use fish for basic usage (completion is great), but when I script I generally use zsh.

                              1. 1

                                I’d start with the zsh-lovers document.

                              1. 5

                                As someone who has occasionally played with Haskell for years and is finally considering to use it for larger projects, this post concerns me. The complexity of monad stacks is a little scary, but I figure the type system makes it manageable. However, if it’s true that monad transformers end up being a source of memory leaks then I’m back to thinking Haskell should only be used for larger, production-level projects by those knowledgeable of GHC internals and edge-case language tricks to hack around inherent problems.

                                Can someone with experience comment on the author’s claims? They do seem weak when no specific examples of memory leaks (or abstraction leaks) are provided.

                                1. 4

                                  Do not use StateT or WriterT for a long running computation. Using ReaderT Context IO is safe. You can stash an IORef or two in your Context.

                                  Every custom Monad (or Applicative) should address a concern. For example a web request handler should provide some means for logging, to dissect request, query domain model and prepare response. Clearly a case for ReaderT Env IO.

                                  Form data parser should only access form definition and form data and since it’s short lived, it can be simplified greatly with use of ReaderT Form stacked with StateT FormData. And so on.

                                  https://www.fpcomplete.com/blog/2017/06/readert-design-pattern

                                  1. 3

                                    Yes it is know to never use RWST+ with a Writer Monad in the stack because of space leak.

                                    The choices about big-code organisation in Haskell is large. You have:

                                    • run like in imperative language, everything in IO
                                    • split using the Handler pattern https://jaspervdj.be/posts/2018-03-08-handle-pattern.html
                                    • use the MTL, in that case you should not use WriterT (if I remember correctly)
                                    • use the ReaderT Context IO Pattern
                                    • use Free Monads, the paint is still fresh here apparently.

                                    I used MTL-style to make a bot with long living states and logs (using https://hackage.haskell.org/package/logging-effect). It works perfectly fine for many days (weeks ?) without any space leak.

                                    I now started to go toward the simpler route of the Handler Pattern I pointed out. And, in the end, I tend to prefer that style that is very slightly more manual but more explicit.

                                  1. 39

                                    I’m really burning out on “simplicity” posts. I get it, simplicity is good. But that doesn’t actually inform me as a developer. Why do things become complex? What kinds of simplicity are there? How do we detect simplicity? How do we know when we shouldn’t simplify? None of these posts ever answer that.

                                    It’s like if I stood on stage and said “Be good! Don’t be evil! Being evil is bad!” Sure, everybody agrees with that, but does it actually help people make moral choices?

                                    (Also the analogy is dumb. Yes, we should totally base our engineering practice on a movie! A movie where the engineers are wrong because of magic.)

                                    1. 11

                                      Why do things become complex? What kinds of simplicity are there? How do we detect simplicity? How do we know when we shouldn’t simplify? None of these posts ever answer that.

                                      Because your questions are difficult and answers are dependent on a lot of factors.

                                      I’ll tell you what I do to detect simplicity, maybe you’ll find it useful. Let’s start with a real-life example.

                                      I needed tokens for authorization, I reviewed existing formats, JWTs look conservative and Macaroons look powerful.

                                      What do I do? I dissect the formats. For JWTs I read the RFCs and implemented software to create them and verify them (each in 2 languages) for various options (key algorithms).

                                      For Macaroons I read the whitepaper, then implemented verifier based on the whitepaper, reviewed existing implementations, found out differences between the whitepaper and de-facto code with explanations. While comparing my implementation I found out some security issues with existing code. Additionally I implemented the rest of the stack (de/serialization, UI for manipulation of Macaroons). After two months I knew precisely where does complexity lie in Macaroons and of course there are the only spots all blogposts don’t mention (spoilers: cycles in third party caveats, no standards for encoded caveats…)!

                                      Then I looked at my JWT proof-of-concept code - it uses base64(url) and JSON, primitives that basically all programming environments have built-in. After limiting the algorithms used the entire verifier takes just a couple of lines of code! It’s vastly simpler than the Macaroon one.

                                      What’s the moral here? That you need a lot of time to see for yourself what is simple and what is complex. Now every time I see a post recommending Macaroons I can already see the author didn’t use them in practice (compare that with the Tess Rinearson post linked at the end of that article).

                                      That’s only the example, I routinely implement various protocols and re-implement software (ActivityPub, Mailing Lists, roughtime client) and each time I discover what’s simple or what’s complex in each one of them.

                                      (By the way your book is excellent!)

                                      1. 9

                                        I get it, simplicity is good.

                                        Alas, not everybody gets it. The best that these kinds of exhortations can do (all that they aim to do, as far as I can tell) is to persuade people to modify their own set of values. This doesn’t immediately result in better code… but I think it’s a necessary precondition. The only developers who will even ask the good questions you suggest (let alone look for good answers) are the developers who hold simplicity as a value.

                                        (The analogy is pretty dumb though, and not especially motivating.)

                                        1. 10

                                          I’ve never met a developer who does not claim to hold simplicity as a value. But as a concept it is so subjective that this is meaningless. It’s extremely common for two developers arguing for opposing approaches each to claim that their approach is the simpler one.

                                          1. 7

                                            I get the value of exhortations. I think more examples would be better. Pairs of solutions where the simple one meets requirements with a number of better attributes. Developers often prefer to see the difference and benefits instead of being told.

                                          2. 6

                                            Exactly. This is one of those things you can’t explain in a book. When to compose, when to decompose. When to extract methods, when to inline methods. When to add a layer of abstraction, when to remove one. When is it too flexible, when is it too simplistic?

                                            No amount of rules of thumb is going to answer those question. I only know of one way to learn it: practice. Which takes effort and most importantly, time. Rendering this kind of posts mostly useless.

                                            1. 3

                                              P.S. They do feel good to write though, so people will keep writing them, and there’s nothing wrong with it either.

                                            2. 5

                                              I agree that anecdotes like this can get old, but I’ve been meaning to actually write a similar post to this… on something I’ve been calling the “too many buttons” syndrome. This issue pops up a ton in large pieces of software (Though I’m specifically thinking of projects like JRA and Confluence) where there’s an option for everything.

                                              Not everyone gets that simplicity is good because it can be harder to sell. “If a user wants it, we should do it” is something I’ve heard just a few too many times without bothering to look at the use case or if it could be done better. Sometimes it’s worth stepping back and looking at the complexity something will add to the project (in both code and testing… especially when it comes to options and how they interact with each other) rather than just adding all the little features.

                                              1. 5

                                                In my experience a lot of commercial companies that develop under tight deadlines produce a lot of suboptimal and dreadful code. Often it takes more time, to produce less code simply because the more time you spend on a difficult problem, the better you understand it. I think the reason that most a lot of software is bloated and complex is because it’s “good enough” which is optimal from an economic point of view.

                                                The other day there was a discussion here on Lobsters about all the required pieces needed to run a Mastodon instance and the popular solution of abstracting all that away in a Docker container. There are alternative implementations that depend on a smaller number of components alleviating the need for dumping everything in a container (of course the question is, do these alternatives offer the same functionality).

                                                How do we detect simplicity?

                                                For me personally simplicity has to do with readability, maintainability and elegance of code or infrastructure. If someones solution involves three steps, and someone else can do it in two steps (with comparable cognitive load per step), I would say it’s more simple.

                                                How do we know when we shouldn’t simplify?

                                                If that would cut some features you cannot miss.

                                                1. 5

                                                  You are so right. After years of experience, I only start to clarify my idea of “simplicity”. There are different kind of simplicity most of them are not totally compatible. And in my opinion some need to be preferred to other, but there is no clear rule. To make a choice between different complexity I still use a lot of intuition and I debate a lot, and I am still unsure my choice are the best.

                                                  • only using basic feature of a language (do not use advanced programming language feature) is certainly the most important aspect in simplicity. It will make your code easy to read by more people.
                                                  • don’t use too much intermediate functions, and if possible don’t disperse those function in many different files before really feel you are copy/pasting too much. My rule of thumb is, 2 or 3 times duplications is totally fine and superior to centralisation of code. It start to be really clear that code factorisation is good when you start repeating yourself more than 6 to 10 times
                                                  • only really use advanced feature of the language after having tried not to use it for some time and really lack the ability of that advanced feature. Some examples of what I call advanced feature of a language are; class heritage, protocols in Clojure, writing your own typeclasses in Haskell, meta programming (macros in LISP), etc…
                                                  • prefer stateless functions to objects/service with internal states
                                                  • prefer pure functions (side effect free) other procedures (functions with side effects)
                                                  • give a lot of preference to composable solutions ; composable in the algebraic meaning. For example, I do my best not to use LISP macros, because most of the time macros break composability. The same could be said when you start to deal with type-level programming in Haskell, or when you are doing meta-programming in ruby/python.

                                                  For now, all those rules are still quite artisanal. I don’t have any really hard metrics or strong rules. Everything I just said is “preferable” but I’m pretty sure we can find exception to most of those rules.

                                                  1. 5

                                                    Amen, +1, etc. “Simplicity” often just means that a concept fits cleanly in the maker’s head at a particular point in time. How many times have I returned to a project I thought was simple only to find I had burdened it with spooky magic because I didn’t benefit from critical distance at the time? When was the last time I deemed another person’s work “too complex” because I couldn’t understand it in one sitting and wasn’t aware of the constraints they were operating under? Answers: too often and too recently.

                                                    1. 3

                                                      What kinds of simplicity are there?

                                                      This is a good question (as are the others). Borrowing from Holmes, I’d say there’s a continuum from naive simplicity, to complexity, to simplicity on the other side of complexity (which is what is truly interesting)

                                                      For example, “naively simple” code would only cover a small subset (say, the happy path) of a business problem. Complex code would handle all, or most, of the business complexity but in a messy, complicated way. “Other side” simplicity refines that complex code into something that can handle the business complexity without itself becoming overly complicated.

                                                      1. 2

                                                        What happens to simplicity? We trade it for a other things of course. For example, you can have simple regular expressions, but most people prefer less simple and more powerful implementation like Perls.

                                                        Simplicity is often a tradeoff versus easyness, performance, flexibility, reusability, useability, etc. So simplicity is good, but those other things are also good.

                                                        1. 1

                                                          Most people seem to agree that simplicity is best. However, when it comes down to simplicity for the user versus the developer, I have seen disagreement. Each trade off is going to be situation and implementation dependent, but at my job I’ve been pushing for a simpler developer environment.

                                                          In my office, there is a tendency to create exceptions to rules because it makes things simpler for the user. Since the environment has more exceptional circumstances, it tends to have more errors when people forget the undocumented exception case. In my opinion, this causes an uneven experience for the user despite being “simpler.”

                                                          My experience is coming from a medium sized, non-tech company. I work in the IT department so we are a cost center. There is an emphasis on white glove treatment of the revenue producing portions of the company. YMMV

                                                        1. 1

                                                          Hey guys I’ve wrote some thoughts on working with complex software projects. I would appreciate your feedback!

                                                          1. 3

                                                            Nice article!

                                                            I think you forgot one very important aspect, politic. Most failed project I saw was generally involved with someone lying. Generally I saw at least one person, middle or high in the hierarchy. That person want the project to fail while pretending the opposite during all those meetings. I even saw worse, two parties involved wanted the project to fail while their higher hierarchy pushed to make it a success.

                                                            You can detect such problem because involved people start to adopt a very defensive communication. Mainly each part program the failure, but none will want to be responsible for it. So each part will start to accumulate evidence the problem comes from the other party involved. During each meeting, the same thing will be repeated again and again. Generally almost no written document will be involved. When this is the case, it only address superficial issues, etc…

                                                            Also, I disagree with your “estimate” part. Estimate the time it will take for a very small project is already almost impossible to do. Estimate is in fact a failed metric, only fool can rely on it (see Hofstadter’s law). That being said, I think the methodology of applying the mean of 3 to 4 people estimate is the wrong thing to do. If you want something closer to the reality, take the worst estimate and double it (at least). The rule about the +20% is ridiculous, because, most task will be estimated correctly, but there will be those 2, 3 tasks that instead of taking 1h will take 3 months.

                                                            Also, as you pointed out, in a big project you are almost certain some part will fail to deliver at all. You’ll need to adapt and have a plan B for each potential sub-task failure.

                                                            1. 8

                                                              Its been a very long time since I haven’t updated my blog, mostly about functional programming & Haskell. Still here it is:

                                                              http://yannesposito.com

                                                              1. 5

                                                                As much as I would like a decentralised web to take over. I think I see a major issue with ActivityPub. Apparently, search doesn’t appear to be specified. And one advantage big centralised services will have is the ability to search all their network. For example, I would like to search all the network for a specific username, keyword, etc… Without that, it’s like going back to the net before google search.

                                                                1. 2

                                                                  Yeah, I’m curious if ActivityPub can support arbitrary aggregation over multiple nodes. It seems to me that in this kind of architecture, maybe nodes ought to support publishing changes over websocket to interested listeners. You could have aggregation nodes doing things like search, or post rankings, which could attach themselves in this fashion. Plus this would have the added benefit that if you didn’t like a particular aggregator’s implementation (a hotness formula for example, or search indexing algorithm) you could switch to a different one.

                                                                  1. 1

                                                                    Usernames include the server, so I’m not sure that case makes sense.

                                                                    Not supporting keyword search means I don’t get random and bots sealioning my conversations.

                                                                    1. 1

                                                                      I was thinking about peer tube for example. It would be very nice if I was able to search only for the current node or on all fediverse for the things I’d like to find. Like niche programming languages for example. The goal of publishing something is to be read, even by bots. Also I’m not sure a federated network would be more robust in face of bots. Pretty sure it would be the opposite, because each node would have less data to analyze for bot detection. Still by the end, I’m pretty sure the only good solution would be to have a global “Web of Trust”.

                                                                      1. 2

                                                                        Ahh, I see. 99% of my social media us is to connect with people I already know (or have in my extended network). For that use case, it’s a feature to be less easily discovered.

                                                                        For publishing content it’s definitely the opposite; you want it to be found. Difficult though because now you’re competing for attention with spammers.

                                                                  1. 6

                                                                    mmm…

                                                                    #!/usr/bin/env stack
                                                                    {- stack script
                                                                       --resolver lts-11.6
                                                                       --install-ghc
                                                                       --package protolude
                                                                    -}
                                                                    {-# LANGUAGE NoImplicitPrelude #-}
                                                                    {-# LANGUAGE OverloadedStrings #-}
                                                                    import Protolude
                                                                    
                                                                    main = putText "Hello, world!"
                                                                    
                                                                    1. 4

                                                                      Mainly for that reason I switched to a combo bitlbee + weechat + (screen + mosh + weechat-ncurses)

                                                                      It was a bit long to set up everything and there are still some rough edges. But, now with about 3Mo of RAM I can chat with:

                                                                      • 3 slack community with lot of channels
                                                                      • some gitter channels
                                                                      • many IRC channels freenode
                                                                      • hipchat

                                                                      I have manually set alerts. The text use my preferred color theme. Typically the only other reason to get rid of slack is that I couldn’t have clear text on dark background.

                                                                      Now my chat system feels like a calm place to discuss.

                                                                      1. 11

                                                                        I totally agree with most of the content of this article but. Regarding formatting I would go farther than that.

                                                                        I would love if all programming language had “normal form” for their source code. The very least would be that Formatter should be projections ( format(x) = format(format(x)) ). But a normal form would be even better. I know this is something very difficult to achieve mostly because of social issues regarding code formatting preferences. Typically, what about the 80 column rule? Etc… By normal form I mean a function that would parse your code to an AST and only use the AST to output a formatted text. The normal form won’t be affected by adding/removing a newline, adding spaces/tabs, changing alignments. It should be affected by comment though, that would certainly change some programming languages paradigm because it would mean that the comment should be part of the AST even if ignored. I am sure I forget a lot of difficulties about how to write one.

                                                                        One big advantage about having such a normal form is that it would separate the “syntax” from the “AST” and it would be easier to split the “parsing” of your language for your preferred syntax. It would then be easier to have tools that could provide different equivalent syntaxes for your language. You could choose between “C/C++/Javascript like format”, “Python / Ruby” like format, ML/Haskell-like format, LISP like format, etc… But in the end the code would only be the “AST” using one of those format, typically the one considered the most readable by the team. Also it would greatly reduce “false” commit that are only a reformatting away.

                                                                        1. 5

                                                                          This idea has been tried MANY times. I watched a talk about a Lisp that provided skinnable Python, C, etc. syntax maybe 12 years ago.

                                                                          I would just try making a language yourself that does this, and you might realize why it isn’t a more widely adopted idea. It’s not like nobody has thought of it for the last 20-30 years.

                                                                          1. 1

                                                                            I came up with a very similar idea. I got it from GUI skinning. So, you have the base language represented by a grammar or something. Then, the IDE can let the programmer define their own syntax that any file they receive is automatically converted to. Then, there were certain differences in structuring that might be handled with source-to-source translation like in Semantic Design’s tool.

                                                                            I never went past the concept, though.

                                                                            1. 4

                                                                              You can do this with reader macros in lisp. In addition to switching per file, you can switch between syntaxes in the same file with #. Most people don’t bother, but it’s already implemented if you want it.

                                                                            2. 1

                                                                              Not only is that very difficult to do technically, it ignores that languages are usually chosen to take advantage of concepts that may not exist in another language.

                                                                              If I write a template in C++, how do I translate that into Haskell? How do I translate it to Go? If I use the built-in parallelism in Erlang, how do I translate that to Python? If I use a Python list, what does that become in C++?

                                                                              It’d be incredibly difficult to implement, or it would end up with a common denominator subset of features that wasn’t particularly useful.

                                                                              I think a group of languages could be designed from the ground up to be AST compatible with each other, but I doubt it’s possible to do for an arbitrary group of existing languages.

                                                                            1. 1

                                                                              I considered stow, but I finally decided to use yadm The main reason is because I can easily encrypt sensible files with yadm.

                                                                              Also yadm works with my git aliases. So using yadm feels exactly like using git with my personal preferences.

                                                                              1. 1

                                                                                Hhm looks pretty interesting, does it support other vcs than git? And I think some of the password encryption stuff could be managed by something like pass

                                                                              1. 3

                                                                                I’m a bit of a Scala newcomer and pretty much write all my code to run on Spark, but have completed a few of the Coursera courses. Can someone point me at the shortcomings or danger zones of Scala? To me at least it doesn’t seem like there is a language out there that is functional, type safe and expressive like Scala which works universally in back-end services, web apps, and big data apps running on Spark/Kafka/etc. The only other language that is similar in this respect is Clojure but it’s not statically typed which is something I’m drawn to coming from Python. Are there alternatives to Scala people should be looking it?

                                                                                1. 7

                                                                                  To me, the big danger zone is that Scala has two largely incompatible groups of users. One is happy to use a better Java, the other is essentially writing Haskell on the JVM. Neither group likes the other’s code. The Haskell-flavored-Scala folks tend to be blowhards about it more often than the better-Java folks, so expect that style to win in any given project/organization. Also expect to lose people because of it. I’ve seen this happen from up close and from afar.

                                                                                  I wish Scala the language were separable from Scala the community. There are some great ideas in there, but I’m happier using most of them in a different language.

                                                                                  1. 5

                                                                                    F# and Ocaml seem like strong contenders. Less Ocaml, but that’s mostly because of library support.

                                                                                    1. 3

                                                                                      You can take a look at Haskell or close to Haskell languages:

                                                                                      At least Haskell is functional, type safe, and certainly as expressive as Scala. I used it Haskell along kafka, and for backend services as web apps. The first link talk about how it is used by tweag.io to run on Spark, I don’t have any personal experience with that.

                                                                                      1. 2

                                                                                        I do think Scala is the best language going at the moment. But there are various rough edges, partly to do with JVM compatibility and partly to do with backwards compatibility with previous versions of Scala, plus the occasional bad design decision; just stuff like null, ClassTag, the contortions needed to implement HList and Record (which largely don’t cause problems for correct code but show up in things like the error messages you get when you make a mistake), the lack of enums or any truly first-class kind of sum type….

                                                                                        Pitfalls to avoid: SBT, akka actors, the cake pattern, pattern matching where it’s impossible to tell safe from unsafe, monad-like types that don’t obey the laws, implicit parameters used as actual parameters (something Odersky is now promoting), lack of kind polymorphism…

                                                                                        In terms of alternatives F# and OCaml don’t have HKT; Haskell is an option but seems to introduce as many problems as it solves (laziness making performance reasoning hard, lack of good tooling, limited compatibility with other ecosystems). I had high hopes for Ceylon but I’ve come to think union types are the wrong thing (they’re inherently noncompositional compared to the opaque disjoint union kind of sum type). I’m excited for Idris - that seems to take the good parts of Scala and also bring something new and valuable to the table.