My sense from using Scala professionally for the past approximate five years has been that most of the interests in building out tooling, infrastructure, etc. in that community are predominantly business-driven. That is, the Scala community tends to produce accelerators as a side effect of business need, not ars gratia artis, as many other communities do: Haskell, Rust, Go, etc. Or, somebody produces something once and doesn’t update it and doesn’t have the resources or skills to build a community around a much used project. This is not a condemnation of maintainers who don’t maintain, but rather a failure of the Scala community to produce a group of curators — people who will take on the load when maintainers abdicate —as well formed and intentional as many other communities have, or a culture of curation. “Disarray” is too strong of a word, but as someone who moves between four languages daily, I can compare communities and their resources pretty well and conclude that my technical decision to use Scala is sound while the political decision frustrates me at times.
The main source of the embarrassing cycle of “Hype – Failed Promises – Abandonment – The Shiny New Thing – Hype – …” is in fact coming from the academic side.
To be clear: I’m not blaming students that they drop their work and completely disappear the minute they handed in their thesis, and let other people deal with the consequences. That’s just the way it is.
The problem is the ease with which they can get things added to the language/library, especially compared to the scrutiny outside contributions regularly receive.
This pattern has repeated over and over, and I think it’s one of the unchangeable parts of the language/community.
If you care about a quality, documentation, tooling, then Scala isn’t the right language for you.
Simply because “we managed to ship a new version of Scala without breaking every IDE” is not a topic you can write a paper of.
This is a very important point and I’m glad that you wrote it out.
Would you say that the Scala ecosystem has a continuity problem because businesses and academia are focused primarily on the now and not necessarily building the road ahead for the community and then maintaining those roads once built?
The core open-source community is the pizza dough that provides long-term stability and maintenance, and the academia/businesses provide the toppings.
Some languages have a large base of long-term, open-source contributors – they are family-sized pizzas which can accommodate a lot different toppings. People can get their favorite piece, everyone is happy.
Scala is different. It has barely any substantial open-source contributors remaining – and everyone is fighting over the toppings to be placed on that coin-sized piece of pizza dough. As a result the kitchen is a complete mess, and nobody is happy.
Scala’s problem is not the focus of businesses or academia, but that it doesn’t have any focus on it’s own.
There is literally no one left (since Paul walked away) who is able to establish or uphold any kind of technical standards, or tell people “no, we are not adding another 6 new keywords to the language”. Heck, they couldn’t even get their compiler test suite green on anything newer than Java 8, but they kept publishing new versions anyway since 2017!
Thanks for asking. I based my decision mostly on architectural analysis with a healthy dose of personal experience. We’re building a new product with cloud and on-premises components. Originally, I’d intended to build everything from scratch but my team identified some OSS that did 100% of we needed with some extra complexity in managing it. The trade-off was a much faster time to delivery in exchange for playing in someone else’s sandbox and by their rules. That aspect was worth it but I lament not being able to write it all in Scala!
However, our cloud services are written mostly in Scala. We have a few stateless microservices for which Scala was a great fit: high-performance OOTB for what within six months will be ~500 req/s and within two will likely exceed 4,000 req/s for one component with another being far more I/O bound. We’re integrating heavily with ecosystems that have Java libraries and we’ve found decent Scala wrappers that give us idiomaticity without building and maintaining them ourselves. We could have used just about any stack for these couple of services but Scala’s enabled us to express ourselves in types and exploit the advantages of functional programming. I’m chasing the holy grail of “it’s valid because it compiles” but we’ve got enough unit tests to complement our design that I’m pretty sure we’re on the right track.
One notable failure was on service that is primarily a user-interactive web app. We had two false starts with Scalatra, which we’re using for the other services, and Play! before switching to Ruby on Rails (temporarily) out of frustration with documentation, lack of examples, lack of drop-ins like what are commonly available in the Rails ecosystem, from whence I came many moons ago. We chose Rails because of the component owner’s experience with it as well as my experience with it and JRuby, knowing that if we started to implement any shareable logic, we could do so in a way that all of our apps could consume. We learned about Lift too late and http4s and some others didn’t give us the right impression for a web app. I just learned about Udash last week and it may be a candidate for replacement. However, it’ll be several months: even uttering R-E-W-R-I-T-E would be kiboshed from on high at the moment as the component does what we need right now.
Moving forward, we’ll be looking at moving some of these services to http4s, etc. once more of my team is comfortable with more hardcore Scala FP. Writing AWS Lambda functions in Rust is also on my radar, as a part of our on-prem product is written in Rust.
Out of curiosity, did you evaluate Elixir? The obvious draw is the Ruby-like syntax but it seems that the Phoenix framework has a great concurrency story on top of being a Rails-inspired ‘functional MVC’ framework.
There was a running joke on my team and another about us throwing everything away and doing it all in Elixir! I mused about it a little but decided that we were better off in RoR because we can hire for it more easily if we ended up committing to the RoR implementation long-term and because we could run RoR on JRuby as a transition step back to Scala, should we have enough business logic to merit a shared implementation. So far, the latter hasn’t been the case since the app is 95% CRUD.
I’d really like to see an analysis of the websocket concurrency story of Elixir, Scala, and Rust, with perhaps some others for more general applicability. Our app is using good ol’ fashioned HTTP requests right now but we’ve identified some opportunities to shave some transfer overhead by switching to websockets eventually.
That’s funny. I guess Elixir has some visibility in this space. I’m planning to use it to build a proof-of-concept. For me it’s the developer experience, combined with the performance and concurrency profile. Here are some comparisons: https://hashrocket.com/blog/posts/websocket-shootout
I don’t think that’s fair at all. There are a number of actively maintained projects that are at the center of the community.
I do think there are relatively few well maintained libraries outside of that core, but I think that’s a result of the community being relatively small, and the escape hatch of having java libraries available to do almost everything.
Community (non corporate) projects and organizations:
All typelevel projects
monix
Scalaz
Sbt
Ensime
All widely used, all with large contributor bases.
Additionally the alternative compilers, scalajs and scala native.
I do think there are relatively few well maintained libraries outside of that core
The Scala community does have a solid core and near-core extension community. I consider libraries like Monix, Cats, and Scalaz to be nearly a part of the standard library because of how often they are used. sbt and ensime are important but they’re not exceptional: every stack needs a build tool and editor integration. These are solid now, and I appreciate the work that goes into them. Frankly, it wasn’t until sbt hit 1.0.0 that I considered it ready for widespread use because of its obtuseness/unergonomic interface prior to then. I’m eager to see what Li Haoyi’s mill will become.
Things I’ve noted in the past that I’ve found in a less-than-desirable state compared to other stacks:
Authentication & Authorization - no nearly drop-in solution like Devise in the RoR ecosystem
Project websites being down for weeks because someone forgot to re-up the TLS cert, even in a LetsEncrypt automation world
Out of date documentation to the point of being dangerous practices with little more than a “someone submit a PR to fix that”. I get that maintainers get busy but when safety is the topic, is it acceptable to wait for a drive-by contributor to get it right and contribute the correction? What if they do that and then no maintainer merges it for years?
The Scala Center has the promise of addressing much of it but I speculate that it’s insufficiently funded to be the ecosystem plumbers and teachers it aspires to be. I’ve been impressed with its work so far, though.
From what I see, there appear to be two distinct camps of Scala developers. The first camp consists of OO devs who are looking for a more modern Java, and the second camp consists of FP devs who are looking for a statically typed functional language on the JVM.
Nowadays Java is starting to address many of the gaps while Kotlin provides a much simpler alternative that focuses on the features that OO devs are looking for when considering Scala. I imagine that going forward Scala will become a hard sell for the OO crowd. Meanwhile, Scala is starting to have competition on the FP front from Eta which is a Haskell dialect that runs on the JVM and is able to leverage existing Haskell libraries.
The goal of simplifying the language with Dotty seems like a good idea, but it’s going to create a lot of friction for existing projects similarly to what we’re seeing happening with Python 3. I think that at this point Scala needs to clearly identify what specific problem it addresses that’s not addressed better by other languages.
The goal of simplifying the language with Dotty seems like a good idea […]
That’s not going to happen. The promises on conference slides do not reflect the reality.
There are already 6 or 7 new keywords, not speaking of other additions, and the cleanups do not depend on Scala 3 at all – in fact many of them have been implemented (and shipped behind a flag) for years already.
My sense from using Scala professionally for the past approximate five years has been that most of the interests in building out tooling, infrastructure, etc. in that community are predominantly business-driven. That is, the Scala community tends to produce accelerators as a side effect of business need, not ars gratia artis, as many other communities do: Haskell, Rust, Go, etc. Or, somebody produces something once and doesn’t update it and doesn’t have the resources or skills to build a community around a much used project. This is not a condemnation of maintainers who don’t maintain, but rather a failure of the Scala community to produce a group of curators — people who will take on the load when maintainers abdicate —as well formed and intentional as many other communities have, or a culture of curation. “Disarray” is too strong of a word, but as someone who moves between four languages daily, I can compare communities and their resources pretty well and conclude that my technical decision to use Scala is sound while the political decision frustrates me at times.
I think it couldn’t be farther from the truth.
The main source of the embarrassing cycle of “Hype – Failed Promises – Abandonment – The Shiny New Thing – Hype – …” is in fact coming from the academic side.
To be clear: I’m not blaming students that they drop their work and completely disappear the minute they handed in their thesis, and let other people deal with the consequences. That’s just the way it is.
The problem is the ease with which they can get things added to the language/library, especially compared to the scrutiny outside contributions regularly receive.
This pattern has repeated over and over, and I think it’s one of the unchangeable parts of the language/community.
If you care about a quality, documentation, tooling, then Scala isn’t the right language for you. Simply because “we managed to ship a new version of Scala without breaking every IDE” is not a topic you can write a paper of.
This is a very important point and I’m glad that you wrote it out.
Would you say that the Scala ecosystem has a continuity problem because businesses and academia are focused primarily on the now and not necessarily building the road ahead for the community and then maintaining those roads once built?
Think of it like this:
The core open-source community is the pizza dough that provides long-term stability and maintenance, and the academia/businesses provide the toppings.
Some languages have a large base of long-term, open-source contributors – they are family-sized pizzas which can accommodate a lot different toppings. People can get their favorite piece, everyone is happy.
Scala is different. It has barely any substantial open-source contributors remaining – and everyone is fighting over the toppings to be placed on that coin-sized piece of pizza dough. As a result the kitchen is a complete mess, and nobody is happy.
Scala’s problem is not the focus of businesses or academia, but that it doesn’t have any focus on it’s own.
There is literally no one left (since Paul walked away) who is able to establish or uphold any kind of technical standards, or tell people “no, we are not adding another 6 new keywords to the language”. Heck, they couldn’t even get their compiler test suite green on anything newer than Java 8, but they kept publishing new versions anyway since 2017!
Of course you would claim it’s sound, but is that objective? What do you base that decision on? Genuinely curious here.
Thanks for asking. I based my decision mostly on architectural analysis with a healthy dose of personal experience. We’re building a new product with cloud and on-premises components. Originally, I’d intended to build everything from scratch but my team identified some OSS that did 100% of we needed with some extra complexity in managing it. The trade-off was a much faster time to delivery in exchange for playing in someone else’s sandbox and by their rules. That aspect was worth it but I lament not being able to write it all in Scala!
However, our cloud services are written mostly in Scala. We have a few stateless microservices for which Scala was a great fit: high-performance OOTB for what within six months will be ~500 req/s and within two will likely exceed 4,000 req/s for one component with another being far more I/O bound. We’re integrating heavily with ecosystems that have Java libraries and we’ve found decent Scala wrappers that give us idiomaticity without building and maintaining them ourselves. We could have used just about any stack for these couple of services but Scala’s enabled us to express ourselves in types and exploit the advantages of functional programming. I’m chasing the holy grail of “it’s valid because it compiles” but we’ve got enough unit tests to complement our design that I’m pretty sure we’re on the right track.
One notable failure was on service that is primarily a user-interactive web app. We had two false starts with Scalatra, which we’re using for the other services, and Play! before switching to Ruby on Rails (temporarily) out of frustration with documentation, lack of examples, lack of drop-ins like what are commonly available in the Rails ecosystem, from whence I came many moons ago. We chose Rails because of the component owner’s experience with it as well as my experience with it and JRuby, knowing that if we started to implement any shareable logic, we could do so in a way that all of our apps could consume. We learned about Lift too late and http4s and some others didn’t give us the right impression for a web app. I just learned about Udash last week and it may be a candidate for replacement. However, it’ll be several months: even uttering R-E-W-R-I-T-E would be kiboshed from on high at the moment as the component does what we need right now.
Moving forward, we’ll be looking at moving some of these services to http4s, etc. once more of my team is comfortable with more hardcore Scala FP. Writing AWS Lambda functions in Rust is also on my radar, as a part of our on-prem product is written in Rust.
Out of curiosity, did you evaluate Elixir? The obvious draw is the Ruby-like syntax but it seems that the Phoenix framework has a great concurrency story on top of being a Rails-inspired ‘functional MVC’ framework.
There was a running joke on my team and another about us throwing everything away and doing it all in Elixir! I mused about it a little but decided that we were better off in RoR because we can hire for it more easily if we ended up committing to the RoR implementation long-term and because we could run RoR on JRuby as a transition step back to Scala, should we have enough business logic to merit a shared implementation. So far, the latter hasn’t been the case since the app is 95% CRUD.
I’d really like to see an analysis of the websocket concurrency story of Elixir, Scala, and Rust, with perhaps some others for more general applicability. Our app is using good ol’ fashioned HTTP requests right now but we’ve identified some opportunities to shave some transfer overhead by switching to websockets eventually.
That’s funny. I guess Elixir has some visibility in this space. I’m planning to use it to build a proof-of-concept. For me it’s the developer experience, combined with the performance and concurrency profile. Here are some comparisons: https://hashrocket.com/blog/posts/websocket-shootout
More results available here: https://github.com/hashrocket/websocket-shootout
Thanks for those. I’m sad that they’re out of date but the info is useful nonetheless.
I don’t think that’s fair at all. There are a number of actively maintained projects that are at the center of the community.
I do think there are relatively few well maintained libraries outside of that core, but I think that’s a result of the community being relatively small, and the escape hatch of having java libraries available to do almost everything.
Community (non corporate) projects and organizations:
All widely used, all with large contributor bases.
Additionally the alternative compilers, scalajs and scala native.
At least from my perspective it seems that:
Sam Halliday is not the Ensime creator https://github.com/ensime/ensime-server/blob/0e202f7da0c79c68e1c0d108267bc580747e7fad/NOTICE
The Scala community does have a solid core and near-core extension community. I consider libraries like Monix, Cats, and Scalaz to be nearly a part of the standard library because of how often they are used. sbt and ensime are important but they’re not exceptional: every stack needs a build tool and editor integration. These are solid now, and I appreciate the work that goes into them. Frankly, it wasn’t until sbt hit 1.0.0 that I considered it ready for widespread use because of its obtuseness/unergonomic interface prior to then. I’m eager to see what Li Haoyi’s
millwill become.Things I’ve noted in the past that I’ve found in a less-than-desirable state compared to other stacks:
The Scala Center has the promise of addressing much of it but I speculate that it’s insufficiently funded to be the ecosystem plumbers and teachers it aspires to be. I’ve been impressed with its work so far, though.
From what I see, there appear to be two distinct camps of Scala developers. The first camp consists of OO devs who are looking for a more modern Java, and the second camp consists of FP devs who are looking for a statically typed functional language on the JVM.
Nowadays Java is starting to address many of the gaps while Kotlin provides a much simpler alternative that focuses on the features that OO devs are looking for when considering Scala. I imagine that going forward Scala will become a hard sell for the OO crowd. Meanwhile, Scala is starting to have competition on the FP front from Eta which is a Haskell dialect that runs on the JVM and is able to leverage existing Haskell libraries.
The goal of simplifying the language with Dotty seems like a good idea, but it’s going to create a lot of friction for existing projects similarly to what we’re seeing happening with Python 3. I think that at this point Scala needs to clearly identify what specific problem it addresses that’s not addressed better by other languages.
I think your assessment that Scala faces pressure from both Java/Kotlin as well as functional languages is spot on.
https://www.benfrederickson.com/ranking-programming-languages-by-github-users/ indicates that Scala lost 20% of its active users within a year, while Kotlin tripled its amount of active users, surpassing Scala by a large margin already.
That’s not going to happen. The promises on conference slides do not reflect the reality.
There are already 6 or 7 new keywords, not speaking of other additions, and the cleanups do not depend on Scala 3 at all – in fact many of them have been implemented (and shipped behind a flag) for years already.
Can you imagine someone writing a blog post like “farewell, ball-peen hammers! I will be using framing hammers from now on!”