1. 1

    Hey guys I’ve wrote some thoughts on working with complex software projects. I would appreciate your feedback!

    1. 3

      Nice article!

      I think you forgot one very important aspect, politic. Most failed project I saw was generally involved with someone lying. Generally I saw at least one person, middle or high in the hierarchy. That person want the project to fail while pretending the opposite during all those meetings. I even saw worse, two parties involved wanted the project to fail while their higher hierarchy pushed to make it a success.

      You can detect such problem because involved people start to adopt a very defensive communication. Mainly each part program the failure, but none will want to be responsible for it. So each part will start to accumulate evidence the problem comes from the other party involved. During each meeting, the same thing will be repeated again and again. Generally almost no written document will be involved. When this is the case, it only address superficial issues, etc…

      Also, I disagree with your “estimate” part. Estimate the time it will take for a very small project is already almost impossible to do. Estimate is in fact a failed metric, only fool can rely on it (see Hofstadter’s law). That being said, I think the methodology of applying the mean of 3 to 4 people estimate is the wrong thing to do. If you want something closer to the reality, take the worst estimate and double it (at least). The rule about the +20% is ridiculous, because, most task will be estimated correctly, but there will be those 2, 3 tasks that instead of taking 1h will take 3 months.

      Also, as you pointed out, in a big project you are almost certain some part will fail to deliver at all. You’ll need to adapt and have a plan B for each potential sub-task failure.

      1. 8

        Its been a very long time since I haven’t updated my blog, mostly about functional programming & Haskell. Still here it is:

        http://yannesposito.com

        1. 5

          As much as I would like a decentralised web to take over. I think I see a major issue with ActivityPub. Apparently, search doesn’t appear to be specified. And one advantage big centralised services will have is the ability to search all their network. For example, I would like to search all the network for a specific username, keyword, etc… Without that, it’s like going back to the net before google search.

          1. 2

            Yeah, I’m curious if ActivityPub can support arbitrary aggregation over multiple nodes. It seems to me that in this kind of architecture, maybe nodes ought to support publishing changes over websocket to interested listeners. You could have aggregation nodes doing things like search, or post rankings, which could attach themselves in this fashion. Plus this would have the added benefit that if you didn’t like a particular aggregator’s implementation (a hotness formula for example, or search indexing algorithm) you could switch to a different one.

            1. 1

              Usernames include the server, so I’m not sure that case makes sense.

              Not supporting keyword search means I don’t get random and bots sealioning my conversations.

              1. 1

                I was thinking about peer tube for example. It would be very nice if I was able to search only for the current node or on all fediverse for the things I’d like to find. Like niche programming languages for example. The goal of publishing something is to be read, even by bots. Also I’m not sure a federated network would be more robust in face of bots. Pretty sure it would be the opposite, because each node would have less data to analyze for bot detection. Still by the end, I’m pretty sure the only good solution would be to have a global “Web of Trust”.

                1. 2

                  Ahh, I see. 99% of my social media us is to connect with people I already know (or have in my extended network). For that use case, it’s a feature to be less easily discovered.

                  For publishing content it’s definitely the opposite; you want it to be found. Difficult though because now you’re competing for attention with spammers.

            1. 6

              mmm…

              #!/usr/bin/env stack
              {- stack script
                 --resolver lts-11.6
                 --install-ghc
                 --package protolude
              -}
              {-# LANGUAGE NoImplicitPrelude #-}
              {-# LANGUAGE OverloadedStrings #-}
              import Protolude
              
              main = putText "Hello, world!"
              
              1. 4

                Mainly for that reason I switched to a combo bitlbee + weechat + (screen + mosh + weechat-ncurses)

                It was a bit long to set up everything and there are still some rough edges. But, now with about 3Mo of RAM I can chat with:

                • 3 slack community with lot of channels
                • some gitter channels
                • many IRC channels freenode
                • hipchat

                I have manually set alerts. The text use my preferred color theme. Typically the only other reason to get rid of slack is that I couldn’t have clear text on dark background.

                Now my chat system feels like a calm place to discuss.

                1. 11

                  I totally agree with most of the content of this article but. Regarding formatting I would go farther than that.

                  I would love if all programming language had “normal form” for their source code. The very least would be that Formatter should be projections ( format(x) = format(format(x)) ). But a normal form would be even better. I know this is something very difficult to achieve mostly because of social issues regarding code formatting preferences. Typically, what about the 80 column rule? Etc… By normal form I mean a function that would parse your code to an AST and only use the AST to output a formatted text. The normal form won’t be affected by adding/removing a newline, adding spaces/tabs, changing alignments. It should be affected by comment though, that would certainly change some programming languages paradigm because it would mean that the comment should be part of the AST even if ignored. I am sure I forget a lot of difficulties about how to write one.

                  One big advantage about having such a normal form is that it would separate the “syntax” from the “AST” and it would be easier to split the “parsing” of your language for your preferred syntax. It would then be easier to have tools that could provide different equivalent syntaxes for your language. You could choose between “C/C++/Javascript like format”, “Python / Ruby” like format, ML/Haskell-like format, LISP like format, etc… But in the end the code would only be the “AST” using one of those format, typically the one considered the most readable by the team. Also it would greatly reduce “false” commit that are only a reformatting away.

                  1. 5

                    This idea has been tried MANY times. I watched a talk about a Lisp that provided skinnable Python, C, etc. syntax maybe 12 years ago.

                    I would just try making a language yourself that does this, and you might realize why it isn’t a more widely adopted idea. It’s not like nobody has thought of it for the last 20-30 years.

                    1. 1

                      I came up with a very similar idea. I got it from GUI skinning. So, you have the base language represented by a grammar or something. Then, the IDE can let the programmer define their own syntax that any file they receive is automatically converted to. Then, there were certain differences in structuring that might be handled with source-to-source translation like in Semantic Design’s tool.

                      I never went past the concept, though.

                      1. 4

                        You can do this with reader macros in lisp. In addition to switching per file, you can switch between syntaxes in the same file with #. Most people don’t bother, but it’s already implemented if you want it.

                      2. 1

                        Not only is that very difficult to do technically, it ignores that languages are usually chosen to take advantage of concepts that may not exist in another language.

                        If I write a template in C++, how do I translate that into Haskell? How do I translate it to Go? If I use the built-in parallelism in Erlang, how do I translate that to Python? If I use a Python list, what does that become in C++?

                        It’d be incredibly difficult to implement, or it would end up with a common denominator subset of features that wasn’t particularly useful.

                        I think a group of languages could be designed from the ground up to be AST compatible with each other, but I doubt it’s possible to do for an arbitrary group of existing languages.

                      1. 1

                        I considered stow, but I finally decided to use yadm The main reason is because I can easily encrypt sensible files with yadm.

                        Also yadm works with my git aliases. So using yadm feels exactly like using git with my personal preferences.

                        1. 1

                          Hhm looks pretty interesting, does it support other vcs than git? And I think some of the password encryption stuff could be managed by something like pass

                        1. 3

                          I’m a bit of a Scala newcomer and pretty much write all my code to run on Spark, but have completed a few of the Coursera courses. Can someone point me at the shortcomings or danger zones of Scala? To me at least it doesn’t seem like there is a language out there that is functional, type safe and expressive like Scala which works universally in back-end services, web apps, and big data apps running on Spark/Kafka/etc. The only other language that is similar in this respect is Clojure but it’s not statically typed which is something I’m drawn to coming from Python. Are there alternatives to Scala people should be looking it?

                          1. 7

                            To me, the big danger zone is that Scala has two largely incompatible groups of users. One is happy to use a better Java, the other is essentially writing Haskell on the JVM. Neither group likes the other’s code. The Haskell-flavored-Scala folks tend to be blowhards about it more often than the better-Java folks, so expect that style to win in any given project/organization. Also expect to lose people because of it. I’ve seen this happen from up close and from afar.

                            I wish Scala the language were separable from Scala the community. There are some great ideas in there, but I’m happier using most of them in a different language.

                            1. 5

                              F# and Ocaml seem like strong contenders. Less Ocaml, but that’s mostly because of library support.

                              1. 3

                                You can take a look at Haskell or close to Haskell languages:

                                At least Haskell is functional, type safe, and certainly as expressive as Scala. I used it Haskell along kafka, and for backend services as web apps. The first link talk about how it is used by tweag.io to run on Spark, I don’t have any personal experience with that.

                                1. 2

                                  I do think Scala is the best language going at the moment. But there are various rough edges, partly to do with JVM compatibility and partly to do with backwards compatibility with previous versions of Scala, plus the occasional bad design decision; just stuff like null, ClassTag, the contortions needed to implement HList and Record (which largely don’t cause problems for correct code but show up in things like the error messages you get when you make a mistake), the lack of enums or any truly first-class kind of sum type….

                                  Pitfalls to avoid: SBT, akka actors, the cake pattern, pattern matching where it’s impossible to tell safe from unsafe, monad-like types that don’t obey the laws, implicit parameters used as actual parameters (something Odersky is now promoting), lack of kind polymorphism…

                                  In terms of alternatives F# and OCaml don’t have HKT; Haskell is an option but seems to introduce as many problems as it solves (laziness making performance reasoning hard, lack of good tooling, limited compatibility with other ecosystems). I had high hopes for Ceylon but I’ve come to think union types are the wrong thing (they’re inherently noncompositional compared to the opaque disjoint union kind of sum type). I’m excited for Idris - that seems to take the good parts of Scala and also bring something new and valuable to the table.

                                1. 9

                                  I’m so glad to see this kind of blog posts about Nix.

                                  I’m under the impression that Nix has the same problem than git. Pretty neat tool but with terrible user interface. And for me it ends up on what could look like meaningless details.

                                  Imagine if instead of nix-env -qa <pkg> it was nix search <pkg>, if instead of nix-env -i <pkg> it was nix install <pkg>, if instead of nix-store --gc it was nix clean etc…

                                  Of course I could always use personal aliases. But that’s not the point. The UI change how I can think about the tool, in the same way that git checkout can be used for different purposes in my mind (reverting a file, changing my current branch…)

                                  1. 10

                                    I’m so glad you think so, too. Fortunately we have a new UI in the works, to make the UI reasonable.

                                  1. 3

                                    Stepping aside: Python, Ruby, Go, C++, Java, etc, and focusing on Clojure (as dynamic lang) and Haskell (as static lang) :

                                    Clojure:

                                    • Iteration Speed: YES
                                    • Correctness Checking: NO
                                    • Concise syntax : YES
                                    • Editing support : YES (spacemacs / cider)
                                    • Debugging support : YES

                                    Haskell:

                                    • Iteration Speed: YES (faster and better than Clojure)
                                    • Correctness Checking: YES
                                    • Concise syntax : YES
                                    • Editing support : YES (spacemacs / intero or ghc-mod)
                                    • Debugging support : YES

                                    Nice language features forgotten in the blog post:

                                    • Profiling: Clojure YES, Haskell YES
                                    • Fun : Clojure YES, Haskell YES
                                    • Fast startup time: Clojure NO, Haskell YES
                                    • Frontend Programming: Clojure YES, Haskell YES
                                    1. 13

                                      When I’m talking to people who are evaluating Haskell and trying to determine its risks, these things never come up. It’s always the same thing which can essentially be summed up as “library support”

                                      • Can I do MVC web dev?
                                      • How do I keep the code in sync with the database (migrations)?
                                      • How do I do performance (response time) monitoring?
                                      • Can I still use papertrail (or similar logging aggregation/search service)?
                                      • What’s the equivalent to getsentry.com for haskell?
                                      • How do I do continuous deployments from <my favorite CI service>?

                                      The flaws Mitchell is addressing here real flaws that affect people that are already pretty deep in Haskell-land. All of the above still have unsatisfactory answers IMO, even though it certainly can be done.

                                      1. 14

                                        Can I do MVC web dev?

                                        Yes. Most of the Yesod apps I work on have models, controllers, and views. I used Clojure and Python (Django) before Haskell, so I know what people expect here and it’s fine.

                                        How do I keep the code in sync with the database (migrations)?

                                        http://hackage.haskell.org/package/persistent ( I streamed some of this and web dev last night: https://www.youtube.com/watch?v=uYXX1t3GrsE )

                                        How do I do performance (response time) monitoring?

                                        http://hackage.haskell.org/package/datadog

                                        http://hackage.haskell.org/package/statsd-datadog

                                        Can I still use papertrail (or similar logging aggregation/search service)?

                                        http://hackage.haskell.org/package/katip-elasticsearch + treasure or whatever you want really.

                                        What’s the equivalent to getsentry.com for haskell?

                                        Sentry. http://hackage.haskell.org/package/raven-haskell Might could use some fixing up, but I’ve used this in prod and it was fine.

                                        How do I do continuous deployments from <my favorite CI service>?

                                        rsync a binary from your CI build/run ansible, bounce the upstart daemon. Our deployment stuff is in Ansible.

                                        Also see: http://haskelliseasy.com for other questions like this.

                                        1. 3
                                          1. Can I do MVC web dev? yes with Yesod, Snap and Happstack web framework (but there are other paradigms that really are worth at least an exploration)
                                          2. DB sync, using Yesod and more generally persistent will handle naive case automatically forces you to write the right db migration. So at least with yesod and persistent you can’t run code on a non compatible DB.
                                          3. papertrail (never heard about this service), but I imagine this is a service similar to grayling and yes there is a package to support it
                                          4. getsentry support: yes (https://hackage.haskell.org/package/raven-haskell-0.1.0.1/docs/System-Log-Raven.html)
                                          5. CI: http://yannesposito.com/Scratch/en/blog/Haskell-Tutorials--a-tutorial/#continuous-integration

                                          In my opinion, Haskell is a language completely suitable for production if you want to use traditional paradigms.

                                          But Haskell has a lot more to offer. There are many paradigms shift done in Haskell. For example, you can take a look at transient (see this recent thread for example):

                                          https://www.reddit.com/r/elm/comments/4wq3ko/playing_with_websockets_in_haskell_and_elm/d69o11p

                                          You might dislike the syntax, and believe that the example is quite minimalist. But it show that while MVC has its virtues, it is not the alpha and omega of web development and there can be another way of thinking about a web application as we are used to.

                                          1. 1

                                            compojure-api which is really great.

                                            1. 3
                                              • tried sandboxes (doesn’t work)
                                              • tried cabal freeze (made things slightly better)
                                              • tried nix (on Mac), hard to use and didn’t worked
                                              • tried stackage and Haskell LTS works like a charm. Never had any more problem with it.

                                              So, use global conf for Haskell LTS. Easy and efficient. Think of Haskell LTS as Debian stable for Haskell packages.

                                              1. 15

                                                This article put a finger in a major problem in programming.

                                                One one side, you can choose simples lean libraries. Using them to resolve simple problems is really easy, but once you want to use them to create a big application you are doomed to code yourself the same patterns again and again.

                                                So frameworks appears to fill the gap, and provide you a good architecture to organize your big application.

                                                There are two kind of frameworks:

                                                • the specialized one; they take decisions that remove a lot of your liberties. But are very efficient in their subdomain.
                                                • the framework that promise everything: clean organization and liberty.

                                                The second type of framework are from my experience complete failures. I am under the impression that Angular is in this category. Complex framework that promise everything, and therefore you end up having a lot of false complexity. Angular kind of becomes the Java J2EE of the js world.

                                                1. 6

                                                  I am in no way in allegiance with AngularJs, but if you don’t mind hearing from an actual working developer who actually uses a lot of AngularJs maybe you’ll get a fair representation of it’s use as a front-end framework. I do work at a large enterprise company, and we typically use AngularJs because it’s easy to learn and pickup, and there’s a ton of documentation online on just about everything you’d like to do with a SPA. It’s well supported cross-browser, though we run into a bug or two on IE, generally there’s always an easy work around to fix it and we rarely have to deal with cross-browser issues. Generally speaking, unless you’re building something complicated, a developer can get up and running building an SPA within two weeks – as in, just about anybody, such as a junior right out of community college, can write a decent commerical grade AngularJs SPA within weeks. We rarely use services for anything other than making http calls to our API, but factories can be customized to be used in a lot of other ways. They’re the main way to share data across controllers – generally, you keep one controller per view, and share data between them using services (i.e. factories). The design pattern is meant to be simple, so it’s easy to pick up quickly. So far, we’ve seen nothing but success from AngularJs. It hasn’t failed in any significant way since we’ve started using it a year and a half ago, and it’s significantly reduces dev time (and therefore costs).

                                                  I haven’t had the opportunity to use ReactJS on a significant project, but I notice it’s used a lot by the consulting/developers-who-blog community (as opposed to the working community). From a technical stand point, it looks a lot more flexible, in the sense that it isn’t an MVC (or uses any pattern for that matter), and all extensions are components that can be added on an as need basis. I’ve also read that Om renders faster than angular/ember/backbone. It’s probably a very good tool for building SPAs, but it also looks like it has a stepper learning curve – from my initial impressions of it – though like I said – I haven’t had the chance to use it on anything significant yet.

                                                  I wouldn’t say I’d use AngularJs for everything – I’ve noticed most of Google’s products haven’t been written in Angular, but if you are building a significantly important enough project – I’d probably go the route of not using any framework at all and use all custom JS period.

                                                  Honestly, articles like these are very silly. His opinion was based on a pretty shallow early impression of something he barely knows anything about – yet it still elicited 300+ comments. The fact that someone can write an blog article with a title like “you have ruined javascript” and evoke so much attention disappoints me.

                                                  I think skepticism is a natural healthy reaction towards any system, but I think the issue revolves more around the abuses of big business than the specific technology and technical issues at hand. Also, I think there’s way more excessive crap being built for ember these days within the open source community – from what I’ve noticed around the internet….

                                                  1. 3

                                                    From the technical point of view, Angular if far from being the worst. I came from jQuery land and used Angular for some slightly complex dashboards. Angular was great to use then. Mostly because at the time the documentation was mostly inexistant and we have chosen to use only a small part of what angular propose.

                                                    Now we switched from Angular to React, more precisely reagent. Which make a double jump, one from Angular to React the other from javascript to Clojurescript. While reagent is still not perfect it is far better to use. The actual application is far more complex than before. But with reagent we minimized our boilerplate, and administrative decisions comparatively to Angular.

                                                    So yes, Angular is good, and easy to learn. But reagent is better and easier to learn.

                                                    Actually if I had to choose, I believe I’ll give a chance to elm (elm-html + ports more precisely). Until here the elm concepts feel really adapted to big applications and interoperability with external resources.

                                                    So the technical point is, yes Angular is better than jQuery, lighter than ember, but React is better than Angular, reagent better than just React, elm apparently better than reagent.

                                                    Now, my remark wasn’t only valid for frontend frameworks. But more generally in the “framework vs library” debate, framework are great if they specialize on a narrow domain and don’t try to generalize their concept to all domains. If you don’t know where you’ll end up, chances are libraries are the way to go.

                                                    If you try to make your framework play out of its specialized garden, then you soon start to create a bunch of new organizational concepts. Generally at the end of the day, your framework will look like an insane administration.

                                                    While Angular is far from being the worst, there are better alternatives. Because I sincerely believe that elm is easier to learn and master than Angular (if you aren’t afraid by a new syntax).

                                                    1. 2

                                                      From a technical stand point, [ReactJS] looks a lot more flexible, in the sense that it isn’t an MVC (or uses any pattern for that matter)

                                                      You keep using that word. I do not think it means what you think it means.

                                                  1. 9

                                                    This week, I’ll try to integrate elm with highcharts to make reactive dashboard interface without reinventing the wheel. I already have a demo with one chart that subscribe to mouse events and display them in real time in a chart. The working demo is here: https://github.com/yogsototh/elm-highchart Until here, it is a great experience, the latest changes in the elm core libs are really welcome and the ports part is butter smooth.

                                                    I’ve started to write a blog post about the vim plugins I use. In particular, I find my actual Haskell vim environment just great. It detects errors at save time and I am generally told some good advices to make my program work. I also use quite a nice Clojure programming environment and I wanted to share it.

                                                    I don’t know if I could achieve much during Christmas time.

                                                    1. 4

                                                      my preferred part:

                                                      On the next page of the wizard, select the Web Application check box (1) and then select 3.0 from the Version list (2). (This is the version of the servlet specification to be supported.)

                                                      … and then …

                                                      Clear the Web Application check box. (This check box was selected automatically when you selected Application Server. Because we already have a separate module for developing the web interface, we don’t need web application development support in the module that we are creating.)