This sounds like a release! :)
Oops, didn’t realise there was a tag for that.. added now ?
What other strategies have you found for effectively reporting problems to large companies?
Blog posts and HN seem to be the only way ;-)
Disqus ignored this issue until I blogged it and a Google employee has now responded to my forum thread about this issue since I blogged this too!
Also in high demand if anyone has good ones: What strategies have you found for large companies to effectively receive problem reports?
This is a problem wherein developers don’t build in bug reporting or don’t make bug reporting easy because it’s seen as something that should only be used for beta testers or in-house.
This number is unknowable, but I can imagine there are plenty of people who find weird bugs or edge cases in Facebook, Youtube, whatever, but because there isn’t an obvious way to report the bug; they just never do.
As evidenced by app stores, there are lots of regular people who will write detailed accounts of bugs they are experiencing. Only instead of getting an email about it, you get it in the form of a public rant and a one-star rating.
Have you ever run a popular public service with a bug reporting address? I created security@[business].com on our About page so whitehats could privately report security problems. We immediately got dozens of support emails per day from users who wanted to unsubscribe from mailing lists, etc. It’s intractable. The signal to noise ratio is unbelievably low.
This is definitely a problem, but I think it can be reduced. For example, if the same page that had that email address on it made it clear how to unsubscribe, maybe it’d be used for that less?
I understand it’s hard supporting lots of people - for example if there was a contact email address at the bottom of every YouTube page it’d become quite expensive to triage; but I don’t believe making it impossible to contact anyone to report what might be serious issues is the best (or only) fix.
I have a connection at YT. I was able to repro, so I sent it along. Fingers crossed!
Oh wow, I just tested it, he’s right. How on earth did this get through QA?
My question exactly! I’m not surprised people aren’t convinced it’s a real bug until the test it; it’s pretty crazy :(
How do you know that it did get past QA? A QA Engineer’s job is to report on software quality, not to decide whether or not to ship a product. There could be a reported failure of a test case specifically for this issue, followed by an administrative decision to release the product with the failure anyway.
You might be tempted to ask ‘how on Earth did this get past the Developer who wrote it?’, but you can’t really assume that it got past that person either. Software is complex, and a small change made by someone else at a later date, somewhere far away in the code tree, could have an unintended effect on code that worked just fine when it was first written and tested.
It would be interesting to know what the ultimate root cause of the problem was. Its too bad that outsiders don’t usually get to see that information, because there can be a lot to learn from it.
A QA Engineer’s job is to report on software quality, not to decide whether or not to ship a product.
If the QA department doesn’t own the release, then they aren’t Quality Assurance, they’re merely Quality Information Reporting Staff or something.
I guess I’ve only worked at places where the QA org owns the release, so your comment really surprised me. There have been places that had NO QA at all explicitly, and those had as bad of quality as you would expect.
That’s the ideal situation, but is it common? Releases dates and feature lists are often very “political”, and at many places QA isn’t allowed an outright veto as a result. In games, for example, QA can at best argue for a release date to slip or alternately for scope to be cut given time/resources, but since release dates slipping or features promised in trailers not materializing are both seen as bad PR and seriously impact the entire marketing strategy, someone fairly high up in management would have to OK it, and there’s often substantial pushback. Management would weigh QA’s input as just one of many sets of pros/cons, and it ends up being a judgment call which is less-bad of the options available at that point: “miss the Christmas season”, “ship for the Christmas season but without promised features”, or “ship for the Christmas season with promised features, but they’re horribly buggy”.
Of course you can say games are “just entertainment”, but this tale from SGI, among others, sounds pretty similar.
That’s the ideal situation, but is it common?
I don’t know if it’s common (I suspect not), but where I work we do this! When I (as team leader) am happy with a release (all important items for that version complete, code reviews done, etc.) my responses is always that it’s ready to go when QA say so.
That said, we’re talking about YouTube here. I would be both surprised and disappointed if they chose to ship things with with QA having raised bugs like this one. My guess is that they didn’t know about this and something wasn’t tested thoroughly.
I’m sorry to hear that you’ve only worked at places like that. It’s like working with teams that don’t use source control, or that never do unit tests, or that can’t version a release, or that have no QA at all. Sometimes you’ve got to do it, but it is generally agreed that it isn’t a recipe for long term success.
A Quality Assurance team is an information reporting staff. Its right there in the name.
‘assure’ - 1. tell someone something positively or confidently to dispel any doubts they may have.
Some organizations think they can create a team that can force a high quality release by finding and eliminating all of the bugs, a team that refuses to ship until the bugs are all gone - a Quality Ensurance team.
ensure - 1. make certain that (something) shall occur or be the case
Forming a QE team is a bad gambit. The only way for them to be certain of a bug not going into production is to never ship code. If they do ship code, they become the scapegoat at the first sign of trouble. (“How on Earth did this get past the quality team?”) Its an inherently conflicted group.
Without going into a long dissertation on it, your business will be a lot better off, and you’ll ship better quality software if you have an honest quality reporting team (a QA team) providing actionable data up the chain of command.
The Capers Jones view of the world (inspections, static analysis, formal testing..) has lead to some of the highest quality software I’ve ever been a part of, and it has the research data to back it up.
Your straw man about having a bad culture in a company for how they treat QA in light of total engineering failure (bugs) is one that I have never experienced. I’d love to hear more about your experience and why you feel so strongly on QA not owning releases, or I may be confused about what bothers you so much about QA ownership of releases.
An alternative to Disqus is Isso, which is self-hosted.
One of the great things about Disqus is that you can use it on a “static” blog. My blog (the one hosting this article) is just GitHub pages with posts written in markdown. This has the advantage of being simple and free (and easily to cache/distribute on CDN etc.) but has drawbacks of not being able to have custom code like that.
When my blog was hosted on AppEngine I had self-hosted comments; but Disqus seemed like a much better option. Not so sure about that now though!
I’ve gone back and forth on that, but the solution my current blog is a note at the bottom of each post saying:
Comments welcome: firstname.lastname@example.org
This outsources the infrastructure to email, which already works, with the obvious drawback that the barrier for many people to emailing someone is higher than that for posting a comment. Although that might not be purely a drawback. :-) Another difference of course is that email is private, while some comments might be interesting to other readers, too. I partly remedy that by occasionally posting (attributed) updates at the bottom of a post if someone sends in something I think might be interesting for other readers, as in this example.
Besides not wanting to mess with running either a first- or third-party commenting system, the other motivation is that on a personal blog I feel some desire to keep it as a place for my own writing, not as a general third-party discussion forum attached to every page. So if someone sends in relevant comment I’m happy to post it (or a paraphrase), but I don’t necessarily want comments from random people arguing about tangents to be posted underneath my essays.
I’ve thought about not having comments directly (esp. when HN/Reddit/here usually get more comments than directly on the blog), but I do still think they add value. Not only do I get “Thanks!” now and then which lets me know people are finding my posts useful, but there’s often good discussion between people there.
I don’t get a lot of bad comments, so the only reason to remove them would be to get rid of the scripts but I think (hope) Disqus cares enough about its reputation that they’ll fix this and be more careful in future.
The discussion between commenters on one thread can lead to discovery of new ideas for those people or blog author. That’s essentially what happens here, on HN, etc. Doesn’t happen with email since the readers don’t know of each others' presence much less interesting comments.
Oh, based on the above I figured this was self-install and wouldn’t work for static sites. If it can be used directly from their site though, there’s nothing to stop them making the same mistake in the future? =D
I’m doing the same thing. I have static Jekyll blog, although now on Netlify rather than GitHub Pages because then I can use https with my custom domain.
I built and hosted my own IndieWeb Disqus alternative though. And it’s open for others to use: https://webmention.herokuapp.com/
And there’s other similar services that one can easily self-host. There’s even people who do automatic commits to their static page of any received comments, both from WebMention and through comments form. Been thinking of eventually experimenting with that as well and make my WebMention endpoint talk to my Micropub endpoint (another standard that’s now going through W3C) to submit any received mentions: https://github.com/voxpelli/webpage-micropub-to-github Some are already doing that with their respective endpoints.
I have static Jekyll blog, although now on Netlify rather than GitHub Pages because then I can use https with my custom domain.
My blog (hosting this article) is actually custom domain over SSL on GitHub pages (using CloudFlare to add the SSL). It’s not ideal, but was easy to add to the existing GitHub Pages site rather than migrating!
It uses WebMention (which btw now is a W3C Proposed Recommendation), which removes the need for embedding any authentication mechanisms like Facebook. Instead everyone writes the comments on their own blogs instead and pings my service which then retrieves the comment.
I’d never heard of this, this sounds really interesting - I shall have to read up! Thanks! :-)
without installing Linux
You’re already running Linux. ChromeOS is just a Linux distro with a patched kernel (like most Linux distros nowadays).
You’re already running Linux
It’s probably obvious I know this given in the post we use Bash, download the Linux version of Go and execute a bunch of Linux/Bash commands ;)
ChromeOS is just a Linux distro
There’s a huge difference between using ChromeOS as-is and installing a Linux distro a Chromebook. It might not be obvious to anyone that doesn’t use a Chromebook in dev mode, but it’s incredibly limited. Most people using Chromebooks in dev mode install some sort of Linux (doing so via Crouton seems very popular). There’s no package manager and most of the stuff you’d expect to be there is missing (eg. the Rust setup script fails because file is missing and setting it up manually fails due to no C compiler, which so far I’ve not managed to satisfy). Nobody builds binaries for ChromeOS Linux.
Let’s get our definitions straight. A Linux distribution is a software distribution around a Linux kernel. No package manager required to qualify. No minimum userland requirements either.
Nobody builds binaries for ChromeOS Linux.
I never said ChromeOS was not Linux. My article says “without installing Linux”. “Having a Chromebook” does not mean you have “installed Linux”.
The other stuff I added was just FYI about why making something work in dev-mode without installing Linux is actually interesting and is not just the same as making something work in Linux.
By “nobody builds binaries for” I meant “people that build software don’t build binaries for” not that third parties don’t do it. I trust a binary I download from rustlang.org (who are building binaries used by many thousands of people) much more than a third party making binaries that are used by a handful.
If you’re going to go into Dev mode, is there a reason not to use something like Crouton? It seems pretty nice to me, and I have a few different environments on my little Chromebook…
Just that it’s a bunch of baggage. Why install Linux if you don’t need to?
I did actually have Crouton installed until I powerwashed yesterday (had a weird issue with downloads vanishing). I like to keep things minimalistic - easier to get things back to how they are if anything goes wrong (and, less to go wrong!).
I like to keep things minimal and clean, so the ability to nuke chroots when they get hairy is pretty nice to me. I wouldn’t want to be experimenting with Go, and mess it up (say if I’m mucking about with the core), and then have to start all over in my native environment… crouton seems to make it easier to “get back to things […] if anything goes wrong”, at least in my mind.
I would say I do the same for work tho: I have a per-client per-assessment VM/hyve, and I nuke them when done (makes data clean up easy as pie), so it may just be an extension of that.
Yeah, I can totally understand all that (I liked using Crouton for testing stuff out), but for some stuff (eg. if it’s just downloading an archive with an executable binary like this one) it seems like overkill.
I’m not against it, there just needs to be enough of a benefit to justify the (albeit rather small) overhead.
I think that’s pretty fair; it’s actually spurred me to try a native, non-chrooted Go on my chromebook sometime this week (I’m working on some static analysis tools for it anyway, so it’s a good place to try). Definitely not my normal route, but why not try it?
It’s an understatement to say that this resonates with me. Yesterday was frankly a shit day for me, and about half of it was a write-off, precisely because I was dealing with software bugs. When you encounter a rapid series of them it’s incredibly disruptive to productivity and more than a little demoralising. Here’s a select sample of some of what I saw:
deleted inode referenced
This is one day, and not even a complete recount. It’s not just closed-source stuff, open-source stuff is often equally terrible. I’ve spent this morning rebuilding my VM post it’s total destruction yesterday, and while these days I’m more skeptical of VMware, it wouldn’t be the first filesystem I’ve lost due to rubbish, poorly tested code in the Linux kernel.
Most people don’t have the skills to debug buggy applications. It takes time and is frequently very difficult. Particularly when it’s not your application. If you do try and get to the bottom of the issue, prepare more often than not to enter a rabbit hole that would make Alice herself cry. Yes, this is a rant, but for those of us who try to maintain stable systems and do the same for all of those systems we manage (or develop software for), the current state of software quality is frankly appalling.
My impression is that Linux quality has fallen a lot in recent years. Since switching to FreeBSD I’ve been much happier and my purely subjective impression is that it’s better/more reliable code.
Reading your post reminded me of the time I lost most of a day trying to figure out why our application was entirely broken a few days after I’d upgraded a component, despite me thoroughly testing it.
It turned out to be a bug in only the paid-for version of PostSharp, which you “got for free” on a Monday:
It took a lot of people telling them how stupid this was on twitter before they finally agreed to change how this works to avoid inconsistent builds across different days :(
Addendum: For any SysAdmins here, consider configuring all of your client systems to report crash events. On Windows this is fairly easy: setup a Windows Event Collector on a suitable server and deploy subscriptions to clients to push “Application Crash” events. Check the pulled events now and then, or on small networks, maybe even email on each new crash event.
It’s frequently insightful, genuinely useful, and horrifying. You’ll also find way more crashes than you likely realised, as you’ll see all the crash events for background processes (services, etc…) that won’t typically present a crash dialogue to the user.
He seems to have some points, but I don’t really like how he calls libraries like Angular “fly-by-night”.
Also, I think React probably has some bad points just like any other language. Just showing that other frameworks have bad points doesn’t mean the one you advocate does not.
Lastly, while he shows the difficulties these other frameworks have, he doesn’t actually show how React or Dart solves these problems.
IMO, many devs could use a healthy dose of irreverence, if only to break the status quo.
The author advocates for a simpler approach to programming. Rather than suffering CoolFramework.js' weird query syntax, why not just do what it was trying to do in the first place manually? This may require you write a bit more code, yes. It’s not as sexy. It’s akin to poring over an ORM’s docs for hours trying to write one weird query when you probably could write it in straight SQL and have it be cleaner.
I’m planning to; we;we’re prototyping both React and Dart (which we’d like to use in a similar way to React). I didn’t block because I’m stuck for a solution, I was just dumping my thoughts on the frameworks that are becoming “mainstream”, which I think aren’t pushing web devs in a good direction :(
I hear you, I’ve penned thoughts on this topic as well:
There’s an alarming level of intellectual deference to frameworks. It’s like devs assume that if you throw enough “best practices” at a problem then it eventually results in a best solution. This is harmful because it acts as a cognitive killswitch of writing better code, often by an implicit appeal to groupthink.
The worst part is it’s difficult to call out this laziness without being labeled negative or whiny. This, to me, is the biggest indicator that there’s a problem: the utter lack of introspection.
The worst part is it’s difficult to call out this laziness without being labeled negative or whiny
I can agree with this; see comments on Reddit! ;)
Sorry, the “fly-by-night” comment was meant to be tongue-in-cheek, but I guess that’s not very obvious ;(
I probably should’ve included a React sample; but my post was mainly about what we’re doing wrong; I wasn’t really advocating a solution (it’s easy to point out flaws, harder to solve them!).
Ah ok, understood. Don’t take it too personally, maybe my sense of humor is just out of whack. ;)
I think a React or Dart sample may have been beneficial, even if only for contrast.
I would like you to know that I actually enjoyed reading the article, and I agree with the points you make. I just wanted to point out what I found to be some inconsistencies. Reading my last comment, I think maybe I was too harsh – I did not mean to come off that way. Sorry.
It’s my fault. I write my blog posts as if readers know who I am; it’s rare they’re read by thousands of total strangers who don’t know my personality! Of course, I’m not going to change it; it’s my blog; get your own! ;)
Reading my last comment, I think maybe I was too harsh
I didn’t take it that way at all; I think the comment was entirely fair!
I really wish I’d detailed React better in the post; because so many people are reacting the way I did when I first saw it. I posted this on twitter:
Is this how you would choose to write a TODO app? So much noise and bizzare syntax/funcs/properties :/ #React
It looked horrific to me. However, after one of the Facebook guys told me “Give it 5 minutes”, I had a real quick play with it; and I was very quickly sold. I’m a big fan of functional programming, and this really struck a chord (component rendering feels very much like a pure function, which has many advantages).
I encourage everyone to play with React for 5 minutes; first impressions are very misleading!
The client side bits remind me of Java applets and liveconnect. Like if you compiled any language into java bytecode and just made it interface with the DOM/BOM through plumbing. That’s a “common” VM and intermediate format.
I think the problem with the argument (and not to pick on the author… it’s a broadly discussed topic) is that implementers would have their own versions of this common vm. Their own versions of this plumbing. “Write once, run anywhere” fell short of its vision partly because of so many implementation differences.
It’s not just about typing; it’s also about choice and sharing code.
Choice: On the server we have choices, and people pick different languages - there are merits to them all, whether they’re technical, based on familiarity, or even just available libraries.
I don’t think different VM implementations would be a big problem. Mono seems to be doing pretty well; and you never know, people might share code (until recently, Chrome used WebKit for rendering)!
So the reason we have so many languages on the backend is because as developers we have 100% control over the backend. We own the server-side so we have luxury of doing what ever we want on that domain.
We wouldn’t have different interpretations of languages if we had a common VM. Browser vendors wouldn’t interpret the different languages; only the intermediate format. Similar to the JVM and CLR; these seem to work well.
(Though I don’t underestimate the effort involved in this, even before you consider having multiple companies involved!)
So we’d have to standardise on the intermediate language for the VMs? I think you’d see the similar situation that currently plagues Mono, it’s playing catch up to Microsoft’s CLR. Imagine if we had Microsoft/Apple/Google on different levels of the VM implementation?
Having a VM would improve a lot of things, while making existing things not worse than they are.
Excellent point, I think I got carried away. I agree a VM approach would be an excellent solution, I’m just concerned with a cross platform VM that isn’t controlled by one company.
There’s little info on what’s coming in the next version of F#; yet we know quite a lot of language features coming in the new versions of VB and C#. It feels like C# and VB are charging forwards at a much faster rate than F#. Do we want to get off the fast train and board the slow train?
I haven’t used any .Net language, but in any way is this because F# is already so flexible it doesn’t need much? I am a user of Ocaml and Ocaml simply doesn’t need to change much. Most of the value added when it comes to new versions of Ocaml is things behind the scenes. Sometimes nice little syntax additions but for the most part they aren’t a big problem. Compared to a language like Python or Java, where every new interesting thing you want the language to do generally requires a new version of the language, Ocaml allows users to do quite a bit of growing without modifying the language.
That’s true in some ways (in fact, many of th announced vNext C# features are to add things F# already does, or make functional concepts easier); but there are still a huge bunch of things posted at http://fslang.uservoice.com/ that would make sense to add.
That said; these are language features. If you look at tooling support, then F# is far behind (not just for not being part of Roslyn, but the VS editor is basically a text editor and little more). There’s no word on any improvements there either :o(
It all makes me wonder how much investment F# is going to get over the coming years compared C#. As more functional ideas come to C#; where will we be in 5 years time? Will C# do most of what we want from F#, but have tooling support that is light years ahead?
Sending this to my boss, it’s so true :(
Sometimes people think the cost of doing a new feature is the only cost; but simply maintaining it for years to come takes time. If you keep letting your product get bigger and bigger, it’s going to cost more just to “do nothing” to it.