-
I spent some time thinking about this as I had a database with a high rate of writes. I didn’t get to come up with a solution unfortunately. Copying the production database is certainly useful but it comes with caveats.
For example, if the database is sharded then copying a single shard has limited utility, and copying all of them is going to get expensive quickly.
The other issue I had with using production data for testing is that sometimes it shouldn’t be made available to all and sundry due to privacy concerns, so there either needs to be a process for scrubbing it (error-prone) or we’re back to generating synthetic data.
Finally, in a high write rate scenario there are additional load issues for the application server(s) if you’re trying to write both to the production DB and a copy. It’s another layer of complexity. It’s possible to avoid that by restoring a backup and doing some sort of synthetic load simulation rather than reproducing actual real time writes, but then it’s not so easy to make it realistic and keep it that way over time.
-
Author here.
For example, if the database is sharded then copying a single shard has limited utility, and copying all of them is going to get expensive quickly.
Our database is sharded. We have 10s of thousands of logical shards spread across a much smaller number of physical machines. Almost all the queries we care about are ran across every physical machine. For testing optimizations, copying a single physical machine is the 80/20. It allows us to tell whether the optimization does make the queries we expect to get faster do get faster, and that no other queries degrade. For this to miss regressions, there would need to be a high number of shards on other machines that behave differently under the optimization.
The other issue I had with using production data for testing is that sometimes it shouldn’t be made available to all and sundry due to privacy concerns, so there either needs to be a process for scrubbing it (error-prone) or we’re back to generating synthetic data.
Why would data in shadow prod “be made available to all”? We have the same restrictions on accessing the data in shadow prod as we do for production.
Finally, in a high write rate scenario there are additional load issues for the application server(s) if you’re trying to write both to the production DB and a copy.
For us, our writes do not need to show up instantaneously. Ideally they show up within a few seconds, but it’s okay for us if they periodically take longer. All our writes go through Kafka before they go into our database. Our service that reads from Kafka then writes to both databases. If shadow prod slows things down, all writes will be queued in Kafka. This way, we don’t drop any data. There will be latency between when data shows up in production, which while less than ideal, is acceptable. We’ve discussed having separate Kafka consumers, one that writes to production and one that writes to shadow prod. This would prevent shadow prod outages from causing latency in production. There hasn’t been need for doing so, so we haven’t implemented it yet.
-
Thanks for the extra information, it makes sense.
Why would data in shadow prod “be made available to all”? We have the same restrictions on accessing the data in shadow prod as we do for production.
I think it makes sense for performance testing and general error/no error tests on migrations. In my case, we occasionally had fairly complex data transformations going on during migrations, and if they went wrong or even while writing the migration, we’d need to see what happened to the data. It would have been hugely useful to run this on prod data, but then privacy issues became a stumbling block.
-
-
-
I’m another person who skipped college and went straight into industry after high school. Based on my experience, I recently wrote a post describing what approaches I found to work when looking for a job. I hope to help people who are uncertain about going to school like I was realize they do have other options.
-
My company has been looking for some junior and associate-level engineers and we get a lot of applicants from different bootcamps. It’s heartbreaking interviewing them. Usually they’re coming out convinced that while they’re definitely not fully-qualified as software developers, the bootcamp at least got them to entry-level. More often than not, though, they’re unqualified for even that.
With one particularly awful one, some of us were suckered into mentoring for their students. By the end of the time there the students still couldn’t write a
maxfunction. The bootcamp told them they were all ready for full-time employment. Some of the students had moved from other states to attend.I think the students are the biggest victims of this all. Coding isn’t a skill you can learn in twelve weeks, but many predatory camps, which have cachet as authorities and insiders, push that narrative to outsiders. And the rest of the tech industry tolerates it.
-
I’ve had a similar experience to you with some bootcamps, but they’re not all created equal. The Recurse Center in particular has continually impressed me, for example; I don’t know if their teaching methodology is different or what, but we’ve gotten quite a few candidates who may not be ready for full-time, but who can definitely start at what I might call an advanced intern level, and who have a really good chance to progressing to a normal full-time position at the end of fur months.
That said, it’s worth noting that Khan Academy has a specific program for this situation (we call them fellowships), and our creation of that program was a very deliberate move to extend our education focus internally, in part by accomodating the bootcamps. I’m not sure if even some of the more liberal-minded places I’ve worked in the past would’ve been able to justify doing that, and I’m not sure if candidates from even the good bootcamps, like The Recurse Center, would do well otherwise.
-
The Recurse Center (fka the Hacker School) is not a bootcamp at all (the name change was intended to make this more clear). It recruits students who already know how to program and may already work in the industry and takes more of a professional development/continuing education approach, giving students the time and resources to explore advanced topics that may interest them, or to fill gaps in their existing knowledge.
-
(I attended the Recurser Center in the summer of 2015)
To make it clear, the Recurse Center can’t really be compared to a boot camp or a traditional school. The easiest way I’ve found to describe the Recurse Center is that they take 80 people interested in programming, put them into a room together for three months, and have them work on whatever projects they want with little to no oversight.
There are quite a few differences between the Recurse Center and other programs. First of all, they target all sorts of programmers. AFAIK, the only qualities they screen for are a love of programming and self-sufficiency. This leads them to getting programmers all across the experience spectrum. Some people have been programming for only six months, others for 20+ years.
Second, there is no coursework. “Recursers” (someone who attends the Recurse Center) get to work on whatever projects they want with whoever they want. It’s completely self directed. The projects people work on range from building a programming language, to writing an implementation of Paxos, to learning Haskell.
Third, the program is free to attend. The Recurse Center makes money by matching up Recursers with companies and charging the companies a recruiting free. To make it clear, the recruiting aspect of the Recurse Center is completely optional. If you aren’t looking for a job after the Recurse Center, they won’t push you to look for one.
Overall the Recurse Center is completely different from any kind of boot camp. If you are interested in it, I highly recommend applying. It’s a really awesome experience.
As for the recruiting from the Recurse Center aspect, they are also great. 6 of the 20 full time software engineers at my current current company, Heap, were sent to Heap by the Recurse Center. They are by far the best recruiting channel we use.
-
The Recurse Center needs to be in more locations than just NYC. With a family, it is extremely expensive and difficult to secure accommodations for 6/12 weeks, and there are also issues of what the family would do during the day.
-
-
-
-
-
I think the students are the biggest victims of this all. Coding isn’t a skill you can learn in twelve weeks, but many predatory camps, which have cachet as authorities and insiders, push that narrative to outsiders. And the rest of the tech industry tolerates it.
I’m not sure what kind of coding we speak about. General? No. Specialised? Yes. I know quite some people that learned the coding needed to support their current task (e.g. a statistical model) very fast. The problem is that a bootcamp rarely facilitats that, they have a one-size-fits-it-all curriculum.
In many cases, I have the impression a trusting employer and some individual coaching at that location would help much more.
-
-
-
peter
1 year ago
|
link
| on:
Do we need specialized graph databases? Benchmarking real-time social networking applications
I really would like to see the queries and explains for Postgres and Virtuoso, or at least know what algorithm was attempted. For Postgres to be that much slower, there has to be some algorithmic issue.
-
-
I saw that, and wasn’t 100% sure that’s the one they used. For example, does that query work on both Postgres and Virtuoso?
If that is the query, it’s terribly inefficient. They copied the Postgres docs example to enumerate an entire subtree, and modified it to return only the shortest path. So to compute the shortest path between A and Z, it starts at A, lists all of A’s connections, and scans them for Z. It then non-uniquely lists of A’s second degree connections. So if A knows D via B and C, it will list A B D and A C D, and then double-search for D in the next stage. This is particularly bad in social networks, where 2nd degree connections are likely to also be 1st degree connections.
But this isn’t a tree, it’s a graph. Instead, they should only keep the shortest way to get to any point, and work from there. So if they have A C, and they discover A B C, we should throw that out and keep A C only.
-
-
-
This article seems to be a bit fear-monger-y to me. This is (A) documented behavior in the PostgreSQL manual, and (B) If you’re using a SQL database it is your responsibility to determine the level of isolation that your query should need.
I typically prefer coalescing multiple selects into a more complex query, and only very rarely need to fall back to multiple selects where higher levels of isolation would actually be necessary.
-
This is (A) documented behavior in the PostgreSQL manual, and (B) If you’re using a SQL database it is your responsibility to determine the level of isolation that your query should need.
While it is documented, I find most people aren’t familiar with it. I could see half of the examples being completely unexpected to someone who knows Postgres, but not in depth.
Also, I find the documentation in the Postgres manual around the different anomalies to be unclear. I believe this is due to historical issues such as the SQL standard being designed for lock based databases, as well as the SQL standard itself being fairly unclear. See A Critique of the ANSI SQL Isolation Levels.
-
-
The content of this post is about the same as PG’s own documentation, however the author left out that you can change the isolation level even as far as serializable, which removes all of these issues if you don’t mind paying the cost.
https://www.postgresql.org/docs/9.1/static/transaction-iso.html
-
the author left out that you can change the isolation level even as far as serializable, which removes all of these issues if you don’t mind paying the cost.
My next two posts are going to cover the main ways of avoiding these issues (row level locks and different transaction isolation levels), and how each of them affect the different examples in the post.
Serializable also doesn’t exactly solve all of the problems. For the lost updates, skipped updates, and serialization anomaly examples, serializable will abort one of the transactions, which is less than desirable behavior.
-
-
So the tl;dr is that an insert heavy workload benefited from a batch write?
These tools are awesome, and deep diving into the source code to find the specific things that make batching more efficient is very educational (apparently some table related metadata is cached).
For applications that can tolerate bounded latency on their writes (even if it’s only 1s) batch writes are usually a huge win.
-
Yes, but it was unclear that was the case at the beginning. We had (incorrectly) assumed CPU was mostly spent on evaluating the partial index predicates. Based on that assumption, we thought batching wouldn’t have had much of an effect. It wasn’t until we actually examined what the CPU was being used for did we realize that our assumption about CPU usage was completely wrong and that batching would actually have a dramatic impact.
-
-
[Comment removed by author]
-
-
There is a possibility someone can do this, somewhere on this planet. They probably wouldn’t be looking for work ever :)
Nah, they’re probably in that 150+ IQ set that struggles with corporate bullshit.
I’ve seen quite a few IMO medalists (at least five) get absolutely raped in the GoogFace companies. It’s at the point where I advise them, if they’re set on leaving academia, to go to finance where their talents will actually be appreciated. You put a legit 150+ in a hedge fund, and he’ll never be a “rainmaker” who goes out and gets new investors, but he’ll ease into a quant role and do it well. He might make “only” $500k/year as a senior quant, while smiling bullshitters make millions… but that’s better than getting PIP’d because you didn’t do enough “user story points”. You put a legit 150+ in a GoogFace, anywhere other than the research division, and it usually ends badly.
Of course, this phenomenon may exist because the 150+ who end up in regular engineering roles already have serious health problems (or, as in certain cases, had health problems in the past) or social deficits, because otherwise they’d be somewhere other than “mainstream eng” at a tech company.
GoogFace and VC startups are all about pedigree, not raw intelligence, because people who have the ability to detect the latter are extremely rare outside of the R&D divisions. In the tech industry, a legacy-admit Stanford kid will dominate a 150+ from Penn State. Whereas in the hedge fund world, the 150+ has a shot of getting spotted and fast tracked, while the legacy Stanford kid probably won’t get hired. Greed isn’t always good but it’s better than self-protecting corporate stupidity… which is why Wall Street is more hospitable to top talent than Silicon Valley.
-
-
I’m an Ocaml user and, except for a few rare conditions, I’ve found I much prefer a result type to exceptions. My response will be based on Ocaml which may not be the same as F# so if they don’t apply there then ignore it.
Some points I disagree with the author on:
AN ISSUE OF RUNTIMEI didn’t really understand the example here. How is the author accessing an optional value? In Ocaml we have to use an accessor that would throw an exception if the value is not present or pattern match the value out. This doesn’t seem to have anything to do with exceptions or results, just an invalid usage of an option.
AN AWKWARD RECONCILIATIONThis is the case in Ocaml as well, which is why many libraries try to make exceptions never escape the API boundary. But writing combinators for this are really quite easy. A function like
BOILERPLATE(unit -> 'a) -> ('a, exn) resultis available in all the various standard libraries for Ocaml.The author should be using the standard applicative or monadic infix combinators. Maybe F# doesn’t allow that. In Ocaml the example would look like:
WHERE’S MY STACKTRACE?let combine x y z = pure (fun x y z -> (x, y, z)) <*> x <*> y <*> zThis is the one I disagree with quite a bit. If I am using exceptions then yes, I want stacktraces, because it’s a nearly unbounded GOTO. But the value result types give me is that I know, using the types, what errors a function can have and I have to handle it. This makes stacktraces much less valuable and the win of knowing what errors are possible and being forced to handle them. I’d much rather have this than stacktraces.
THE PROBLEM WITH IOThe problem here doesn’t have anything to do with exceptions, it’s that the return type should be a
STRINGLY-TYPED ERROR HANDLINGresultwhere theErrorcase is a variant of the various ways it can fail. Ocaml makes this much much easier because it has polymorphic variants.Yeah, use a variant not a string.
INTEROP ISSUESThis can indeed be a problem. It’s also a problem with exceptions, though.
-
-
-
WHERE’S MY STACKTRACE?
This is the one I disagree with quite a bit. If I am using exceptions then yes, I want stacktraces, because it’s a nearly unbounded GOTO. But the value result types give me is that I know, using the types, what errors a function can have and I have to handle it. This makes stacktraces much less valuable and the win of knowing what errors are possible and being forced to handle them. I’d much rather have this than stacktraces.
This is a case where you can eat your cake and have it too. Java has checked exceptions which the compiler enforces are handled. When call a function that can throw a checked exception, the calling a function either has to handle the exception in a
tryblock, or include in its signature that it can throw an exception of the specified type.You can also do the opposite and add the stack trace to the result type. Most languages provide some way to obtain a stack trace at runtime, so all you need to do is attach the stack trace to the error when it is instantiated.
-
Checked exceptions in Java are a nice experiment but a rather colossal failure, unfortunately. Since the compiler cannot infer checked exceptions you have to retype them all out at each level and it becomes unwieldy. The situation is even worse with lambda’s where one has to turn a checked exception into an unchecked one.
-
Is it simply type inference on function declarations that you see as the difference here? I am curious because as a Java programmer by day, I don’t see a ton of difference between “
-> Result<FooVal, BarException>” and “FooVal someFunc() throws BarException { … }”.Granted the implementation is quite different (unwinding the stack and all that), but is it simply ergonomics that makes the latter a “colossal failure” in your mind?
-
No, the difference is that results are just types and values. From that you get all the great stuff that comes with types and values. For example:
- Type inference. I only specify the types of my functions at API boundary points.
- Aliasing types. If I have a bunch of functions that return the same error I can just do
type err = .....rather than type all of the errors out each time. - They work with lambdas!
- They work with parametric polymorphism. I can write a function like
'a list -> ('a -> ('b, 'c) result) -> ('b list, 'c) result. - And, probably most importantly, it does not add a new concept to the language.
That checked exceptions do not compose with lambdas in Java basically tells me they are dead. All the Java code I’m seeing these days makes heavy use of lambdas.
-
Gotcha, thanks for the reply. I don’t disagree strongly, but I feel like what you are arguing for is Java, minus checked exceptions, plus more pervasive type inference, plus type aliases, plus several other changes. Which, that’d be pretty cool, but I think at this point we’re discussing sweeping language changes as opposed to the merits of checked exceptions strictly.
For example, simply replacing checked exceptions in modern Java with use of a
Resultwould (at least as far as I can imagine) still result in a lot of verbosity. You’d just be typing “Result<Foo, Bar>” a lot as opposed to typing “throws Bar” a lot.Not to be overly argumentative or anything. But “colossal failure” seems a little strong to me! :)
-
-
-
-
-
In Go:
type X struct { y Y } func (x *X) gety() *Y { return &x.y }Like C and C++, it’s still possible to return an interior pointer, but unlike those languages doing so is still guaranteed to be memory safe because of garbage collection. This might mean that your object ends up being punted off to the heap because the *Y outlives the X, but the actual access to the pointer carries no extra cost.
-
-
-
-
-
I’m surprised Java doesn’t handle that correctly, but I’m no Java expert. In languages like Lisp, ML, etc. it works fine, and it’d be surprising if it didn’t. Those kinds of high-level GC’d languages generally have managed/tagged pointers that they communicate to the GC, using e.g. a pointer map (a technique dating to Algol 68), which should mean handling derived/interior pointers works fine, and it’s pretty much a bug if any legal reference to an object gets collected while the reference is still live, no matter what else it sits inside of. Even the textbook GC implementation given in Andrew Appel’s Modern Compiler Implementation book discusses how to handle derived pointers (in Section 13.7, “Garbage collection: Interface to the compiler”).
-
I’m certainly not an expert on GC design but my understanding is that the JVM uses the fact that interior pointers are illegal to perform some optimizations. Google came up with http://forum.dlang.org/thread/o5c9td$30ki$3@digitalmars.com which looks like a pretty good and recent discussion of how interior pointers limit the possible optimizations.
-
Sure, but I think it’s a case of finessing the meaning of “legal reference”. If you make it illegal/impossible to construct an interior pointer, or make it illegal/impossible to hold one without holding a pointer to its container for a superior duration, then you can rightly say that your GC handles all legal references properly, while ignoring interior pointers and having generally less overhead.
-
-
I think the this in
Most GC systems do not support this at all (e.g. Java)
is unclear. I believe you are referring to interior pointers, which aren’t a thing at all in Java. I believe mjn thought you were referring to the general pattern of returning a pointer to an instance variable.
-
-
-
-
-
Does anyone know what is the current state of the art of profiling tools is? I recently had the opportunity to use JProfiler, a JVM profiler, for my work. It provides the same functionality Spacetime provides and much more. One of the big differences is JProfiler attaches to a running process, so you can interact with the application while profiling it. Similar to Spacetime you can inspect what objects are currently filling up memory and the stack traces that generated those objects. JProfiler also lets you poke around the heap and see what objects a given object points to, and what objects point to a given object. One JProfiler feature I found particularly useful was being to visualize the state of each and every thread over time (at what times each thread is busy/idle/waiting on i/o). This allowed my team to find the bottleneck in our a system. We were able to see that all of the threads of a particular thread pool were almost always waiting on i/o, and discovered the thread pool wasn’t sized properly.
-
Please don’t pay attention to TIOBE for determining language popularity. It is literally languages ranked by how many links come up when you Google the language.
-
I didn’t know about
DEFERABLE. That’ll be useful in the future. At work, we have a script that moves data between different tables. It has an intermediate state in which some fkey constraints would be violated. Since we didn’t know aboutDEFERABLEat the time, the script drops the fkey constraints, does its job, and then re-adds the constraints, all in a single transaction.The script one time caused a pretty serious production outage. One of the members of our support team, who had a basic understanding of transactions, copy and pasted the script into psql. He executed everything in the script except for the final
COMMIT, and then double checked everything to make sure it was okay if he ran the script. Because modifying a constraint on a table grabs a lock that blocks reads to the table, queries to the table weren’t finishing. This caused our connection pooler to quickly become saturated with queries to the one table. The connection pool saturating then all queries to the db to hang waiting for a connection. Since the db was down down, the support person decided not to execute the finalCOMMIT. If he had executed theCOMMIT, the db would have come back up. Since the outage, we now require all administrative commands to be slack commands. While the script does work, the right way to do it is withDEFERABLE.-
Oh, and I hadn’t noticed that
DEFERRABLEcould be used on foreign key constraints. :) The docs confirm:Currently, only UNIQUE, PRIMARY KEY, REFERENCES (foreign key), and EXCLUDE constraints are affected by this setting.
Thanks for pointing that out!
-
-
A nice overview. Some things that jumped out to me:
-
pg_blocking_pids. You used to have to do a gnarly select onpg_stat_activityto figure this out; this seems much better. -
jsonb_insert. Updating jsonb has been painful in the past, glad to see it get a bit better. -
pg_stat_progress_vacuumseems like it could be nice for monitoring. - parallel sequential scan (and apparently aggregations too).
- I can’t quite figure out what the deal is with the “Avoid Full-Table Vacuum” bit but it certainly sounds useful.
-
I can’t quite figure out what the deal is with the “Avoid Full-Table Vacuum” bit but it certainly sounds useful.
Postgres periodically has to run what is called a “freeze vacuum”. Previously a freeze vacuum would require reading all of the data for a table. In 9.6, they changed it to read only a subset of the table.
Freeze vacuums are necessary because of how Postgres handles transactions. When processing a query, Postgres will ignore every row that was inserted after the query started, or deleted before the query started. To do this, in every row, Postgres keeps both the id of the transaction that inserted the row, and the id of the transaction that deleted the row. Since transaction ids come from a counter, the two transaction ids in the row, along with a list of transactions that were running when the query started allow you to determine whether a row should be visible to the query. If the transaction that inserted the row wasn’t running when the query started, and the id of the inserting transaction is smaller than the id of the transaction of the query, the row was inserted before the query started. Similarly logic is used to determine whether the row was deleted after the query started.
The problem with this scheme is that transaction ids are only 32 bits. To work around this limitation, Postgres will periodically cleanup old transaction ids, so it can reuse them. When the newest transaction id reaches 2^32, the transaction id will wrap back around to the smallest transaction id (I believe 3 is the smallest since 0-2 are reserved for special purposes). By the time this happens, Postgres will have already replaced everywhere the transaction id 3 is used with a special transaction id (I believe 2) that represents “a really old transaction”.
Making sure old transaction ids are no longer mentioned is done by the freeze vacuum. Once the difference between the oldest id still used in a table and the newest transaction id reaches some threshold, a vacuum freeze will be triggered on that table. The vacuum freeze will replace all transaction ids older than a certain point with the special transaction id.
The vacuum freeze used to read every single page for the table and check every single row on that page. This means if you ran a vacuum freeze twice, it would read all of the data for the table twice, even though the second freeze was a no-op. The way they fixed this in 9.6 is by externally keeping track of what pages have only frozen tuples (tuples with the special transaction id). When the freeze vacuum freezes every row on a page, it will set the bit for that page. When a new row is inserted onto that page, or a row from that page is deleted, the bit is unset. The freeze vacuum will then skip over pages for which the bit is set since it knows that all of the rows on those pages are already frozen. If the table is append-only, this means the freeze vacuum won’t redo any work previously done.
-
-
The people who work on evil software like this need to be ostracized by the rest of us. (Yes, it is evil to deploy software into the wild against unwilling and non-consenting “users” two days after receiving constructive notice by the SF Bicycle Coalition that the software constitutes a deadly hazard to cyclists.)
-
There’s an interesting aspect to this, though, in that while releasing safety-related software like autonomous vehicles before it’s mature and stable is capital-B Bad, the development itself of the software is not (as opposed to, e.g. creating databases of minority citizens so they can be tracked). The premature release could have been (and, if Uber’s software developers are anything like any of the ones I’ve worked with, probably was) demanded by management over the protests of developers.
-
The premature release could have been (and, if Uber’s software developers are anything like any of the ones I’ve worked with, probably was) demanded by management over the protests of developers.
Very likely so. It doesn’t make me less uncomfortable.
When we say “Uber is an evil company”, we’re not saying that every programmer who works there is evil. That would be nonsense. We’re talking about the management. However, if someone works there and has the option to work elsewhere, then that person is complicit in the conduct of the company’s management. Software engineers, speaking of the group in aggregate, don’t care enough about ethics to do something about their management. They’ve had years and nothing has happened.
This industry of ours is starting to do serious harm, and not because programmers are bad people. We’ve built Weapons of Mass Unemployment and we’ve contributed to multiple misinformation campaigns that may have influenced elections, in our country but also abroad. It also has the potential to do a lot worse.
-
here is item #1 from the association of progressional engineers and geoscientists of bc
(1) hold paramount the safety, health and welfare of the public, the protection of the environment and promote health and safety within the workplace;
https://www.apeg.bc.ca/For-Members/Ethics,-Law-and-Conduct
I say this a lot but the software industry needs to stop being a bunch of unprofessional hacker types. Uber is on my shit list for violating ethical and legal principles from top to bottom. The programmers can’t use managers as a scapegoat. It’s their responsibility too!
-
Can you clarify what you mean on weapons of mass unemployment?
-
-
-
There’s an interesting aspect to this, though, in that while releasing safety-related software like autonomous vehicles before it’s mature and stable is capital-B Bad, the development itself of the software is not (as opposed to, e.g. creating databases of minority citizens so they can be tracked). The premature release could have been (and, if Uber’s software developers are anything like any of the ones I’ve worked with, probably was) demanded by management over the protests of developers.
That aspect ceases to be interesting, though, if the developers continue to participate in development after they know their work has been deployed in the wild with disregard for human life.
If you know management is using your work to endanger peoples' lives, and you continue to enable management to do so, you’re, in my view, as morally culpable as they are.
-
Yes, certainly. I am not usually one to call for software development to be more akin to other kinds of engineering–but I do like the concept of an “engineer of record” with the power to make final calls on issues like this, and the corresponding responsibility. E.g., in the case of the Hyatt Regency walkway collapse, the contractor who proposed the fatal changes to the plan was not held responsible–the engineers who approved those changes were.
As it stands today, in most software development environments I’ve seen, there is no clear responsibility in these cases. That has to change.
-
As it stands today, in most software development environments I’ve seen, there is no clear responsibility in these cases. That has to change.
Clearly the market disagrees, and right or wrong that’s what counts–not our handwringing as professionals.
-
-
-
Yes, this should be supported by ethical and community boundaries, but this is also why we have laws, I guess? The CA DMV wrote a letter telling them to stop, and I am expecting them to issue heavy fines if they try to keep driving.
In the event that someone actually does get hurt or killed by an poorly written self-driving car without the permit necessary to be on the road, I also expect the California legal system to look very poorly on Uber’s behavior.
-
Software can’t be “evil”. People can be evil, and they can create and use software to do evil things, but I don’t think that’s happening here.
Uber’s software is buggy and not very good at driving, but I’m willing to give the devs the benefit of the doubt that they were working in good faith, trying to make the software work correctly and safely for everybody on the road.
If anybody is “evil” in this situation, it’s the people who knew the software didn’t work or didn’t meet regulatory requirements, but insisted it be released anyway. I’m sure there were some developers involved, but it’s probably not all of them.
-
Claiming software can’t be evil is naive. Yes, someone had to create that software, and may have had malicious intent, but the software itself is still doing malicious things.
The software itself is still “being evil”. It can be changed and corrected, but it’s still arguably being evil. The argument that a tool created by humans can’t be evil is silly.
-
The argument that a tool created by humans can’t be evil is silly.
Claiming that a collection of algorithms which knows neither Right nor Wrong and which cannot make a choice about how it reacts even were it to have that knowledge is silly.
On a related note, that is the precise reason why autonomous software is so distressing–it is not Evil, but amoral. There is nothing to appeal to, no greater sense of purpose or destiny: just a bunch of state and instructions.
-
On a related note, that is the precise reason why autonomous software is so distressing–it is not Evil, but amoral.
At the risk of just discussing semantics, software can (like any other lifeless object) encode ethical rules. To take one of Bruno Latour’s examples – a speed bump encodes a long process of mediation of ethics (people should not drive too fast in Urban areas, because people’s lives are worth more than other people’s time), laws, violation of those laws, etc. Or to take another example, ‘the wall that Mexico is gonna pay for’ is an encoding of the moral point of view that Mexicans have no right to move to the US relatively freely. So, I think it is fair to consider lifeless objects can be moral or immoral. Obviously, until we have full AI, it’s of a different nature than in actors that have free choice.
-
So, I disagree (surprise).
There’s an important idea of choice involved in ethics, and the speedbump example you have provides a great case study.
You contend that the the speed bump encodes somebody’s ethics (“we have to slow down to save lives”), when it could just as easily encode greed (“my cousin has a speedbump company and needed a handout”) or design error (“this street should’ve had 8 speedbumps per 500 yards, a typing error resulted in 5 per 500 yards”). If the same embodiment of ethics can also come about through nonethical means, I question if that embodiment can be claimed to unambiguously represent morality manifest.
A similar problem I have is that your argument naturally extends to, say, written rules. Imagine that there is a holy document that says “We must always eat < 1 baby per day”. Due to a typo in printing, it instead is reproduced as “We must always eat > 1 baby per day”. Clearly, one of these outcomes is “evil”–at the the same time, it is so through no fault of the book itself: the book was cast with what we consider an evil thesis, but was cast accidentally.
In your system, that book would be Evil (immoral)–though it never had a choice in the matter (or any at all…it’s a book!). It seems a bit odd to me to have a notion of Good and Evil which doesn’t require freedom of choice.
Additionally, it suggests that a single character being swapped is all that is require for something to become Evil. I rather prefer moral systems that are robust in the face of simple signalling errors.
This is always a fun topic though. :)
-
-
-
-
-
The people who work on evil software like this need to be ostracized by the rest of us.
How, when many of us insist on ignoring politics? Have you noticed how many yes-men were celebrating the “no politics” week on HN?
-
-
-
-
You really need to provide some examples of those comments, because it can be really hard to tell the difference between comment quality actually dropping and people just wanting to say “back in my day this was awesome and now it sucks”.
Especially accusations of trolling need to be substantiated better because the word tends to be grossly overused.
-
I didn’t want to call anyone in particular out. But since you asked, here are some instances:
- https://lobste.rs/s/yktsdu/practical_common_lisp/comments/vgcnoj#c_vgcnoj
- https://lobste.rs/s/g83zqt/amazon_employee_attempts_suicide_after/comments/atbldi#c_atbldi
- https://lobste.rs/s/3cjjml/european_union_wants_regulate/comments/0974hr#c_0974hr
- https://lobste.rs/s/kjfuvp/great_third_pound_burger_ripoff/comments/yiwezt#c_yiwezt
- https://lobste.rs/s/kfikya/julias_drawings/comments/jgaxcm#c_jgaxcm
-
It seems just about every comment you linked was received with polite, but firm criticism/sensible answers, and didn’t end up spoiling the thread or the community’s view. Now while it’d be great to not have these comments at all, I think those examples actually show the bigger picture - the high maturity level of the people in community.
Perhaps the lobsters software should be able to track repeated troll attempts from a single user and raise an alert for moderators to step in. One thing I’ve experienced from moderation of a few communities is that it is generally better to accept more users cheaply, and have stricter rules to kick them out if/when they misbehave.
-
It seems just about every comment you linked was received with polite, but firm criticism/sensible answers, and didn’t end up spoiling the thread or the community’s view. Now while it’d be great to not have these comments at all, I think those examples actually show the bigger picture - the high maturity level of the people in community.
Amen. I agree, and this is why I don’t think censorship is needed.
Hacker News is heavily modded and it’s still a cesspool.
The best way to handle the problem of bad users is not to attract them in the first place. I think that we’re doing a good job of keeping the forum in a state that doesn’t attract the YC type.
-
You are such a prolific commentator here, that one of the explicit benefits of HN (compared to lobsters) is that you aren’t there.
[EDIT] I stand by what I said above, but @angersock is right, I probably could have expressed it better. Some clarification: https://lobste.rs/c/01bj1d
-
This is the sort of feedback that is best left to private messages, or that really requires further elaboration and generalization of principle in order to raise the level of discourse. Please consider either of those options in the future.
-
IMO, michaelochurch’s comments are a non-trivial portion of the low quality comments I’ve seen on lobsters. Virtually every single comment by him either insults entire classes of programmers with absurd generalizations or participates in revisionist history.
This is the sort of feedback that is best left to private messages
I generally agree. I’ve mostly stopped interacting with michaelochurch because all previous interactions have been remarkably negative. But if we’re going to participate in a meta discussion about the comment quality on lobsters, then it seems more than appropriate to air grievances.
-
While I don’t always agree with michaelochurch’s comments, and sometimes they’re only vaguely related to the parent post (which can be disruptive), I think he’s a valuable member of the community. He holds a minority opinion on a number of issues, but argues them in a thought-provoking way. I’d hate to see lobste.rs as a community push people out because of contrarian viewpoints.
-
I’d hate to see lobste.rs as a community push people out because of contrarian viewpoints.
I wonder if you’d actually walk the walk too.
-
-
-
Virtually every single comment by him either insults entire classes of programmers with absurd generalizations or participates in revisionist history.
While we’re on the topic of quality content and all, it would be great if you could back up your claims by quoting something Michael said and telling us why he’s wrong (or why it’s reasonable to get “offended” or upset by it).
-
I provided links and have otherwise said enough. At this point, it’s up to folks to come to their own conclusions.
-
I didn’t see anything wrong with what Michael said in the comments you linked to, so you definitely haven’t said enough.
-
-
-
IMO, michaelochurch’s comments are a non-trivial portion of the low quality comments I’ve seen on lobsters.
Generally, I think this sort of stat-waving is in poor taste, but I have a higher average karma-per-comment than you do.
all previous interactions have been remarkably negative.
You made the first personal attack, not me.
-
-
-
This thread is bringing out some of the worst in our posters I’ve seen in a while–let’s not exacerbate things further.
-
-
-
You are such a prolific commentator here, that one of the explicit benefits of HN (compared to lobsters) is that you aren’t there.
Banning me from HN was part of a larger effort. They forced Quora (which YC bought) to ban me. On Reddit, they used to attack me heavily with sock puppets and brigades. Then I started getting the death threats, including harassment from homeless on the street (presumably paid off by YCs; it is a common tactic) when I was in the Bay Area. On one occasion, those assholes tried to get me fired.
I suppose you’re a fan of all that, too?
If you wonder what I did to piss them off, I wrote a blog post in 2013 where I used the term “chickenhawk” to describe VC’s attraction to inexperience founders. I never mentioned Paul Graham once in that context, and did not have him in mind, but he took the post to be about him, and the rest is history.
I’m sure, though, that you think you dislike me because you think for yourself and not because you’ve been told what to think by Paul Graham and his menagerie of boypets. Carry on, then.
-
If you wonder what I did to piss them off
You’ve conveniently left out some important details that might color one’s perspective. For an example of such a detail, see: https://news.ycombinator.com/item?id=10017538
-
I have no idea what point you’re trying to make.
-
I’d imagine the point was that you were warned by a mod to stop doing something and then banned after you kept doing it.
Either those posts were not in fact written by you (which would be consistent with your accusation that they are trying to get rid of you by any means necessary), or you broke the rules of their private space and got kicked out for it.
I’m not going to tell you they aren’t out to get you - I have every reason to believe PG would act like that - but the HN ban sure looks like more like regular old moderation than some kind of conspiracy.
-
Either those posts were not in fact written by you (which would be consistent with your accusation that they are trying to get rid of you by any means necessary), or you broke the rules of their private space and got kicked out for it.
The rules, to the extent that they can be argued to exist, are inconsistently enforced. People who point out that Silicon Valley has devolved into a pyramid scheme, and that Y Combinator is morally culpable to a large degree, are treated differently from people who aren’t perceived to represent a threat to Paul Graham’s economic or cultural interests.
I’m not going to tell you they aren’t out to get you - I have every reason to believe PG would act like that - but the HN ban sure looks like more like regular old moderation than some kind of conspiracy.
They definitely know who I am. I have a couple sources inside Y Combinator (they’re not all bad people).
[ETA.] Oddly enough, Paul Graham isn’t as bad as he’s made out to be, and he’s been pretty much retired for close to 2 years. I wouldn’t call him a good person, but he’s not Hitler either. PG can be childish and vindictive, but the evil that YC is known for comes mostly from people under him.
-
-
-
-
They forced Quora (which YC bought) to ban me. On Reddit, they used to attack me heavily with sock puppets and brigades. Then I started getting the death threats, including harassment from homeless on the street (presumably paid off by YCs; it is a common tactic) when I was in the Bay Area. On one occasion, those assholes tried to get me fired.
What do you think would cause a diverse group of people across a number of sites to all attack you like that? They can’t handle the truth?
-
It wasn’t a diverse group of people. It was a small number of people (maybe five). Y Combinator owns Quora, which explains the ban.
The death threats could have come from anywhere, and although the Reddit brigade detected last April consisted of 45-70 accounts, it’s overwhelmingly likely in my mind that it was fewer than five people, working together and possibly in the same physical space (YC headquarters).
Of course, I don’t know for sure, but I know how these people fight. It’s more likely that a small number of people are doing bad things than that there is a large conspiracy.
What motivated them? It’s not that they “can’t handle the truth”. They know the truth. What they don’t want getting out there is how much of this current “startup” bubble is outright fraudulent, not only against employees and customers, but also against the institutional investors who provide the capital.
-
-
[Comment removed by author]
-
-
As if a blog post could do something like that.
At my peak, I got about 2,500 uniques per day. I had a low four-digit Alexa rank in the SF Bay Area.
I’ve pulled out of that game. I don’t care about this industry. I enjoy programming, but the tech industry can go to hell (who would know the difference?)
-
[Comment removed by author]
-
I certainly poked the bear, although I didn’t intend to provoke the specific response I got.
In July 2012, I wrote an essay called “Don’t waste your time in crappy startup jobs”. It got about 200,000 hits. That put me on Paul Graham’s radar and soon afterward he put me on “rank ban”, a Hacker News “feature” that would cause my comments to fall to the bottom no matter how many upvotes they got. It wasn’t until 2015 that Gack (the current moderator) admitted to this, but most people in-the-know were aware of it, and I wasn’t the only person affected by it.
It wasn’t a personal grudge, on Paul Graham’s end, until about a year later when I wrote this blog post. He thought “chickenhawk” was intended to refer to him. It wasn’t. I didn’t even have him in mind, to be honest. This is probably an exaggeration, coming from one of my sources inside YC, but I was told that after reading that essay, PG couldn’t even get out of bed for three days. At that point, the grudge was personal. Even though is essentially retired these days, he encouraged his puppies, Gack and Paul Buchheit, to attack me at every opportunity.
I was very active as a technology writer. I’ll admit that it took some effort to get the Paul Grahams of the world as pissed off as I did. It’s not something that you just fall into. What I didn’t expect is that these people would take a difference in economic and cultural interests and then try to spin it into something personal and vicious.
-
-
-
-
You’re omitting a few details. You were banned from Wikipedia for sockpuppeting, you were banned from Hacker News for calling Marissa Mayer the C-word, and you were banned from Quora for repeated sockpuppeting.
-
You’re omitting a few details. [.. snip ..]
Uh.. I totally understand why you posted that, and won’t call it out for being entirely unreasonable given the way this thread (unfortunately) went. So don’t take this personally.
But as a plea for the future, could we all please not dig up dirt on our community members? I really think it is one of the saddest things one can do here. And if we really have to judge somebody, then it should be based on their contribution here on lobsters. Not elsewhere, and definitely not over ten years ago elsewhere.
There are multiple reasons for this. Through such external sources, we catch a glimpse of community drama and claims without context, with no way to verify these claims, with no way to understand the background. No way to know who’s lying and who’s saying the truth. That community might be toxic, and toxicity often breeds toxicity. I admit, I can be quite toxic on the trollfest that slashdot is. And the past is past, people can change. I no longer participate on slashdot.
Along these lines, I can ascertain that when we have a nice friendly community here, then the people here are naturally encouraged to play along and be nice regardless of how they do elsewhere. That is what matters.
But when people come in and bring personal grudges and vendettas and dig up dirt, they bring in the toxin from these other communities. It evokes negative feelings and it hurts, and when it hurts, it is easy to forget what a nice community we have here. And so the poison spreads.
-
But as a plea for the future, could we all please not dig up dirt on our community members?
If you peruse this particular community member’s comments, you will note that he speaks frequently of his past interaction with various folks. It at least seems clear to me from his comments that he’s quite willing to discuss the past and his interaction with communities he’s been banned from. He may very well be telling the truth about many things (as you say, there’s no way to know), but one thing is very very clear: he omits critical details that are terribly inconvenient to his narrative. If he’s willing to talk about it, then adding additional context to what he’s saying seems absolutely fair to me.
-
one thing is very very clear: he omits critical details that are terribly inconvenient to his narrative.
I omit details that are irrelevant, regardless of whether they are favorable or not. It’s not like I post, “I’ve received death threats from YC partners” at every opportunity, because who cares? What would I gain from that? I come here to read and talk about technology, not this sort of shit.
I don’t talk about this stuff except when asked or provoked. The record shows that you, not me, are the one who turned this thread into a personal-attack-driven shitshow. And you owe an apology to the Lobsters community for doing it.
-
And you owe an apology to the Lobsters community for doing it.
As I said, I could have expressed myself better. I never intended for anything I said to be a personal attack, but I can absolutely see how I came across that way. For that, I apologize to you. My intent was to express how unfavorably I view your contributions to this web site. Intent doesn’t count for much, but there it is.
In any case, I’ve learned from my mistake. This will be the last time I respond to you on this web site.
-
-
-
In general I agree with you, but in this case I was responding to a comment in which Church claims he was banned from HN and Quora as part of a larger conspiracy against him (that includes YC paying the homeless to harass him). When someone makes a claim like that, I feel like I need to point out there were several clear reasons for why he was banned.
-
“Point[ing] out” things that aren’t actually true isn’t a public service. It’s annoying and, frankly, you aren’t very convincing or talented at it.
-
-
-
You were banned from Wikipedia for sockpuppeting,
That user’s hate page was debunked a long time ago. Most of those accounts don’t even exist. Granted, I did some stupid shit on Wikipedia back in 2004. Just not that.
you were banned from Hacker News for calling Marissa Mayer the C-word
Not true. I used a different word, “queynte”, specifically because some people consider “cunt” to be a gender slur when applied to a woman. The best translation of “queynte” would be “ornament”, not “crude term for vagina”.
you were banned from Quora for repeated sockpuppeting.
I am aware of that being their stated reason. However, those sock puppet accounts didn’t exist.
Back when I had an active blog, Marc Bodnick posted a comment putting the blame on Paul Buchheit who demanded it. Paul Buchheit denied it. I don’t know who’s responsible for that. What I do know is that Marc Bodnick got fired a few months later, because Adam D'Angelo specifically blamed his moderation for the collapse in user engagement and comment quality.
Please find a way either to become more intelligent, or to become more graceful in apologizing for what you currently are.
-
-
-
-
-
-
I’d agree that the number of bad comments has gone up, but I’m not sure that the S:N has gotten worse.
polite, but firm criticism/sensible answers, and didn’t end up spoiling the thread
We have quite a low quantity of BS, so it’s relatively low-effort to refute (which keeps the place nice). There’s a threshold beyond which people stop being willing to invest time doing that.
accept more users cheaply, and have stricter rules to kick them out if/when they misbehave
My only concern with this approach (which works well in genereal) is that the failure mode is collapse (when e.g. a key moderator is absent for a few months and there isn’t suitable handover).
If that were our approach, I think it would become important to recruit a larger pool of moderators to reduce this risk.
-
Disclaimer: I’m one of the word-criminals listed above.
I pointed out what I consider to be an obvious fact - that Common Lisp itself is not very practical, but didn’t want to go through the effort of trying to convince people of it. For example because if it’s not obvious to someone, he probably wouldn’t be amenable to convincing either.
Someone who’s never considered CL impractical but does have an open mind, might benefit from seeing the idea, in case it led to him investigating and reaching the same conclusion himself.
It seems just about every comment you linked was received with polite, but firm criticism/sensible answers, and didn’t end up spoiling the thread or the community’s view.
Yes, someone asked the reasonable question: “Why?”, and someone else provided a great answer.
All in all, which would you say caused a greater disturbance to Lobste.rs’s peace & harmony: my comment, or this thread? It could be argued that whoever started this thread is sowing discord!
The thing is, we all interpret quality content and whether an article “belongs here” in different ways. Lobste.rs itself can reasonably be found highly lacking in greatness, even if it is better than HN in some ways.
-
-
So, to summarize those examples for people that don’t want to follow links:
- Throwaway comment saying Clojure is more practical than Common Lisp.
- Comment asking why news about a suicide of a non-notable person is being posted to Lobsters.
- Comment expressing skepticism about EU competence on regulating crypto based on linked material.
- Comment (mine) tersely pointing out misuse of
mathtag and panning article source. - Comment wondering why so many Julia Evans drawings (simple diagrams) keep showing up lately.
With the possible exception of the first comment, those all seem like reasonable comments to me and are not particularly trollish (compared with, say, this or some of yui’s stuff.
I think something worth considering is the content of articles all of those comments were in reply to: we need to all remember that a bad submission (like somebody deciding to kill themselves, or spamming pretty drawings, or public policy news) will usually breed bad comments, either asking “why is this here on lobsters?” or failing to have useful content for discussion.
In short, if you submit garbage, don’t be surprised if you attract flies.
-
bad submission (like somebody deciding to kill themselves, or spamming pretty drawings, or public policy news)
I wouldn’t call any of those submissions bad. News about tech industry’s culture affecting people’s mental state, public policy related to tech and other “meta” articles are relevant to lobste.rs, in my opinion. The pretty drawings in question were educational and about tech. Although I didn’t necessarily like some of those submissions, they’re still on-topic.
-
those all seem like reasonable comments to me and are not particularly trollish
IMO, not all low quality comments are trolls. I agree with the OP that comments like the ones linked are nearly content free, and I find it disappointing that they’re appearing on lobsters with increasing frequency. I don’t have any good solutions, unfortunately. Ideally, we as a group would discourage those sorts of comments from existing in the first place. Perhaps @nickpsecurity is right in that the only other choice is heavier moderation, but I don’t really like that choice either. sigh
-
Three of them aren’t content free though–they are meta comments on the submission. There is a place for such comments and unfortunately they are necessary if we want the community to self regulate properly.
Perhaps the increase in bad comments you are seeing is due to an increase in bad submissions?
-
Im pushing two: careful who you invite to point you audit prior comments or behavior (approximates friend-to-friend model); heavier moderation if discouraging specific behaviors that persist. I think the invites arent usually handled like in the first. Many were casting a wide net.
-
-
-
I personally put sub-par comments that spark good discussions into a different category
I think this is important. There are so many ways a sub-par comment that on its own contributes nothing can lead to very fruitful or informative discussion that is worth having, and quite likely would not be had if it were not for that comment. Sometimes, these little comments can even seem a little trollish or otherwise inflammatory. That is one way to spark discussion; perhaps it is not perceived to be a good way, but it can be very effective. Of course there is no way to know in advance what such a thread will turn into.
-
I disagree, given that the brevity of such comments is usually more likely to produce misunderstandings and hostility than creative discussion. Additionally, the brevity of such comments increases the odds that any subsequent discussion is likely to be less topical because of lost context.
Sure, we get occasional gems, but the aggregate effect is always going to be junk commentary and poor decorum.
-
The negative effect of lack of context is important. I overlooked that in my response. It does usually result in people talking past each other until the “real” point comes out. Happened to me here a few times.
Therefore best to write at least enough of a comment that claim and context are clear. This might be worth becoming a guideline at some point.
-
-
This seems to happen most when the comment represents a common misconception that many other readers might have. On HN, I often give a detailed counter with evidence and upvote the comment so corrected information reaches that commenter and others reading along. I also upvote the correct ones past it. Can’t recall how much I do it on Lobsters.
The idea being that just filtering out very different views doesnt make them go away. In absence of correcting feedback, misinformation remains with self-reinforcement and more gets built on it. Im still undecided on best strategy here but think it’s worthwhile keeping and countering low-value comments reflectinh misconceptions if person’s other comments were decent.
-
-
OK, I’m a newcomer here
Are you also this “hga”? https://news.ycombinator.com/user?id=hga who was recently banned?
Jews are to be “excluded if not eliminated from society”, as in all societies that are not Israel. You’ve got your own homeland now, which we of the Alt West fully support, relocate yourself there.
-
is also correct, and prompted a discussion I at least think is worthwhile,
I vehemently disagree with that characterization. The OP posted a useless comment. Of what use to others is to state conclusion without an argument or observation so the we can reach our conclusions? Then ssl appears to have attempted to use the maieutic approach to teach the OP about the importance of backing up your conclusions. At which point you derailed the discussion posting a bunch of incorrect statements that because they take more time and effort to refute normally go unchallenged.
Furthermore I see no good discussion that sparked from it.
-
-
-
I’m surprised the author doesn’t bring up loose index scans, which are potentially more efficient for count distinct queries. The idea is you look up the smallest (or largest value) and use a recursive CTE to lookup the next largest value. This gives a huge speedup when the number of distinct values is small, since you skip over all of duplicate values, and regular
COUNT(DISTINCT ...)reads every single row.-
Is there a reason why this isn’t the preferred strategy that Postgres query planner employs when performing a
COUNT(DISTINCT ...)on a btree-indexed numerical column? Seems to be better in the vast majority of cases (except when the table has high cardinality relative to rows).
-
-
It’s a two-sided lemon market, insofar as most development jobs are also of low quality, and the cause of the lack of prepared, capable programmers isn’t that most programmers are stupid, but that work experience also has a pyramidal shape to it. There aren’t many people who have high quality work experience, because there isn’t a lot of high quality work experience to be gotten.
Most employers reject candidates as soon as there’s a whiff of negative experience: a job that didn’t go well, a company with a negative reputation, outmoded technologies. Then they try to underpay, or they hire people for one job but assign them to something else, and whine about their retention and hiring problems. I’ve met entitled “star” engineers but they’re rare and don’t age well, but entitled employers, who only want 9s and then expect those perceived 9s to wash their dishes, are so common that it’s unremarkable.
The best way to be employable, in this superficial industry run by non-technical and emotionally incontinent monkeys, is not to have any reason, of any kind, for someone to reject you. Even good things can be reasons to reject someone. Ten years of machine learning experience at a prestigious lab? “Too theoretical”. Over 50? “Resistant to change.” Made the mistake of becoming well-known for being too ethical? Well, see my experience. No wonder then that we have such an age discrimination problem; the only way to be compliant is to be a complete blank slate, and fresh college kids are best in that regard (even if most of them don’t know anything).
The truth about this massive lemon party is that no one has any real business need to make it better. Companies get funded and acquired and priced according to headcount, and MBA-toting “star” managers judge opportunities based on the sizes of the teams they’ll get to run (regardless of what those teams do) so there’s no real cost to the business in hiring lemon developers or managers, and there’s even less perceived cost in rejecting people for stupid reasons. It’s technology people who get hurt by it; not only do we end up with lousy co-workers, but when we try to hire good people, we see them getting rejected for stupid reasons.
-
The truth about this massive lemon party is that no one has any real business need to make it better.
That’s not totally true. There’s a niche of companies that do high-integrity or high-security development of software. Some warranty the results. They charge a premium of at least 50% over other companies. All the ones I’ve seen stay growing with referral from clients. There’s companies that internally do something similar with IT either in general (rarest) or for specific teams on critical stuff (uncommon). So, there’s some demand that comes in many forms with serious money to be made. Not entirely lemons but mostly lemons.
Rest of your post is spot on. Curious, what did you mean by “Well, see my experience.”
-
That’s not totally true. There’s a niche of companies that do high-integrity or high-security development of software. Some warranty the results. They charge a premium of at least 50% over other companies.
That’s a fair observation. Unfortunately, those companies seem to be very rare. Is there a list of them published somewhere?
Curious, what did you mean by “Well, see my experience.”
I’m glad that you’re asking, which means that my publicity has faded a bit.
I used to be a bit notorious for some high-profile actions that, while ethical to a fault, were judged negatively by some notable technology companies. There are rumors that I attempted to unionize one company where I worked; this is not true, although I have spoken sympathetically on the concept of software unionization (not that it will ever happen). My name was (erroneously) listed on a “suspected union organizer” blacklist in Silicon Valley for a while. In fact, I don’t know the first thing about organizing a union (and, at this point, I could care less about the Valley).
I’ve survived (and just barely) some attempts to destroy my reputation and career, and my faith in this industry is nonexistent. We are mostly in the business of helping rich guys, who have no concern for ethics or law or social justice, unemploy people. Software could be so much more, but we’ve let it become this disgusting business that I’m embarrassed to have been a part of.
-
Hate to hear it happened to you. However, I totally agree with software developers unionizing. I agree with most in middle class sectors doing that. The reason in IT is that the job is critical, it’s prone to layoffs, there’s lots of discrimination, mismanagement is rampant, performance metrics suck, and mostly importantly the major companies were price-fixing labor. They devolved into a cartel just like I suspected and what I predicted would happen to most oligopolies. Except this was unusual given it crossed some market sectors with fiercely-competitive companies instead of incrementally-competitive ones that are typical (eg AT&T vs Verizon). A situation this bad for labor that only drives money into CEO’s or founders' pockets is exactly the reason unions existed in the first place.
Now, people reading might think of bad unions with unreasonable demands and costs blowing out of proportion. Not necessary. When talking about giving more to workers, I like to use low-margin companies as examples given that higher-margin companies should be able to match at least what low-margin ones do. The best example overlapping low-margin and union is Kroger chain of grocery stores with UFCW union. I’ve actually read that contract and talked to their union representatives about all the stuff they deal with. Here’s what its terms are like:
Workers get paid a percentage better than minimum wage with earnings going up over time as experience increases. They get paid a higher rate for higher positions. These are standardized for common roles so there’s no discrimination on pay. Working in a higher position temporarily to cover a spot (eg someone is sick) earns you the higher position’s pay for however long you worked it. You get overtime if you work overtime regardless of what state says.
Company offers heath insurance, dental coverage, and retirement package. Union manages that side of things so company can’t screw with it.
Workers are guaranteed 10 hours between shifts so they can at least attempt sleep. They can waive it for extra money but can’t be forced to.
There’s standardized breaks and lunches so people can get some rest. The contract also mandates a breakroom so people can’t bump into them asking for them to work on their break. A biometric, time clock records the shifts, breaks, and lunches. That data can be read by both management and the Union.
Workers get a few paid sick days and 1-2 vacation weeks per year depending on position. The vacations are paid. For hourly workers, the system averages the hours they worked then pays the average.
Best for last: due process.
I’m dedicating a paragraph to that as I think it should be a national law. :) Union reps say the crap management pulls on workers is endless. Employees will tell you, too. Management is at Walmart’s level or worse. Everything from racism to forcing young people to skip lunches to intentionally losing paychecks to deploying practices that result in broken backs and shit. Union resolves most of it in heated negotiations without it going further. Sometimes it just takes a call.
Due process is the solution to this. It says Kroger can’t just arbitrarily fire a worker without basically paying them a good deal of money for a period of time. It’s why they didn’t do layoffs during recession. Just hour cuts. They have to come up with performance metrics and policies defining good work. Then, they must write-up workers who violate those with evidence they do. After a certain number, they can fire them. For any termination, a worker can challenge it. If unreasonable, the union will defend the worker first in negotiations and then in arbitration. Union reps say the workers usually get their job back since they were doing good but fired on a technicality for political reasons. Or rep sits in the store for hours seeing tons of violations that get no write-ups but that one employee is getting singled out. Due process with clear standards for performance is union’s first line of defense against bad management.
So, I look at all these companies that treat IT workers like shit. I notice that most of them have higher than 1.6% profit margin Kroger does. Many have tons of revenue. Management make good money with executives making a killing. The things above actually don’t cost much for most mid-sized or large firms. IT will still be plenty productive. They still oppose such things. That’s just because they’re fucking evil. ;) So, unions and then campaign contributions for better labor laws are necessary evil if workers gotta face evil every day.
Note: I did this one last night but the submission disappeared for some reason. Had to redo it… (sighs)
-
-
“If you stand for nothing, Burr, what’ll you fall for?” – Hamilton
I think that we’re often attracted by software because it promises us lucrative jobs and an opportunity to get paid for working in a world of abstraction and syntax (code). And “Big 4” cultures are designed to appeal to people who want to believe in institutional meritocracy, even though the evaluation of “merit” becomes, at an individual level, more political and malignant in the corporate world than it ever was in school.
I’d imagine that the cultures in government and research are very different, but private-sector software is this culture built up by people who (a) are attracted by the money and (b) don’t really stand for anything in an ethical sense. This isn’t a dig, because I was a greedy douchebag when I was younger too– trust me, I’m in no position to cast any stones– but now that I’m older and aware of just how poisonous that moral emptiness can be, what used to seem like an abstract shortcoming (i.e. “we’re not working on things that matter”) is now more existentially pressing.
I also think that we (and the media) tend to trivialize this by looking at, e.g., Snapchat and saying that our generation is being “wasted” on frivolity. That’s true, but the frivolity isn’t the worst part of it, and most of what the VCs are funding isn’t frivolous tech but WMUs– weapons of mass unemployment. It’s more pernicious than just “frivolity”.
To answer the broader question, I think that Corporate America is in (welcome) decline. Kids in college still want to be investment bankers and Big4 programmers, but less so than when I was in school. You’re seeing more interest in public service and research, and less blind herd behavior. As for the self-contradiction of Corporate America, Donald Trump (the id of the corporate class) is showing us that this greed, money-worship, and self-absorbed careerism lead to narcissism, then full-blown egomania, and then frank destructive fury that hurts everyone. Hilary Clinton is a flawed person but she is a public servant and she holds to her values and has a vision for where to take the country. Whether that vision is correct, I’d rather not debate here, but she has one.
The next-quarter mentality isn’t limited to software, and it seems to be destructive everywhere. It’ll take a lot of work to remove it. The pressures involved are too much for most people, right now, to resist.
-
I’m older and aware of just how poisonous that moral emptiness can be, what used to seem like an abstract shortcoming (i.e. “we’re not working on things that matter”) is now more existentially pressing.
I wish there was a club for people like us… You know what is also fun? Balancing your own company on the edge between making money and making a difference.
-
So you think the core problem behind low-quality jobs, funky hiring practices, and frivolous products is that people want money and don’t have ethical values? What is the source of that then? If it’s simply human nature then what is causing the change you describe in young people? If it’s not, then there must be a deeper problem.
I don’t necessarily disagree with you, but I’m not convinced either. I would love for one of these ennui-laden discussions to yield something even remotely actionable, and just saying “it’s greed” seems like a total dead end to me.
-
I’ll obviously let Church speak for himself here, but I my feeling is basically thus (very depressing read ahead):
The zeitgeist of the time is hopelessness, distraction, and greed. We are all, at some level or another, greedily trying to buy distractions from the hopeless nature of things.
At the risk of repeating platitudes, some observations: We use the iPhone, but pay no attention to the e-waste dumps and factory suicides. We inhabit Facebook, but try to ignore the massive surveillance it is predicated on. We praise Uber et al., but tacitly ignore the continued abuse of municipal laws. We espouse diversity, but only when it is applied to viewpoints we like. We enjoy fictional violence, but isolate ourselves ever-further from actual displays of force.
Simply put, we are all drifting further from the reality and impacts of our lifestyle decisions, and at least for people trained in systems thinking (e.g., developers) there is the undeniable slow creeping sensation on the back of the neck that things aren’t quite right, that the sums and figures don’t quite come out correct, and that sometime soon the music is going to stop and we’re all fucked. Those of us not occupied with academic pursuits and the makework of refurbishing the Javascript ecosystem, that is.
Knowing this, and knowing just how ruthless and brutal the system is about optimizing away things (read: people) that are extraneous to a particular economic objective, we make the rational decision and start trying to grab as much money as we can before we can’t anymore.
The obvious posing and fake advertising and frivolity of social media makes it even easier to see our fellow man as marks, makes it easier to justify extracting maximum revenue from them. Whether it’s the dolt retweeting everything Trump says, or the VC who cuts a check on anything which mentions ML or IoT, or just a dumb public official who needs to show their constituents that they are “investing in the future” is of no practical matter: there are just the people who have resources, the people that can become resources, and the people who know how to harvest resources. As developers, we think ourselves in the third category though we’re basically just the second.
I used to think that there was enough room on the bus for everyone, that we could elevate and enlighten and teach and move people (as a whole) forward. I’m increasingly of the opinion that there are simply the people driving the bus, the people under the bus, and extremely limited seating for folks that are neither.
There’s no immediate solution either, right?
Do you believe in your community, in the common good? Both candidates in the current election have basically disowned the other’s would-be voters. The legislature will merrily play chicken with budgets in order to score political points. Even the very notion of belonging to one’s country or civilization is under attack by various philosophies popular in educated circles!
How, how are we to believe in the greatness of the people and the goodness of man when we are constantly reminded of this? How are we to devote our efforts to holding up a tottering dam holding back chaos when that act is criticized by some, used as a cheap revenue source by others, and actively hindered by the rest of folks too stupid not to play with matches next to the piers?
We have an additional mokita: more than ever we individually are both more aware of our mortality and limitations and at the same time further isolated from them. War and famine are things that we see on Twitter and the news but which never really effect us. We can learn more than ever before about any given subject, and yet we are continually overshadowed by stories and articles about people who are either best in their field or just exceptionally good at advertising.
In such a situation, what is the value of your life? What is its purpose? What makes it special or desirable? Why bother? At a large enough scale and with good enough coverage, we’re all just Brownian noise in the lifestream–and that’s where we are today. There’s no point in being the best person in town at doing foo, because we all read about foo on the ‘net and know how far we’d have to go.
So, instead, maybe we can get enough money to paper over that existential void. Maybe we can buy enough things or influence to secure a place in history. Maybe this time we’ll pull it off…maybe.
-
Knowing this, and knowing just how ruthless and brutal the system is about optimizing away things (read: people) that are extraneous to a particular economic objective, we make the rational decision and start trying to grab as much money as we can before we can’t anymore.
This is right on the nose.
Before 1980, when many of our parents were growing up, it was socially unacceptable to say that you just wanted to make a lot of money or make connections so you can get a job where you don’t really have to work. Donald Trump became the zeitgeist of a materialistic, crass era (the 1980s) for a reason: it just wasn’t acceptable to be like him. Avarice and egotism still existed (see Mad Men) but were considered crass and pathological. And consider that the office politics of Mad Men, although nasty in their time because advertising had a similar flavor to entry-level investment banking today, aren’t all that bad by the modern standard. In the ‘70s, staying till 6:30 meant you were a hustler. Work was a more civilized game, and people played for the long term. Greed and ego have always been factors, but people were more intelligent in going about their objectives and there was more of a long-term mentality which precluded a lot of the worst plays.
The era of the lifelong technologist seems to be drawing to a close, except in academia and in some government agencies. In the private sector, this is definitely a game where if you’re not a founder or an “angel investor” (read: rich) by age 40, people will ask why. These days, being a software engineer means contending with micromanagement (Ministry of Agile) that is designed for children, that “we” have had to accept because the conscientious objectors have all been fired and replaced with compliant, often belligerently incompetent, unprepared neophytes out of college or “boot camps” (which are, mostly, fly-by-night trade schools with no quality control). The pay is decent. Not amazing, but decent. It’s one of the few genuinely middle class jobs left. Still, the low status of the job means that as soon as the market softens, programmers are going to take a hard fall while the management will be just fine: VC associates who don’t make partner will circulate elsewhere in private equity, founders will end up in $250k jobs at hedge funds, and startup executives will ride their coattails. Meanwhile, a generation of programmers will be left in the cold with nothing to say for itself.
-
-
“If it’s simply human nature then what is causing the change you describe in young people? If it’s not, then there must be a deeper problem.”
I threw a savant brain at the angle for years trying to figure it out. I eventually did come up with a model that explains it. Tried to find every work-around I could to change it. Ended up really depressed when I couldn’t find one permutation likely to work without straight-up revolution. It got to the point where I could predict what would happen with national events or elections at an abstract level. I originally thought it was emergent behavior but it’s increasingly clear the worst parts of what are happening are by design and emergent behavior within that design. Occasional outliers but the system the elites put in place is pretty air-tight.
It’s too complex to explain in one comment but I’ll give you a few key points.
It starts with capitalism at banking and industry levels. They realized certain practices would get them rich. One of those was screwing workers. The vast majority of people starting businesses are selfish enough to want to be very rich. They managed to use money to Congress and media presentation to voters to make sure vast majority of wealth produced by majority of workers went to tiny few with similar interests. They started forming monopolies using their vast capital. When that was busted up, they started forming oligopolies w/ cartel agreements to prevent competition. They get outrageous CEO compensation since the boards that keep that in check often have other CEO’s, founders, etc They’re all on each others' boards (“interlocking boards”) with emergent behavior of “you get me rich, I’ll get you rich.” They also pushed for patent and copyright to be strong to allow both selective monopolies plus prevent or financially drain competition. We’re already a plutocracy at this point where one cartel controls whole financial system plus others control most markets key for survival. Worse, the incentives say reduce cost/quality/safety while charging more.
Congress will save the day with laws “for the People,” right? Congress and Executive branch are corrupt so no. First, they need tons of money to get elected. A Presidency costs around $200 million these days. Congressional seat at least millions probably tens. That means only people that can run are rich people or those backed by them. Also, existing system lets current legislators attempt to filter out anyone less corrupt that bypasses that problem. The voters themselves are extremely superficial where character and voting/business history are mostly ignored in favor of whatever a candidate looks like, does in personal time, says during campaign, etc. What professional liars say >= what they do or did. (???) Once in Congress, they spend most of their time preparing for next election. They pay back contributors, mostly elites, with laws that benefit them at people’s expense. Most Congress has portfolios of stock in dirtiest companies. They also ensure votes by sending massive amounts of pork to their districts which is why they waste so much on “Defense” spending building shit we don’t need & broken welfare systems. Those are tied to millions of votes directly impacted by changes. All adds up to preserve the status quo.
Media will inform us so we change votes and overthrow the system, right? Media is a bunch of for-profit corporations whose business model is making money off adds by getting people to look at the screen as long as possible. They are not there to inform! It’s a business! Getting people’s attention meant they covered key stories, had important people on the air, etc. The also are run by elites who like the system as it is since they’re rich and powerful. Changes would impact them. So, they always practiced self-censorship where they collectively avoid topics or avenues of investigation that would lead to radical change while focusing on issues that appeal to each’s demographic with a mix of emotional responses. They’ll break veil of censorship if someone hits critical mass where they can’t be caught ignoring it. At that point, they either present it in a non-actionable way focusing on blame instead of solid response or start covering shock stories that distract people. This already worked way too well to point the corporate media is single greatest threat to American democracy in existence. Fox improved the model by basically turning up the bullshit to extreme levels with a format that focused on people fighting with each other, use of fake experts, tying message more to viewer, and getting more viewer involvement that doesn’t really do anything but feels like it does. Record-breaking profits and dominance on right-leaning side followed. Others copy their techniques now. You often can’t talk about a key issue for over a minute before host interrupts you or Americans tune out. So how can you change anything again? Go to another of those tens of thousands of stations that are all owned by the same 20+ for-profit corporations with interlocking boards? Good luck.
Our eduction system will help people figure it all out, right? I still haven’t read Gatto’s Underground History of Education to see if it’s legit or bullshit but the abstract I saw a while back seems true. It was elites like Rockafeller that started the system as industrialization kicked in where they needed tons of workers smart enough to do the shit jobs they were creating that made elites rich. As Carlin said, “smart enough to operate the machines but not smart enough to” know how much they were being screwed. The education system dictated what people would learn at what pace with promises of them making millions over time if they followed it plus severe punishments for those that didn’t. The process itself combined rote memorization of material from authority figures, rigid routines, punishment of dissent, and simple metrics to assess skill. Smart people had to teach themselves shit constantly outside of school plus fight with educators over being taught ineffective methods “because it’s required by the bosses.” Basically like working in a factory or big corporation. It’s not education people: it’s conditioning humans like dogs with bare minimum in education. No wonder elites sent their own children to expensive private schools, got them tutors, and brought them along to see how they did business at executive level. That class gets educated where I’m still fighting to learn some aspects of what the C-level people do.
There’s also surveillance/police state, what the U.S. military actually does, systematic suppression of dissent from voting rights to business, and so on. However, the above combined with human nature are all that’s necessary for a successful plutocracy. Human nature is herd-minded, terrible at long-term risks, focuses on here/now presented to our faces, prefers easy battles to hard ones, wants to maximize individual gain in local context, and has trouble being vigilant. The education and media combine to create a mental maze for average person where they go in the directions that are safe for the system. Some will fight it in ineffective ways while others will defend it thinking it benefits them. Some will make themselves and elite investors rich improving something, most will expand on the profits of incumbent elites, some break away from the system without achieving critical mass to affect it, and rest fall through the cracks. Capital almost entirely in rich’s hands combined with corrupt, capitalist Congress reinforcing system in law means the battle will always be uphill. Media continues to suppress key issues, like how most problems are due to Congressional bribes, but will endlessly repeat or generate frivilous stories that maximize their revenue. Americans stay fighting with each other in the maze instead of the elites that built it. Most successful, reality-distortion field I’ve ever seen.
So, there it is in a nutshell. I have no hope of fixing it. The system is too robust after the decades they spend working on it. There’s a small chance that a solution can happen involving Internet media if the both the bait messages and the presented solutions are ultra-simple. The problem and solution have to be simple with candidates and legislation ready to go. They have to be willing to vote people out of office. The source(s) can’t make one mistake in accuracy plus need several tuned to different demographics all pushing same thing from different perspectives. There’s a chance that several classes of problems could be knocked out that way. Most aren’t using this strategy, though. The few that are use it on the wrong messages that just add to lower classes fighting each other.
-
-
-
-
-
-
Your comment is just a bunch of rhetoric without much content.
You’ll have a heard time convincing anyone that companies don’t care about the productivity of their engineers. Without productive engineers, they aren’t making money. Why would they go against their own self interests?
And of course there’s a cost to the company if they hire a lemon. Have you ever worked on a team with a bad developer? A single bad developer ruins the productivity of the entire team they are on. That’s why companies are so careful about avoiding bad hires. A single bad hire is equivalent to missing out on multiple good developers.
-
A single bad developer ruins the productivity of the entire team they are on.
What would you call the manager that allows this to happen? Inexperienced? Ludicrously incompetent? Is this just the assumption, that no manager ever was able to stop someone from building a project’s foundations out of balsa wood or nitroglycerine?
It takes more than a single bad individual contributor to ruin a whole team. Management is supposed to be there to evaluate progress, identify, and address problems. Yes, management does not always succeed.
-
Management are usually the cause of the problem - they are there to manage and lead - many are not competent at either task - hence the Peter Principle
-
I had this recently happen to me. We had an individual who would continuously argue with others. Management repeatedly tried coaching him, hoping he would improve. It wound up taking two months for management to decide it would be better to let the individual go. After he was gone, everyone immediately felt much more productive.
-
-
There’s a steady stream of articles and comments about how broken hiring is, how poorly good developers are treated, discrimination against older ones to get less-skilled people at lower cost, how most projects fail due to poor management, how execs look at IT as a cost center knstead of strategic enabler… all this stuff indicates they dont give a shit in practice. Most also will punish attempts at reform.
So, his comment is pretty consistent with what I read from insiders instead of rhetorical nonsense.
-
-
Many specific problems with enough detail for HR or senior executives to take action on. His comment would be useless if he just said the two things you just said. Strawmen are easier to knock down, though.
-
-
-
-
-
I’m sure you all know this already, but: partitioning relational database tables is a double-edged sword. You almost always never need to do so.
The reasoning being that the instant you have a query that doesn’t have a constraint that matches up with the partitioning constraint, you know have to check n different “tables” and n different indexes, with the potential concomitant I/O overhead.
The only times it really makes sense is when queries are easily constrained by whatever partitioning predicate you chose; when the vast majority of data will be at rest and accessed only occasionally; when you want to spread the same table across multiple disks for performance or space reasons; or when you have a dataset where you often want to delete huge chunks and those chunks naturally fall on partition boundaries.
IMHO, of course.
-
I believe there are a few other cases where table partitioning is useful. Since sequential scans are much faster than index scans, you get a large speedup if you can partition your data so you get sequential scans over some of the partitions. As an example, if you are running a query that touches all data over several days, partitioning by each day can help since you will get sequential scans over all of the tables the query touches except the boundary ones (alternatively, if you can keep your table clustered, you can use a BRIN index to accomplish the same thing). The only other case I can think of is for inheritance (as mentioned in the article).
-
While I enjoyed the post, the comparison at the end is unfair. The author compares ZFS with a 475GB NVMe drive as a cache to XFS without an equivalent cache.
The initial comparison with XFS is somewhat unfair as well, though: does XFS provide the same data integrity features that ZFS does? It’s hard, really, to compare file systems with vastly different design centres and feature sets – which feels like the point they’re trying to make, really.
Was the comparison looking at data integrity though? I didn’t see any mention of that anywhere – everything I saw was entirely about performance. If you’re doing a performance comparison of two filesystems, comparing them on (very) different hardware doesn’t seem real meaningful.
The author mentions the possiblity of comparing against something like bcache (which would then be a zfs vs. xfs+bcache comparison rather than strictly a filesystem comparison), but then handwaves it away as “exotic” and concludes, essentially, that “zfs plus additional fancy hardware and a bunch of manual tuning outperforms xfs”. Well…big deal.
At what point do you need to assume integrity as a baseline though? This is a database blog we’re talking here.
Unrelated observation: it’s tragic that most production databases out there aren’t running on ZFS, and says a lot about the priorities (and less charitably the general ability) of our industry.