1. 12

Hi Fellow Lobstres,

I recently returned from vacation. While away, I had an incredibly interesting conversation with a family friend. I know that this person is highly skilled in the production and repairing of custom circuit boards. Spending long hours in front of a microscope near her assembly line with her peers.

On their latest project, she told me that they had been doing work for Tesla Motors. Several days ago she said that the plant manager had called a plant wide meeting. In it, the manager revealed that there was a mis-build on the Tesla boards. Someone did not read the engineering document correctly and the wrong flux was used for over 9000 boards. This means that 9000 specialized boards will have to be RMA’d or replaced. The speculation is that the loss could be as high as 10 million dollars.

When I asked what she felt the problem was, she said that the plant had almost doubled its staffing in the last two years. going from 600 to 1000 people. She has seen this hiring behavior before and it has always led to mis-builds. She said that she doesn’t know what the fallout would be but that she was worried.

I was kind of pissed on her behalf because as I see it, a bean counter did some shitty math and figured that if they just double their staffing, they will double their output. Instead, they didn’t take into account that a highly specialized skillset and mindset is required to produce this equipment. So now, it makes my friend and her coworkers look bad when in actuality, it was a poor managing decision.

How have you handled this type of challenge in your career or life? What was the fallout of the conversation and the decision?


  2. 8

    I have no advice to give, but are you sure you should be mentioning the brand of car here? I would imagine that kind of information could be damaging to the company, especially a publicly-traded one.

    1. 6

      This is not a loss for Tesla. Their shareholders should not be concerned. The loss falls onto the manufacturing company building this part. This company was not named.

      Thanks for your concern.

    2. 6

      Brook’s “The Mythical Man-Month” covers this well, and is still largely relevant even though it was published in 1975.

      Anecdotally, I have been involved in late software projects, and as expected there were a few proposals such as “can’t we just add some contractors and speed things up”? Thankfully those managing the projects, in general, carefully explained the lead time for training on the codebase, how this would take time away from those currently working on the code, impact due to disruption, etc. Instead we would usually end up trimming a couple features for the first version, and to lower delivery time enough to be palatable.

      1. 3

        I strongly recommend “The Mythical Man-Month” as well. After reading that book, I have found it much easier to see tell-tale signs of a software project going awry. That said, I am surprised the manufacturing company did not push out a small batch of boards (say 100), and send back to the company for testing. This adds lead time, but often avoids major financial catastrophe (such as this case). It can also save your customer money, if the drawing wasn’t quite as thorough as they thought!

        Hardware prototypes can be absolutely crucial to determining if a) the drawing is correct b) it has been interpreted correctly by both parties and c) gives the manufacturer a chance to become comfortable with the build process and confirm the tiny details which are sometimes missing, even in the best drawings.

        If this was a misbuild AFTER prototyping, the question turns to how - how did company process controls/QA allow a change to a previously approved build? How can the company prevent something like this again in the future?

        A lot of food for thought here. Thanks!

        1. 2

          She said there is a thorough control process for getting parts for your builds. If one was lazy, you (the tech) could save time by assuming you know what is best and take parts from other assembly lines near by. Potentially thinking “Flux is flux! It is all the same!”

          She has been in the industry a surprisingly long time so she has the knowledge to observe the whole rapid growth and hiring phase and correlate that with poor quality. This clicked with me as I recalled projects that were mismanaged and how I was unable to steer the conversation effectively.

        2. 2

          I find that the majority of CS majors have had to read it. The business manager’s I’ve worked with however tend to have not read the book. And when I have lent my copy out, the consumeristic nature of people (at least in the States) has led some to as me why I loaned them such an old technology book. Appearing to think that newer is better? :(

          I really appreciate you sharing. Ideally, I am seeking more current and personal experiences to this ongoing issue. If we could quantify different managerial or personality types maybe there is a common dialect or way in which to communicate this concern and actually be heard. Or maybe you attack it on the front end with a basic risk assessment strategy for producing technology products (software, pc boards, etc).

          I realize I’m probably dreaming… :-)

        3. 6

          I don’t think in this particular case the hiring decision was inappropriate. If you are a manufacturing company and you are getting a higher order volume it absolutely makes sense to take on more staff (or find other ways to increase your production rate). The mistake made here was not training the workers adequately and not having good quality controls. If the boards were being tested as they were going out of the plant, the issue might have been caught before 9000 bad units were sent to the customer.

          1. 1

            I agree, the company’s decision to increase its labor force is totally normal. How quickly they do it and how they train their staff during the on-boarding period appears questionable.

            1. 1

              Was thinking the same. Mythical Man Month is about software development, not assembly line production.

            2. 3

              to answer your question directly: the famous quip (from, i think, “the mythical man month” by brooks) is that 9 women can’t make a baby in a month. and the more politic way of explaining that is to buy someone the book.

              BUT… only yesterday i was getting stressed because someone above me decided to shift emphasis in our current work, so there was less technical responsibility for me, with some work delegated to someone else.

              i had to sit down and think.

              i realised that my response is always to look for a technical fix. because i am a smart guy who can solve technical problems (and i don’t have to rely on someone else who might be flaky). so my natural reflex, when challenged, is to look for a technical solution that i can manage.

              and i realised this had caused problems in the past. on a previous project i worked round various issues with technical fixes that ended up causing further issues. not because they were wrong, but because the client didn’t understand them. and when they did, they suggested simpler compromises that we had to implement instead.

              so i realised that sharing things with more people sometimes makes sense. and that i am biased, favouring keeping things under my control. in short, i am optimising to lower short term risk of failing, at the expense of the long-term success of a (more client-friendly) product.

              now, in my enlightened state ;o) when i read your question, i wonder whether you have a similar reflex. it is safer to keep things local. but it’s not always the best idea. maybe the company needs to grow anyway? maybe this is the cost of bringing in new people, but the long-term gain is worth it?

              1. 3

                Yeah, growth is going to happen. I’m not arguing against it. More like I’m trying to tie long term growth, risk, and incentives together to have people make sustainable decisions . A poor example would be money markets where the incentives and long term risk rarely intersect. Anyone have a good example?

              2. 2

                I’ve found that if a higher-up makes a decision that is not based on data or sound reasoning then it is unlikely that data and reasoning will convince her to change her mind. Sadly, sound bites and populist rhetoric (that is, politics and propaganda) can be more effective in the right circumstance.

                See also I need this baby in a month - send me nine women!

                1. 2

                  You can’t make 9 women produce a baby in 1 month.