Tesla is a weird company. They are very bad at what their nominal business is (making cars), and they spend a lot of time, even a lot of “building car” time, on stuff that seems pretty immaterial. I wonder what the future of the company is – probably an battery IP holding company, honestly. Too, this read like an ad for Tesla, which is all fine and good, but given their hardware track record, I would be hesitant to go trumpeting their custom SOC as somehow a differentiator.
They have a significantly less complex problem than any IC car manufacturer (to say nothing of hybrid powertrains), and yet they can’t build cars at anything like rate that that plant can support. For whatever reason (probably related to Elon Musk, but), they seem to lack any understanding of industrial operations.
I mean, I like their cars fine, and they are on the right side of history, but man alive.
You are assuming they have no bottlenecks in any of their supply chain. Battery production has held them up over the last year (this has been publicly acknowledged) and the next issues are neodymium[1], gallium[2], bauxite[3], and copper[4]. The United States is currently 100% reliant on foreign sources for gallium.
The mining and production of these need to catch up.
Do you know what those tents are constructed out of? They’re not something Elon picked up at REI. Please don’t buy into this short trope.
The Tesla Fremont factory was operating three assembly lines (four with tent?). That factory was originally producing Chevrolet Prizm, Toyota Corolla, and Toyata Tacoma. The Prizm and Corolla are almost identical, so they can probably share a line.
Do you think Tesla can share production lines between the Model 3, Model S, and Model X? I don’t believe so. Not easily as they don’t share parts. At peak production in 18Q3 which included the tents it is my understanding that they built a new line in the tents while trying to rework the Model S and X lines so they could build Model 3s on them as well. The details here are fuzzy.
The point is, Tesla ran out of room. Demand was too high and their lines weren’t ready for it. Luckily this won’t be a problem for long even with the Model Y because they’ve got permits from Fremont to double the size of the factory.
Yes, Elon made a huge mistake thinking he could automate all the humans away. That’s not possible until he simplifies many of the parts and wiring harnesses, etc. He was far too eager here.
When Tesla’s computer vision is better trained I think we will see him attempt this again. The robots need to know if a part has rotated and needs to be picked up or grabbed differently. This was one of his problems but I think it’s largely solvable.
Industry experts that have toured the assembly lines have marveled at how much simpler everything is compared to ICE vehicles. Once Elon is done learning the mistakes Toyota and others figured out decades ago I think production will pick up massively. Likely this will come from the Chinese Gigafactory which will have better designed lines and input from a culture that knows this problem space better. How long those learned lessons may take to trickle back to Fremont is unknown, though.
Elon should be doing better, I agree. Tesla should be cranking out more cars. But that doesn’t mean the cars are bad. They’re great cars. Electrics will be the future, and the big ICE manufacturers are not equipped for the transition to this different tech. Sourcing batteries, building equally good motors, the software required, redesigning the chassis around the new battery and drivetrain… it will take years.
My friend has an original Model S P85 (VIN ~6000) and it’s the best car he’s ever driven – over 150,000 miles, “War Machine” is still going strong. I took a ride in it when I visited him in Hawaii March 2013, a few days after delivery. I knew then it was the future and I would own one someday.
But he hates the bad decisions that have been made. They did stupid shit like the wiring harnesses were using solid core wire. Guess how many times you can open and close your door before wires break? Original doorhandles that glide out when you walk up – beautiful! Very cool! But they used potted metal (remember Audi doorhandles breaking off if you pulled too hard?) and there’s a plastic gear inside that snaps and breaks the doorhandle so you can’t open your door ($1000 fix each; aftermarket stainless gears now available for ~$100 if you want to DIY). The window regulators are garbage because they use too much plastic and they fail, so your windows fall down. But recent Audis, Porsche, etc all use a similar design, so this isn’t strictly a Tesla problem.
He hates his P85. He gets furious when these things break. He paid good money for it. He was promised it would be upgradable to Autopilot, etc. Elon went back on his word because it was too difficult to retrofit.
He hates his Tesla so much he bought another Tesla – a Model X.
Why?
Because the equivalent Audi is worse and much harder to work on.
Which model Tesla are you driving? The internet is rife with bad QC stories on the Model 3 (paint issues is a big thing). Come to think of it, there’s the “whompy wheel” issue of the Model S.
I lease an Audi and it’s been in and out of a body shop in around a week. I’ve seen accounts of Teslas waiting months for repair parts.
Tesla owners are being hit by large insurance premium increases, presumably because even fender-benders incur large costs in repairs and substitute vehicles.
I own a Model 3, VIN 65xxx. The bad QC stories are a lie. Even Bob Lutz has admitted as much in a recent article [1].
The “whompy wheel” issue is a smear campaign. I’m not aware of any incident that didn’t include a massive amount of force being applied to that wheel. It’s possible they exist, but I haven’t seen one. It is unlikely the suspension failed like that while driving normally. In almost every wrecked Tesla I’ve seen the wheels come off, which I think is by design to absorb some of that energy in the wreck.
As for insurance – Progressive is only costing me $260/6 months for a policy that includes 250k/500k, comprehensive, collision, etc. No shortcuts taken on the insurance policy.
Odometer is at 19,999 miles right now, took delivery around Oct 1st.
Multithreading usually requires a bit more programming work to distribute tasks properly, but hey, this is Tesla we’re talking about — it’s probably a piece of cake for the company.
I don’t think that thread says anything about the expertise of the team that would have to implement multithreaded code, or anything about the overall level of development expertise at Tesla, really. If you’ve worked in software for a while, you should have plenty of stories like that yourself. (If you don’t, I contend you’ve been unusually lucky with your choice of employers.)
There is a somewhat qualitative difference between a phone switch crashing and a car suddenly unable to steer or brake going through a schoolzone and crashing.
I don’t mind reaping child processes in my programs, but I’d prefer my sedan not duplicate my behavior.
If you mean the part where the computer runs a bunch of nasty heuristics to convert camera pictures and radar scans into second-by-second actions, don’t systems like TensorFlow normally use SIMD or the GPU for parallelism rather than threads, to avoid the overhead of cache coherency and context switching? When your tolerance for latency is that low, you do not use Erlang.
If you mean the part where you use map and traffic data to do your overall route, I don’t think you need to be that fast. You’re spending most of your time waiting on the database and network, and could probably use Erlang just fine. The important part is the fast self-driving heuristics system cannot block on the slower mapping system. The driving system needs to send out a request for data, and keep driving even if the mapping system doesn’t respond right away.
I was being facetious, really. You wouldn’t run BEAM on the AP computer; it’s not meant for that kind of data crunching.
It is my understanding that the MCU is the middleman between the AP computer and the car’s hardware – this is how it also applies firmware updates to various parts of the car.
So I would write the AP control plane in Erlang/Elixir for extremely reliable low latency message passing. I expect the MCU is receiving values from the AP hardware which it acts upon by sending the signals to the car’s hardware. This also means it’s extremely unlikely to crash the process from bad data coming from the AP computer.
This is a guess based on what I’ve seen inside the MCU, but haven’t bothered digging too deep.
I’m also confused about why you think Erlang is not low latency?
The language that’s designed for safe multithreading and high performance is Rust. BEAM languages wouldn’t provide acceptable performance for this use-case.
The languages used in successful projects in safety-critical field had no formal spec. That’s mostly C and assembly with some Ada, C++, and Java. So, Rust would probably be an improvement unless it was a group throwing every verification tool they can at their C code. It has the most such tools.
To be fair Ada has a pretty decent specification and SPARK/Ada probably has the most usable verification tools for industrial usage today, as long as you want specifications that are more expressive than what your type-system can capture. The Rust system may be very good at catching ownership-related mistakes, but there still currently exists no automated tools to verify that, say, a function that claims to be sorting data actually returns a sorted result.
You’re right in that Ada/SPARK can get further in correctness. Most in safety-critical systems use C subsets with no formal methods, though. There’s lots of review, lots of testing, and recently more use of automated analyzers.
Even so, Ada still has Rust beat on that given there’s more tooling for analyzing and testing it. C has even more than Ada.
No, but I wonder how this works in relation to JVM / BEAM. Is the formal spec really about the specific language or is the behavior of the VM sufficient? I’m not aware of different JVM or BEAM languages being able to do things that are impossible in Java/Erlang.
Need more info, but it’s interesting to think about.
I watched the latter part where they describe the neural network accelerator, and well, it seems to do matrix products pretty fast! This is the kind of stuff that keeps NVIDIA awake at night :)
Would be interesting to see how the low-level programs for this chip look like. Do you get just a million lanes wide multiply-add and a couple of instructions to calculate the activation functions?
Anyway, Tesla is betting heavily on deep learning here, which shouldn’t surprise anyone since Andrej Karpathy is their AI lead. I hope the new system will be much better than the current Autopilot that has already caused deaths of four people.
cleantechnica has a bit of a pro-tesla bias (similar to electrek.co). I’m a bigger fan of following journalists behind the podcast autonocast.com.
To elaborate further on the power tradeoffs from the autonomy day presentation - notably the chip design was for keeping things within ~100W usage compared to some other solutions which may use more power (e.g. nvidia drive px pegasus has a 500W tdp), as more power at these levels starts to cut into driving range. Part of this was also to allow for retrofitting earlier model 3’s with the newer drive computer.
I definitely got a pro-Tesla bias too, but at least they admit it up front. From the author’s bio:
That is why he has a tremendous admiration for Elon Musk and his companies, foremost because of their missions, philosophy, and intent to help humanity and its future. He sees Tesla as one of the few companies that can help us save ourselves from climate change.
Tesla is a weird company. They are very bad at what their nominal business is (making cars), and they spend a lot of time, even a lot of “building car” time, on stuff that seems pretty immaterial. I wonder what the future of the company is – probably an battery IP holding company, honestly. Too, this read like an ad for Tesla, which is all fine and good, but given their hardware track record, I would be hesitant to go trumpeting their custom SOC as somehow a differentiator.
Why do you believe they are bad at building cars? In my experience they produce a better car than anything I’ve driven from Audi, Mercedes, or BMW
They have a significantly less complex problem than any IC car manufacturer (to say nothing of hybrid powertrains), and yet they can’t build cars at anything like rate that that plant can support. For whatever reason (probably related to Elon Musk, but), they seem to lack any understanding of industrial operations.
I mean, I like their cars fine, and they are on the right side of history, but man alive.
You are assuming they have no bottlenecks in any of their supply chain. Battery production has held them up over the last year (this has been publicly acknowledged) and the next issues are neodymium[1], gallium[2], bauxite[3], and copper[4]. The United States is currently 100% reliant on foreign sources for gallium.
The mining and production of these need to catch up.
[1] https://www.reuters.com/article/us-metals-autos-neodymium-analysis/teslas-electric-motor-shift-to-spur-demand-for-rare-earth-neodymium-idUSKCN1GO28I
[2] https://pubs.usgs.gov/fs/2013/3006/
[3] https://www.prnewswire.com/news-releases/global-market-study-on-bauxite-mining-surging-demand-for-aluminum-products-to-reinforce-demand-for-bauxite-and-alumina-300811691.html
[4] https://banyanhill.com/tesla-needs-copper/
Is it batteries that’s making them assemble cars in tents in the parking lots? I am not being smart – I simply don’t know.
Do you know what those tents are constructed out of? They’re not something Elon picked up at REI. Please don’t buy into this short trope.
The Tesla Fremont factory was operating three assembly lines (four with tent?). That factory was originally producing Chevrolet Prizm, Toyota Corolla, and Toyata Tacoma. The Prizm and Corolla are almost identical, so they can probably share a line.
Do you think Tesla can share production lines between the Model 3, Model S, and Model X? I don’t believe so. Not easily as they don’t share parts. At peak production in 18Q3 which included the tents it is my understanding that they built a new line in the tents while trying to rework the Model S and X lines so they could build Model 3s on them as well. The details here are fuzzy.
The point is, Tesla ran out of room. Demand was too high and their lines weren’t ready for it. Luckily this won’t be a problem for long even with the Model Y because they’ve got permits from Fremont to double the size of the factory.
Thanks, this is useful. However:
That’s uh, entirely the point of industrial operations?
Yes, Elon made a huge mistake thinking he could automate all the humans away. That’s not possible until he simplifies many of the parts and wiring harnesses, etc. He was far too eager here.
When Tesla’s computer vision is better trained I think we will see him attempt this again. The robots need to know if a part has rotated and needs to be picked up or grabbed differently. This was one of his problems but I think it’s largely solvable.
Industry experts that have toured the assembly lines have marveled at how much simpler everything is compared to ICE vehicles. Once Elon is done learning the mistakes Toyota and others figured out decades ago I think production will pick up massively. Likely this will come from the Chinese Gigafactory which will have better designed lines and input from a culture that knows this problem space better. How long those learned lessons may take to trickle back to Fremont is unknown, though.
Elon should be doing better, I agree. Tesla should be cranking out more cars. But that doesn’t mean the cars are bad. They’re great cars. Electrics will be the future, and the big ICE manufacturers are not equipped for the transition to this different tech. Sourcing batteries, building equally good motors, the software required, redesigning the chassis around the new battery and drivetrain… it will take years.
My friend has an original Model S P85 (VIN ~6000) and it’s the best car he’s ever driven – over 150,000 miles, “War Machine” is still going strong. I took a ride in it when I visited him in Hawaii March 2013, a few days after delivery. I knew then it was the future and I would own one someday.
But he hates the bad decisions that have been made. They did stupid shit like the wiring harnesses were using solid core wire. Guess how many times you can open and close your door before wires break? Original doorhandles that glide out when you walk up – beautiful! Very cool! But they used potted metal (remember Audi doorhandles breaking off if you pulled too hard?) and there’s a plastic gear inside that snaps and breaks the doorhandle so you can’t open your door ($1000 fix each; aftermarket stainless gears now available for ~$100 if you want to DIY). The window regulators are garbage because they use too much plastic and they fail, so your windows fall down. But recent Audis, Porsche, etc all use a similar design, so this isn’t strictly a Tesla problem.
He hates his P85. He gets furious when these things break. He paid good money for it. He was promised it would be upgradable to Autopilot, etc. Elon went back on his word because it was too difficult to retrofit.
He hates his Tesla so much he bought another Tesla – a Model X.
Why?
Because the equivalent Audi is worse and much harder to work on.
To him, Tesla is a Lego car.
https://www.cnbc.com/video/2018/08/28/tesla-owner-frustrated-so-fixes-his-own-model-s-easy-as-legos.html
Wasn’t the latest news that the plant they got from toyota is now producing more cars/day than toyota was?
That might have been an issue a year ago or even a few months ago, but they seem to have figured it out now.
Edit: this of course, only applies to cars and provides a pretty biased view of their manufacturing operations
Which model Tesla are you driving? The internet is rife with bad QC stories on the Model 3 (paint issues is a big thing). Come to think of it, there’s the “whompy wheel” issue of the Model S.
I lease an Audi and it’s been in and out of a body shop in around a week. I’ve seen accounts of Teslas waiting months for repair parts.
Tesla owners are being hit by large insurance premium increases, presumably because even fender-benders incur large costs in repairs and substitute vehicles.
I own a Model 3, VIN 65xxx. The bad QC stories are a lie. Even Bob Lutz has admitted as much in a recent article [1].
The “whompy wheel” issue is a smear campaign. I’m not aware of any incident that didn’t include a massive amount of force being applied to that wheel. It’s possible they exist, but I haven’t seen one. It is unlikely the suspension failed like that while driving normally. In almost every wrecked Tesla I’ve seen the wheels come off, which I think is by design to absorb some of that energy in the wreck.
As for insurance – Progressive is only costing me $260/6 months for a policy that includes 250k/500k, comprehensive, collision, etc. No shortcuts taken on the insurance policy.
Odometer is at 19,999 miles right now, took delivery around Oct 1st.
[1] https://www.roadandtrack.com/new-cars/a28008116/tesla-model-3-build-quality-bob-lutz/
haha hahahaha oh oh oh yeah definitely this is Tesla Motors we’re talking yeah
I don’t think that thread says anything about the expertise of the team that would have to implement multithreaded code, or anything about the overall level of development expertise at Tesla, really. If you’ve worked in software for a while, you should have plenty of stories like that yourself. (If you don’t, I contend you’ve been unusually lucky with your choice of employers.)
[Comment removed by author]
I really don’t like this idea. Think of all the edge cases. shudder
I get the concern, but there are telephone systems which have over 20 years continuous uptime using Erlang hot code updates without a visible outage.
There is a somewhat qualitative difference between a phone switch crashing and a car suddenly unable to steer or brake going through a schoolzone and crashing.
I don’t mind reaping child processes in my programs, but I’d prefer my sedan not duplicate my behavior.
Sure - avoiding crashes in the phone switch isn’t even that important, and they still did it.
If anything, that’s a stronger argument that it’s hard to get wrong.
What exactly are you planning to write in Erlang?
If you mean the part where the computer runs a bunch of nasty heuristics to convert camera pictures and radar scans into second-by-second actions, don’t systems like TensorFlow normally use SIMD or the GPU for parallelism rather than threads, to avoid the overhead of cache coherency and context switching? When your tolerance for latency is that low, you do not use Erlang.
If you mean the part where you use map and traffic data to do your overall route, I don’t think you need to be that fast. You’re spending most of your time waiting on the database and network, and could probably use Erlang just fine. The important part is the fast self-driving heuristics system cannot block on the slower mapping system. The driving system needs to send out a request for data, and keep driving even if the mapping system doesn’t respond right away.
I was being facetious, really. You wouldn’t run BEAM on the AP computer; it’s not meant for that kind of data crunching.
It is my understanding that the MCU is the middleman between the AP computer and the car’s hardware – this is how it also applies firmware updates to various parts of the car.
So I would write the AP control plane in Erlang/Elixir for extremely reliable low latency message passing. I expect the MCU is receiving values from the AP hardware which it acts upon by sending the signals to the car’s hardware. This also means it’s extremely unlikely to crash the process from bad data coming from the AP computer.
This is a guess based on what I’ve seen inside the MCU, but haven’t bothered digging too deep.
I’m also confused about why you think Erlang is not low latency?
The language that’s designed for safe multithreading and high performance is Rust. BEAM languages wouldn’t provide acceptable performance for this use-case.
Rust has no formal spec and should stay far away from systems that control the life or death of humans for now.
The languages used in successful projects in safety-critical field had no formal spec. That’s mostly C and assembly with some Ada, C++, and Java. So, Rust would probably be an improvement unless it was a group throwing every verification tool they can at their C code. It has the most such tools.
To be fair Ada has a pretty decent specification and SPARK/Ada probably has the most usable verification tools for industrial usage today, as long as you want specifications that are more expressive than what your type-system can capture. The Rust system may be very good at catching ownership-related mistakes, but there still currently exists no automated tools to verify that, say, a function that claims to be sorting data actually returns a sorted result.
You’re right in that Ada/SPARK can get further in correctness. Most in safety-critical systems use C subsets with no formal methods, though. There’s lots of review, lots of testing, and recently more use of automated analyzers.
Even so, Ada still has Rust beat on that given there’s more tooling for analyzing and testing it. C has even more than Ada.
We should strive to make better architectural decisions though.
Does Elixir have a formal spec? :)
No, but I wonder how this works in relation to JVM / BEAM. Is the formal spec really about the specific language or is the behavior of the VM sufficient? I’m not aware of different JVM or BEAM languages being able to do things that are impossible in Java/Erlang.
Need more info, but it’s interesting to think about.
Countless other languages are designed for high performance and sufficient safety.
There’s a presentation about this computer on youtube https://www.youtube.com/watch?v=-b041NXGPZ8 - it’s mind blowing.
I watched the latter part where they describe the neural network accelerator, and well, it seems to do matrix products pretty fast! This is the kind of stuff that keeps NVIDIA awake at night :)
Would be interesting to see how the low-level programs for this chip look like. Do you get just a million lanes wide multiply-add and a couple of instructions to calculate the activation functions?
Anyway, Tesla is betting heavily on deep learning here, which shouldn’t surprise anyone since Andrej Karpathy is their AI lead. I hope the new system will be much better than the current Autopilot that has already caused deaths of four people.
I definitely got a pro-Tesla bias too, but at least they admit it up front. From the author’s bio: