Hacker News

54 Comments:
not_kurt_godel said 2 months ago:

I don't doubt the line is much more efficient and flexible as claimed, but the article doesn't do a very good job explaining exactly how the gains are achieved beyond hand-waving about clever contraptions that are faster than robots and the ability to dynamically move stations around. It seems difficult to imagine that hordes of industrial engineers haven't done the calculations on each of these methods countless times at all car manufacturing companies. There must be deeper, more subtle factors that explain why Toyota's particular combination of techniques yields such superior results. Perhaps it's just the culmination of a lot of long-tail optimizations that only Toyota has had time and capital to let mature or maybe there really is just "one secret trick" that underpins it all - the article doesn't really give a good sense of what it might be. In either case it'd be interesting to see a detailed breakdown with numbers comparing the various lines with explanations for why particular decisions were or weren't made to truly understand why the system works so well.

rubinelli said 2 months ago:

> It seems difficult to imagine that hordes of industrial engineers haven't done the calculations on each of these methods countless times at all car manufacturing companies.

Yes, it's the Ford method: you hire the smartest people you can to lay out the best plan, and a bunch of interchangeable grunts to just execute. But it turns out that the person that has to do that work 40 hours a week has a much better chance to find ways to make it more efficient, and if you build a culture where that's encouraged, that's what they do. If there is "one secret trick", it's probably that an employee that feels that they own their work does a much better job than one that acts like a cog in the machine someone else designed.

georgeek said 2 months ago:

That's the whole concept of kaizen, or improvement: https://en.wikipedia.org/wiki/Kaizen

stdbrouw said 2 months ago:

> It seems difficult to imagine that hordes of industrial engineers haven't done the calculations on each of these methods countless times at all car manufacturing companies.

Difficult to believe unless you've ever worked in an industrial setting.

spyckie2 said 2 months ago:

it sounds like the benefits of the line is the flexibility, not necessarily efficiency gains. I think the idea of the article is that fixed production is a business weakness in today's fast moving consumer market.

This also doesn't sound like something Tesla needs yet, as this seems to be great for building different types of cars based on demand rather than maxing a single car output.

FPGAhacker said 2 months ago:

I thought this was basically what Tesla had to do with the add on tent manufacturing the 3’s.

_pmf_ said 2 months ago:

> There must be deeper, more subtle factors that explain why Toyota's particular combination of techniques yields such superior results.

Why? That's how engineering works; small improvements aggregated over time.

nkoren said 2 months ago:

No, that's how certain types of engineering optimisations work, but as a process is susceptible to getting stuck in local minima. This is clearly a different type of optimisation, requiring the simultaneous adoption of a number of different yet complementary techniques.

linksnapzz said 2 months ago:

For anybody that cares, much of this process has its origins in Taiichi Ohno's work in the '70s. His original "Toyota Production System" book of 1978, which is still in print, is short, accessible, and enormously enlightening. Likewise, a comparative study of why the Toyota system works in comparison with other manufacturing approaches is the 2nd edition of Womack's "Machine that Changed the World". Also, other books by Shingo, Imai, Deming et al. have lots of good information.

bluGill said 2 months ago:

The gain is this: it doesn't produce a car that Toyota doesn't sell. That is if Toyota sells 90 cars they make 90, if they sell 100 they make 100. If gas is cheap they can make more gas guzzlers that consumers want, if gas it cheap they switch to efficient cars. In a traditional line you make the same number of cars every day. If consumers wants something else - too bad for the line, either you stockpile cars and hope they sell before you clearance them at a loss; or you shut down the line and produce zero - too bad for the customers who do want that one.

Note that this is less efficient if you correctly forecast how many cars of each type you sell. An assembly line optimized to produce a lot of one car is cheaper IF you actually sell a lot of those cars. Which way to make the trade off is a complex decision with pros and cons. As technology changes different factors become important and the best changes.

bluedino said 2 months ago:

Based on their product line I'm surprised Toyota is so 'flexible'. They seem to go the longest between model revisions and are selling cars without features that were widely used 10 years ago. They've been using the same interiror parts in some vehicles (runner) forever.

mox1 said 2 months ago:

And yet, they continue to basically sell out year after year.

Additionally, you cannot order most Toyotas, the dealer usually doesn't even know what color or trim levels they are going to get in advance.

Its the Trader Joe's model. Have super high quality (and/or well priced) goods and consumers will bend a lot on choice. In fact, studied have shown that too many choices "freeze" consumers and they just get frustrated.

I'm sure a lot of consumers would take a 2nd or 3rd choice color Toyota, over ordering whatever they want from a domestic brand.

tim333 said 2 months ago:

I think the root of a lot of Toyotas performance is cultural. When asked if they were worried about VW selling more

>a Toyota spokesman said that the brand was not focused on chasing volume for the sake of it. "We believe that our sales volume is just the result of our focus on making ever-better cars and providing better customer experiences."

And that has mostly been true - focus on better cars and presumably better production techniques over many decades. It may seem obvious I'm not sure other companies focus like that.

athenot said 2 months ago:

I know that car analogies for software is an over-done metaphor but while reading this article I'm thinking of heavy processes for software deployment that are hard to change and in my mind I'm screaming "this is what agile looks like!"

Processes and controls are very important for quality, but as business goals shift and evolve, the processes also need to evolve. But designing processes that can evolve gracefully is not a trivial task, especially in risk-averse organizations.

sodosopa said 2 months ago:

> this is what Agile looks like

FYI Kanban came out of the process on Toyota assembly lines.

redwall_hp said 2 months ago:

QR codes, too.

Xunxi said 2 months ago:

What I would've loved to see is the comparative efficacy captured in motion for (lets say) a "layman's observation" of production. Just a glimpse into how it works out on the "flexible line"

nabla9 said 2 months ago:
csours said 2 months ago:

That's a very nice video of a production plant, but I don't think it's the one in question.

csours said 2 months ago:

As far as I can tell they only talk about the Takaoka #1 Line, which has a rate about 1 car every 57 seconds. This is just about normal for a regular car plant.

For comparison Honda has a flexible plant in Marysville, OH, which I've heard has a run rate of one vehicle every 4 minutes (but take that with a very large grain of salt).

lnsru said 2 months ago:

“They would tell you that robots are great for repetitive tasks like welding, or painting, but they fail at picking and handling small parts.” Are robots really that bad nowadays? I imagine, it would be a problem in VGA-only camera years. Now we have tens of megapixels, lidars, tons of computing power (cheap!).

csours said 2 months ago:

Sure, sure, now you have a robot. How much did it cost you? How much did you have to renovate to install it? How much did you change your material handling? Your part design? Your vehicle design? Your assembly design?

Robots also only really work in stop stations, not on moving vehicles (with a few notable exceptions like paint shop).

Does all of this cost more than hiring a person? More than twice as much?

How often does your automation fail? Is it more than 1 out of 1000 cycles? You have over 10,000 automation steps. That would leave 10 errors per vehicle. When automation fails, it often stays down for a while for investigation and repair. Any station down will stop all stations both before and after it (eventually)

With 10,000 automation steps, and a run rate of 1 vehicle per minute, how often can you lose a robot?

----

Disclaimer, I work for GM; any opinions are solely my own.

linuslarsson said 2 months ago:

I think a lot of people see "automation" as this end-all solution. When in reality it's expensive, unflexible and unstable. Most producing companies don't have good enough preventative maintenance for it to work at all. When you have recurring problems that your own technicians can't solve you're in deep water. It's easy to hire and (unfortunately) fire workers, but a bot you're stuck with.

I see robots/automation as a solution to a problem. That problem is safety. Either repetitive work that slowly breaks down the human, or "this is not good for you"-work that's directly dangerous.

I like that question from the sensei in that goldratt book: "did the robots make you more productive?".

Trying to invest your problems away is a waste of time and money if you don't know, and continously work with your processes. Or what they call kaizen in the article.

lnsru said 2 months ago:

Thank you! That’s exactly what a guy from Volkswagen plant told me. They don’t need a 99,9% working robot, because it’s not enough when manufacturing one car each minute. The robot might be cheaper to buy, but they cannot afford to stop the line for such errors.

xvilka said 2 months ago:

It is just a matter of time before better, more precise and accurate robots will be created.

linuslarsson said 2 months ago:

Precision is not the biggest issue here. There are very precise robots out there.

The big issue is improvement of standardized work. If you automate early you get tethered by subpar standards. This is natural when you have workers: you iterate work instructions and what tools to use together as a group. This is basically how Toyota is so good at producing: every worker know what to do and when to do it. They haven't always been like this; they have slowly iterated into what they are today.

Improving on work instructions for a robot is tideaus at best, and really, really costly as a worst case: you either re-program, or you're forced to reinvest if the robot isn't capable of the new movements/steps.

carlmr said 2 months ago:

Yes, but you still have the problem of bringing this process into your production line and finding out how it makes you more efficient or better.

Musk also had to learn some lessons Toyota learned years ago, because he trusted a bit too much in automation.

If you want to automate something, you need to know exactly all the corner cases, every step of what the human is doing to make it work. The robots are precise enough, but it's often not that easy to describe the tasks to them.

lnsru said 2 months ago:

How are these robots programmed? Just a sequence of position movements and affector actions depending on few digital inputs of light curtains or sensors?

bluedino said 2 months ago:

In the most simple cases, yes. Parametric programming is popular, where you can define an item and then scale it in a number of factors and only need one program for thousands of possible premutations of that item.

Think about a company that produces screws, they don't have a program for every possible screw they make. Same goes for a more complicated item like a metal box with a door on it.

fiblye said 2 months ago:

I work on the automation for some car lines.

The robotics are fine. The problem is, in the off chance that an error happens, the results are absolutely disastrous. The biggest problem being that there are too many potential things that can go wrong and it's very difficult to program the machine to be ready for every possibility.

Some known potential errors are checked for, but a problem is that when the machine detects an error, the line has to shut down so that someone can run over and resolve it. If they have a human doing the job in the first place, the issue can be resolved immediately.

For the most part, automation is just used for major error detection every step of the way (errors being things like cracks in the engine), but a human assembles everything and checks for any errors that the machine can't detect.

GlennMercer said 2 months ago:

Exactly. Thus we see ultra-high automation levels in stamping, painting, and welding, where deviations are minimal. But once in final assembly/trim etc. there is a deviation every few cars! The headliner flops around and won't snap in correctly, the line jerks to a halt just as a nut driver engages, etc. COULD you automate for all these case? Yup. But why would you, when you can have ultra-flexible humans. This was always my question about Tesla's automation quest (now apparently dropped): assume maybe 7 hours in final assembly, say $70 an hour labor fully loaded, call it $500. Put in $100 million in extra automation cut the 7 to 3, how long before THAT pays off? Maybe you CAN do it, but it verges on Juicero: did you really need to automate squeezing a bag of veggie pulp?

rohit2412 said 2 months ago:

I'm not sure of the numbers, but.

Cut 7 to 3, cut 500$ to 200$ say by rounding. Save 300$ per car.

How many cars do you need to recoup the 100 million$? 100m/300= 300k cars. Just one year of model3 production.

nexuist said 2 months ago:

I think the bigger problem is in sensor precision, particularly those sensors responsible for reporting what state the arm in (i.e. so the computers onboard can calculate the position of the arm relative to the object it needs to pick up). Sensors with low tolerances are extremely expensive and those particular robot manufacturers may have opted for less accurate sensors to save on costs. They may work perfectly fine for moving things like doors and frames; but the differences make themselves known as soon as you try to interact with bolts and etc.

When I did robotics we had a perfectly capable Linux box running PID loops several hundred times a second; but our robot still sucked b/c our on board motion sensors were simply not precise enough to give us accurate data to feed the algorithms. As usual, it's all about the data.

2YwaZHXV said 2 months ago:

As someone who works with modern industrial robots, like you'd find in a car factory, the sensors they're using for the arm positioning are unbelievably precise. I can have the robot stopped, with the brakes on, and push on the end of the arm with my fingers and see the joints registering the deviation (and not by 1 or 2 counts, but many more), while I don't perceive any significant movement of the robot.

metaphor said 2 months ago:

Forgive me for perhaps being naively optimistic, but I'd probably give industrial robot design engineers applying optimal control methods to solve performance robustness issues the benefit of the doubt that their error budget would properly account for and derate any critical sensor.

jbay808 said 2 months ago:

Did you remember to low-pass filter the derivative gain? Properly accounting for sensor noise is just as important as getting low-noise sensors.

bluedino said 2 months ago:

The vision systems aren't very good in practice.

2YwaZHXV said 2 months ago:

The interest level in achieving the best result (and the required associated time, patience, planning, skill, and potentially cost) is likely the more limiting factor. I"m saying this as someone that works with vision systems for industrial settings. Often I see garbage images with garbage logic and they complain about garbage output. Then the camera is removed because "it doesn't work".

crb002 said 2 months ago:

Takaoka Line 2, when Category Theory meets auto manufacturing. Love the carts they move cars on. Curious if Tesla attempts such a line next.

protomyth said 2 months ago:

The article spends its last three paragraphs talking about Tesla:

When we were theorizing about a gifted car plant, the example may have vaguely sounded like Toyota’s NUMMI plant, that was, for practical purposes, given to Tesla in 2010 in exchange for shares. A year later, Toyota opened its first simple, slim and flexible plant as a pilot north of Sendai, Japan, and a few years thereafter, Toyota sold its shares in Tesla. If the relationship would have lasted a little longer, and if Elon Musk’s hubris would have been a little less pronounced, Tesla could have learned something.

Before landing in production hell, and pitching a tent to make his Model 3, Musk promised that he would out-Toyota Toyota when it comes to lean manufacturing. He had a catchy name for Tesla’s non-existent miracle plant. He called it the Alien Dreadnought. I ask Akahane what Toyota calls its super-flexible line.

“We don’t really have a name for it,” Akahane says, and he doesn’t seem to think it needs one.

Judgmentality said 2 months ago:

You should never try to automate something before that something is already working. You can't optimize something that doesn't exist. It's the equivalent of scaling a company before you even have a product, let alone product-market fit.

I completely agree with you, I'm just surprised how often I see people try to do this. As someone with lots of experience in advanced robotics, people ask for my advice on automating things quite a bit. I always tell them they're approaching the problem wrong, and they never listen until it's too late and usually (because it's often a startup) the company just outright fails.

You are not the first company in the world to try to automate <insert blank>, it just turns out it's roughly 2 orders of magnitude harder than you think it is. No, I will not build a production ready system that involves an unsolved robotics problem with a ~$100k budget.

rangersanger said 2 months ago:

“The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.” - Bill Gates.

He didn't have a third rule that says "If there's no process, automating a new will expose your hubris" but it's implicit.

bryanlarsen said 2 months ago:

Elon just claimed that designing a rocket manufacturing process is 10X as difficult as designing a rocket, and that designing a car's manufacturing line is 100X as difficult as designing the car. I'm fairly certain he didn't believe that 10 years ago. :)

nikofeyn said 2 months ago:

> Curious if Tesla attempts such a line next.

and how exactly would they do that? their non-automated lines barely work.

nkoren said 2 months ago:

That's FUD BS. You don't produce nearly a quarter of a million cars if your production line "barely works". The truth is that they're now routinely exceeding a thousand Model 3 cars per day.

https://www.bloomberg.com/graphics/2018-tesla-tracker/

nikofeyn said 2 months ago:

they have a history of manufacturing problems, do they not? they may not be having them at this very moment, but it is well documented and discussed that they have had issues. i think when you're building cars in a makeshift, outdoor assembly line just a year ago, you can't say that you have everything together. maybe they've hit their numbers, but their manufacturing process hasn't been elegant. is this incorrect? and this is a personal opinion, i think founded in some objective reality, but i don't believe teslas to be well-manufactured or designed from a non-owner's perspective. i have sat in them and ridden in one, and it just is not an impressive design and fit and finish to me compared to more entrenched car companies. this is both a design and manufacturing issue, from my understanding.

the point of my comment, which i thought was clear, was a counter to the continual tesla hype in that "can't wait until tesla addresses <this> problem" as if they've conquered all problems that have come before. toyota has it together and has a significant amount of r&d that goes into their advanced assembly lines. it's a bit laughable to think of tesla suddenly taking them on in automation given their historical troubles. or am i wrong?

also, those numbers are fine, but tesla isn't even the largest electric car manufacturer in the world, by sales numbers. and they don't approach traditional car makers. for example, kia sells around 600,000 cars in a single year, and that is including say three times as many models as tesla has available. tesla isn't some manufacturing darling story as far as i can tell. they have no ability to yearly iterate like other car companies. or do they? how do they update their models?

i think the point remains: it is flippant to suggest they can just suddenly innovate on an automated line.

colordrops said 2 months ago:

> their non-automated lines barely work.

citation?

zemvpferreira said 2 months ago:

Tesla production problems are well documented: https://www.wired.co.uk/article/tesla-model-3-production-sto...

Heck, their line was catching fire on a weekly basis a few months ago. Academic citations hardly necessary.

jsight said 2 months ago:

Now you are just making things up. Citations are really necessary when you start claiming things are catching fire on a weekly basis.

nikofeyn said 2 months ago:

i think they must be referring to tesla factory fires happening in fall of 2018, which is easily searchable (how i found it). apparently there were reports of frequent fires before a big one.

jsight said 2 months ago:

I am aware of fires both last year and this year. I am not aware of fires on a production line on a weekly basis. That claim is simply an absurdity.

pwaivers said 2 months ago:

> "After pulling a few strings to get into the Takaoka plant, you will see the Gordian knot become untied."

Um... the Gordian knot was not untied, but rather cut by a sword. It's supposed to be a bold, out-of-the-box solution to a seemingly impossible problem. I don't know if the author of the article chose the wrong word on purpose.

said 2 months ago:
[deleted]
said 2 months ago:
[deleted]