The Ugly Truth about Software

[zefram47]

I thought about this one for a bit.. and if it hadn't been fixed so quickly I was going to try different scenarios to reproduce. One thought I had was I wonder if they are using a queue for tasks. They may do this for long running processes or just to keep too many operations from happening at the same time. anyway.. one thought was if you happened to touch the door open/close area it may add to the queue but can't act on it yet because the charger is plugged in. Who knows how the timed close worked, it could have cleared out the queue for door operations when unplugging?

I'm really just guessing of course.

I've definitely seen some weirdness like if you interact with a hardware button before the seat/wheel go to their set position sometimes it behaves in an unexpected way. If you try to drop/raise more than one window at a time they don't always auto-up/down as expected.

That said, when the chop chop was happening I didn't have it happen for a few days and then every unplug event got the chop. Didn't seem to be any rhyme or reason to it from the outside.

Sponsored

 

[Sgt Beavis]

On virtual machines running on a 4 year old vsphere host and an internet exposed vCenter.

Yes, I have seen this more often than I care to mention….

 

[the long way downunder]

I'm not sure the "writing assignment" of this thread is sufficiently unambiguous to be achieved, but in the spirit of most of my software career, I won't let a lack of definition of success deter me from trying. I'd draw attention to Musk's manta of "all input is error" and how I wish I'd heard those words in the 80's or 90's.

I could and should write a book on my software career in Silicon Valley since '93 … before that in the Navy.

My world was Unix replacing mini-computers and mainframes. We went abruptly from buildings full of floors of deafening refrigeration and supermarket aisles full of drives, to equipment showing up from Silicon Graphics, Sun, Wang in boxes smaller than the printers of the day.

My primary field is data design … building a database that repels errors and reveals weaknesses in business processes. That's a field of work that requires more than I can post to a Rivian forum. : )

I cut my dev teeth on compartmented mode workstation RDMBS image management (medical records, long before object oriented languages or databases) then system yield optimization (pretty much built google's page rank for a government document system that was bigger than the Internet back then, late 80s.) Not one of us at the time (a team of several thousand devs) once thought "oh this is such a cool search engine, let's index the Internet.") We coulda/shoulda/woulda been billionaires, if but for a single creative orthogonal thought to have come from any one of us. Back then, income wasn't a priority, we were altruistic scientists. The reward was designing perfect software, elegant code, peer review and the love of the game. It wasn't till I got to Silicon Valley that encountered the whole other culture of "we're here for the stock price" and I still wonder how anything ever got done with buildings full of people preoccupied with the horse race of Oracle vs IBM vs Sun not as technological pioneers, just ticker symbols.

I was lucky enough to retire from a successful startup in '99 and do my own thing after a year or two of "retirement" realizing that my vocation was my avocation and I missed having at least two keyboards in front of me.

The first time I saw google.com I quit everything else and gathered my shekels to buy shortly after the IPO. Google opened my eyes to orthogonal problem solving. The second half of my career was guided by the philosophical caution: "when you find yourself in the mainstream, it's time to start swimming sideways." I think that's a paraphrasing of a Mark Twain witticism.

I wish Elon Musk in the 1990s had shared some things about systems design which he's spoken about in the last 10-20 years. He's also said some profoundly stupid things, but such is the nature of scientific endeavor. People outside the systems development industries tend not to realize the inevitability and necessity of crashing a few rockets in order to build one that doesn't crash too badly. One example of brilliant insight is his borrowed paradigm of "all input is error." We worked according to that "wise old saying" and I put that term in quotes because I used to work for a couple of PhD research scientist geniuses who often asked their underlings to stop using complex terms "you're not being paid by the weight of the word, we're asking for clarity and brevity." I'm already breaking that rule by posting to this thread. : ) They would cc:all on our in-house secure email system any time someone used "didactic" or "aphorism" for example, then archive the offending email to await it being sent according to their mantra of clarity and brevity. The purpose of the lesson was to bring that mindset to systems development.

From the UX down to the system architecture, Rivian, Tesla – just about everyone – tends to forsake the criteria of clarity and brevity.

 

[the long way downunder]

I thought about this one for a bit.. and if it hadn't been fixed so quickly I was going to try different scenarios to reproduce. One thought I had was I wonder if they are using a queue for tasks. They may do this for long running processes or just to keep too many operations from happening at the same time. anyway.. one thought was if you happened to touch the door open/close area it may add to the queue but can't act on it yet because the charger is plugged in. Who knows how the timed close worked, it could have cleared out the queue for door operations when unplugging?

I'm really just guessing of course.

Having played around with CAN bus systems for years in Porsches, I'd say it's in the communications and state of the device as it receives various signals. The nature of systems design is to have self-contained objects. The tendency in iterative development is to solve problems using methods in ways that have not been fully anticipated or intended.

It's a problem I call "is it the singer, or is it the song?" In this "chop chop" charge port cover example, I imagine there's a time-out conflicting with "ready to close" state. There's also some sort of force-feedback loop (either software or hardware that prevents the cover from closing with too much for that could damage the mechanism) which may have been defeated or disabled.

In IBM, this is called "mature system" design. Things like a toggle switch are eschewed for their ambiguity. "The door is half open. If I press the toggle will the door open or close?" IBM would have a close method and an open method. This benefit in UX is better illustrated in the ill-fated power tonneau control. Instead of a toggle, which can never be predicted with certainty, there should be an open button and a close button.

From bumper to bumper, Rivian has left the R1 design with loose ends which should never have been signed off as good to go at the systems analysis and design stages. One glaring example is the number of modes that prevent the vehicle from moving, or moving at full power. There must be an override setting for the driver to control the vehicle, it is not for a software developer in Palo Alto back in 2018 to, for example, put the vehicle into turtle mode because the latch on the gear tunnel door isn't fully closed (or has malfunctioned.)

 

[s4wrxttcs]

there should be an open button and a close button.

Anyone that has ever tried to close their tonneau half way would likely agree. The single button press means you don't really know which way its going to go when pushed.

This one button press thing is also why I never linked homelike to summons on my Tesla. I do miss Homelink as Rivian doesn't have it, but I never did upgrade my garage door to one with a separate open and close functionality.

As to the lack of redundancy I've been really annoyed with that in recent years with modern cars.

When my camper van got rear ended when parked I couldn't get it to turn on because the CAN bus was shorted in the tail light. I had to go to the HW store to get wire clippers to get myself going again. In the tail light was the sensor for the blindspot monitoring which had nothing to do with starting the vehicle, but since the CAN bus is shared a lot of things didn't work.

With my previous Tesla Elon told people it would be fault tolerant and could drive even when there was a failure of a motor. But, it turns out it was only with a rear motor failure.

The Rivian R1 is no where near as a fault tolerant as it should be given that its an adventure vehicle. I should be able to drive with one motor and all the tunnel doors open.

I can't really say I'm happy with any of my modern cars when it comes to redundancy. They're more like delicate little flowers that can't take a punch.

 

[dsmithsalinas]

Yes to all of this, but also, I don’t think people can really understand the amount of time it takes to get from the start to the end and amount of prioritization that happens in software development.

I see that all the time as a PM, “oh adding that button seems easy, we can knock that out” then turns out adding that button impacts xyz and now the “small enhancement” is a large Boulder.

 

[jakef801]

I, for one, have been very impressed with the monthly (on average) OTA rollouts since I took delivery in Aug. '22. I can only imagine what goes into all of it - way beyond anything I claim to understand. I appreciated the Q&A with the head software developer on IG yesterday. It sounds like a lot of new and exciting things are in the works. I'm especially interested and curious in this "big" towing update, expected in summer, as I tow A LOT.

 

[Craigins]

Many of Rivian's software pitfalls are not even on the programming side. They are on the teams generating the software requirements.

This is very obvious in the driver+ code. Take follow distance as an example, the truck tries to keep an exact follow distance from the vehicle in front of you. It seems like 1 inch too close and it breaks, 1 inch too far and it accelerates. It should be a target region not an exact number. You see this with the lane centering, it constantly adjusts back and forth to be exactly in the middle.

The charging thresholds used to be this way too. Set it for 70%, when it hit 70% it turned off, when it hit 69.9% it turned back on.

This isn't bad software, it is bad requirements. The person in the middle that translates from what the hardware engineers say to software requirements isn't doing a good job.

I'm sure the chopping charger door was a similar issue. Just like the dirty front camera disabling driver+.

 

[OldGoat]

Many of Rivian's software pitfalls are not even on the programming side. They are on the teams generating the software requirements.

This is very obvious in the driver+ code. Take follow distance as an example, the truck tries to keep an exact follow distance from the vehicle in front of you. It seems like 1 inch too close and it breaks, 1 inch too far and it accelerates. It should be a target region not an exact number. You see this with the lane centering, it constantly adjusts back and forth to be exactly in the middle.

The charging thresholds used to be this way too. Set it for 70%, when it hit 70% it turned off, when it hit 69.9% it turned back on.

This isn't bad software, it is bad requirements. The person in the middle that translates from what the hardware engineers say to software requirements isn't doing a good job.

I'm sure the chopping charger door was a similar issue. Just like the dirty front camera disabling driver+.

I joked earlier about saying that some unexpected behaviour was a feature; however, your are spot on. I've seen many times where the development side thought a requirement was stupid, expressed that concern, and were told they were the stupid ones. So the 'feature' was included only to have to the feature eventually removed, changed or never be used because the clients thought it was stupid.

I am reminded of the time I was complaining about the fact that a new client I had just met with was asking about functionality that not only didn't exist in our product but was damn near impossible to implement. The head of sales put his arm around me and said "never confuse sales with implementation" Alas, s#!t flows down hill from sales to design to development to implementation to customer support. (I've always said that every sales person should have to spend a month in customer support!)

 

[Zoidz]

My company writes PLC code for customers, the stuff that controls manufacturing plant floor equipment and processes in real time. We start with fail-safe/alarm coding with the intent that it will prevent product loss, damage to equipment or injury to people if something goes wrong with the equipment or sensor devices. For *troubleshooting and testing*, PLCs also include a feature known as “forcing“ which allows you to bypass all logic and force enable/disable an input or output.

The problem we see several times a year is that at 3 AM when a sensor fails, a third shift tech forces (fakes) an input or output signal to keep production running because they don’t have a spare replacement device. It’s a quick and easy shortcut, and they go back to the breakroom to drink more coffee. Then they forget to tell first shift about it, and the force becomes forgotten. Or first shifts orders and installs a new device, but it does not respond as it should because of the still enabled forced signal. At some point, we get a support phone call because someone realizes the system “is not working like it used to” and calls us to fix *our* buggy programming! We tell them never to use forces for production, or to at least do an email blast to all techs and engineers, but they don’t. We have seen tens of thousands of dollars of product loss due to things like this at a couple factories, fortunately no injuries. It has resulted in a technician getting fired at one of our customers.

 

[quartz]

You want the ugly truth?
- IRS is still using COBOL systems from 70's to process some taxpayer records. They have to bring this one person out of retirement for consulting because everyone else who knows the systems are dead.
- NASA is still using excel spreadsheets as databases.
- Many American-only software jobs are outsourced to foreign staffing companies that outsource those jobs back to Americans.
- I once worked with someone who added a timer in DB query, and over the next six months lowered the timer by a little every few weeks and claimed "performance improvements".
- I also worked with someone who outsourced their job to someone in India, gave them VPN access, etc. So they could run their own business while on the clock.
- These two people both worked at a government facility where you need an escort to piss if you have a contractor badge and they also threw retirement parties every other week.

Sponsored