The cost of good Programmers/Developers.

[Cosmik]

I have a washing machine, I use it all the time. No one knows that washing machine like I do. I am an actual expert on it.

Except I have read reports that on occasion the drum flies apart when spinning at 1600 rpm, making a huge mess. Do I trust that mine will never do the same? No. Do I trust the engineers who designed it to have put in safety features to stop it? No. Do it think that the employment of 9$-an-hour programmers is the cause of the problem? No.

So I stand well back when it's going...

 

[DonH56]

So has your brother ever had this system kick in?

Yes.

And does he know the designers and software engineers and programmers, and does he trust them? Does he have a view on whether a single angle-of-attack sensor is sufficient and whether he would always be able to take over if the thing failed?

He's a pilot, not an engineer or a programmer. As I said, not my field, and not interested in a pissing contest. Hopefully someone with experience can chip in.

 

[mansr]

No, but my brother has, and flies them, and I trust him... Piloting is his job and not something I really know much about. Not really interested in a pissing match over it and don't claim any knowledge of my own. Not sure it is really germaine to the OP.

Your brother is right in saying that automatic systems fail, and when they do, the pilot should take over. In the 737-MAX crash, however, the pilots were unable to take over since the automatic system refused to disengage.

 

[Cosmik]

As I said, not my field, and not interested in a pissing contest. Hopefully someone with experience can chip in.

No pissing competition intended. But the idea that experience offers insights is to me, a non-starter. Like my washing machine, 99.99% of people will never encounter the precise combination of circumstances that could lead to disaster. How can mere experience be useful?

Surely, it's questions such as: is it sufficient to rely on a single angle-of-attack sensor? Would two be sufficient? Should they be different types? What if both go wrong? Should the system be engaged by default and have to be overridden by pilots or the other way round? Or should the whole thing be scrapped in favour of a design that can't do this? Etc.

 

[DonH56]

But the idea that experience offers insights is to me, a non-starter.

On this point we fundamentally disagree. To me experience matters greatly and some of my most valuable insights came from testing and real-world use of my designs.

As for the Boeing issue, I am not competent to comment. My brother was trained with the system working and malfunctioning to learn to handle such situations and has dealt with it in practice (i.e. during regular flights). But if his experience does not matter there's really nothing more for me to say.

 

[Cosmik]

On this point we fundamentally disagree. To me experience matters greatly and some of my most valuable insights came from testing and real-world use of my designs.

As for the Boeing issue, I am not competent to comment. My brother was trained with the system working and malfunctioning to learn to handle such situations and has dealt with it in practice (i.e. during regular flights). But if his experience does not matter there's really nothing more for me to say.

That's why I clarified it with my washing machine example. My extensive experience and, I might add, expertise, gained from 'flying' that washing machine is no substitute for knowledge of the actual issues at the design stage.

 

[Thomas savage]

That's why I clarified it with my washing machine example. My extensive experience and, I might add, expertise, gained from 'flying' that washing machine is no substitute for knowledge of the actual issues at the design stage.

Both situations could lead to brown stains ..

 

[Dialectic]

Some interesting thoughts about this question from former software engineer, entrepreneur and commercial pilot Philip Greenspun:

https://philip.greenspun.com/blog/2019/07/03/boeing-hires-software-engineers-for-9-hour/

Sorry to digress, but he's written a little about audio, too--in particular, an article that is a good read but contains some statements with which most of us on ASR will take issue:

https://philip.greenspun.com/materialism/stereo

 

[Krunok]

But if his experience does not matter there's really nothing more for me to say.

If automatic system refuses to disengage when failing no man's experience nor knowledge can help. In fact, nothing can help. as the machine keeps driving the plane down and won't let go. Simple as that. It has nothing to do with fluid dynamics nor with pilot experience - it's an error in implementation of the algorithm with fatal consequences.

 

[Thomas savage]

If automatic system refuses to disengage when failing no man's experience nor knowledge can help. In fact, nothing can help. as the machine keeps driving the plane down and won't let go. Simple as that. It has nothing to do with fluid dynamics nor with pilot experience - it's an error in implementation of the algorithm with fatal consequences.

Bit like when the Mrs puts a stray pair of black knickers in with your cricket whites..

 

[Krunok]

Bit like when the Mrs puts a stray pair of black knickers in with your cricket whites..

There you can maybe avoid fatal consequences if you keep your mouth shut.

 

[amirm]

Some interesting thoughts about this question from former software engineer, entrepreneur and commercial pilot Philip Greenspun:

https://philip.greenspun.com/blog/2019/07/03/boeing-hires-software-engineers-for-9-hour/

He is off the mark. He says people over 50 are not needed as developers. That is just false. If you know what you are doing, you can easily get jobs without age being a barrier.

On income, many companies have stock/stock options that significantly increase the amount of wealth you can accumulate. He talks about a nurse. Show me a nurse that earns such benefits. Does not exist. Stock options/stock are the key to going beyond paycheck to paycheck.

 

[Krunok]

What may not be so clear to people outside of software development idustry is this: most of you may think that this is a result of developers mistake. And while it is so it is very important to understand that developers are allowed to make such mistakes. And they will always be making such mistakes during the development process, no matter if internal or outsourced development resources have been used. For that reason there are other folks in the process who's task is to develop error checking procedures and folks who execute those error checking procedures in the hope that all errors made by developers will be found and corrected. And this is the point where things got wrong - fatal error existed but was not found. That could happen for a number of reasons: error procedure didn't cover all possible cases, someone was reckless performing error checking procedure or for some other reason in th error checking phase. This usally happens when error checking teams are pressed for time to complete their procedures ASAP which lays ground for this kind of error to pass undetected. Sadly, it happens more often than it should, but such is the world we live in, where "time is money"..

 

[Sergei]

The article reads like an attempt by someone to partially shift the blame

Exactly. Blame shifting on more levels than one.

As I understand, the root cause was Boeing installing a redesigned engine with a better fuel economy, yet not redesigning the airplane frame to accommodate the new engine, which has significantly larger diameter. Instead, they made a decision to install the new engine higher than the engine vertical midpoint position the frame was originally designed for, thus making the airplane aerodynamically unstable.

To counteract the aerodynamic instability, Boeing introduced an additional active-control system, which was supposed to keep the now inherent oscillations in check. In itself, this is not such a bad idea: some fifth-generation jet fighters, for instance, use their purposely introduced aerodynamic instability to facilitate much quicker turning, and stupendously efficient braking through controlled stalling.

There is nothing wrong with any of the basic cost cutting ideas used here, as long as management is up to the job of managing the risks.

And that's where things went wrong. Advanced active-control systems require advanced organizations to develop, produce, and support them. According to the model developed by Ichak Adizes in 1980s (https://www.amazon.com/Corporate-Lifecycles-Corporations-Grow-About/dp/0131744267), a corporation needs to be at the Prime stage to quickly introduce new successful products.

Where do you think Boeing as organization is located today on the Adizes lifecycle curve?
Here? https://adizes.com/prime/
Or there? https://adizes.com/recrimination/

 

[Old Listener]

On this point we fundamentally disagree. To me experience matters greatly and some of my most valuable insights came from testing and real-world use of my designs.

As for the Boeing issue, I am not competent to comment. My brother was trained with the system working and malfunctioning to learn to handle such situations and has dealt with it in practice (i.e. during regular flights). But if his experience does not matter there's really nothing more for me to say.

I think your brother's experience would have been of great value in the design, debugging/testing and documentation /training processes.

A perspective based on real world use of a similar complex system might have exposed the problem before the plane went into use.

 

[StevenEleven]

This thread is a great read. I have known quite well a couple of people involved in managing software development. One of them, whose career was in Silicon Valley, told me that the first rule of managing software development is to assume your programmers are lying to you, and often this is for your own good. However, it is still the manager’s responsibility to see to it that the code doesn’t get released until it is ready for prime time.

As far as “agile” code development, I have not had very good experiences with it. To me in my specific area of work it has come to represent a business model of releasing code before it is sufficiently checked for stability or functionality, and using the customers en masse as end user beta testers, while they collectively lose thousands upon thousands of hours of productivity. Once you are using your customers en masse as beta testers, in my mind you are at a code red level of failure. I am sure there must be instances where the “agile” model works, maybe in other sectors of the economy than what I see, but in my experience it has been a train wreck (or plane wreck) waiting to happen.

Once when I was getting a little stressed one of my colleagues put things in perspective for me by saying we are not flying airplanes or conducting surgery. Well, here they were flying airplanes, and a very bad model for software development, which I have run into many times, reached its unfortunate and logical and fatal conclusion.

 

[Sergei]

One of them, whose career was in Silicon Valley, told me that the first rule of managing software development is to assume your programmers are lying to you, and often this is for your own good.

I agree in the sense that individual software developers usually have no way to accurately assess the overall readiness of a complex software product, yet some of them would still try.

I disagree in the sense that a software developer worthy of keeping on a team in Silicon Valley would be deliberately lying to his or her manager. This effect is well-described in the classic https://www.amazon.com/Mythical-Man-Month-Anniversary-Software-Engineering-ebook/dp/B00B8USS14.

As far as “agile” code development, I have not had very good experiences with it. To me in my specific area of work it has come to represent a business model of releasing code before it is sufficiently checked for stability or functionality, and using the customers en masse as end user beta testers, while they collectively lose thousands upon thousands of hours of productivity.

Classic Agile works exceptionally well for small startups, iterating and pivoting in the search of their first Minimally Viable Product. The larger the development organization, the more modifications the managers have to apply to the Classic Agile to make it work well for them.

Different flavors of Scaled Agile and Enterprise Agile are well-established by now. One of the many entry points into that subject: https://www.planisware.com/glossary/enterprise-agile-framework.

I am not affiliated with, not endorsing or promoting Planisware. They just happen to have a balanced view of the subject IMHO.

 

[Cosmik]

Classic Agile works exceptionally well for small startups, iterating and pivoting in the search of their first Minimally Viable Product. The larger the development organization, the more modifications the managers have to apply to the Classic Agile to make it work well for them.

To anyone who has worked in software development: have you ever been involved in a project that took twenty people two years to burn through several million dollars/pounds and ultimately failed, but which you are confident you could have done yourself, on your own, in a couple of months (if not a weekend)? I have been in situations like that, a few times.

Software people love the idea of organising a project a bit like software itself, and it's overkill. They also like to try 'something new' that will look good on their CV, perhaps.

I remember the development of a trivial product for which only about 300 units would be sold per year for a few hundred pounds each, but it cost millions to develop and ultimately failed, all down to the huge overheads of developing it using 'a methodology'.

 

[Snarfie]

This thread is a great read. I have known quite well a couple of people involved in managing software development. One of them, whose career was in Silicon Valley, told me that the first rule of managing software development is to assume your programmers are lying to you, and often this is for your own good. However, it is still the manager’s responsibility to see to it that the code doesn’t get released until it is ready for prime time.

As far as “agile” code development, I have not had very good experiences with it. To me in my specific area of work it has come to represent a business model of releasing code before it is sufficiently checked for stability or functionality, and using the customers en masse as end user beta testers, while they collectively lose thousands upon thousands of hours of productivity. Once you are using your customers en masse as beta testers, in my mind you are at a code red level of failure. I am sure there must be instances where the “agile” model works, maybe in other sectors of the economy than what I see, but in my experience it has been a train wreck (or plane wreck) waiting to happen.
.

Some of mine best friends are top developers specialized for instance in the Oracle environment. Regarding checking code one of them had several times overcome an culture difference. When a change in code has to be done that is complex an critical (let say a change in the source code for international payment on the institutional level) it is divided in lets say 2 developers. When the developers are done with writing their code both of them are checking each other code. After his check an testing both of the developers have to explain their work an advice their manager as a team. If 2 dutch developers check each other code an their is any criticism/comment from the other developer when valid it is accepted the code could be changed so the quality improves. Basically you learn from your mistakes or less experience. He had several times the situation that he worked with developers from India en Iraq. When he checked their code an the discussion with the manager was done he was more or less threatens not to speak any more that negative about them in an aggressive manner. It is not done in their culture to speak critical even with best intentions about somebody despit a crappy code the manager eventually had to intervene.

 

[captain paranoia]

IMHO, 'Agile' has been adopted too widely. It is most applicable to non-critical apps or web services. but much less so to critical system development.

There's also been a growth in the 'paradigm market', selling courses for methodologies. the original 'Agile' proponents have recently come out against the formal 'Agile' methodologies that have grown like topsy.

or
http://codemanship.co.uk/parlezuml/blog/?postid=1580

From the latter:

And the technical side of it is all but forgotten. I recently reviewed more than 100 CVs for an "Agile leadership" role a client was advertising. More than 70% of candidates had never written code. Most of the rest hadn't written code in more than a decade. Pioneers of early lightweight methods were always clear about this: decisions should be made by people with the necessary expertise and experience to make those decisions.

Meanwhile, armies of "Agile Coaches" - who are also mostly without technical experience - guide development teams in the prescribed process. Agile "maturity models" abound. It's big business now.

And for every 10 Agile coaches, you might find one developer guiding teams in the technical practices. Teams get very little help in things like unit testing, TDD, refactoring, CI/CD, architecture & design, etc. I think, from my own experiences working in this space, this is because most managers are from non-technical backgrounds and find things like Scrum, Kanban and SAFe accessible. They see the value in them. They don't understand code craft, and don't understand the value in it. So they don't invest anywhere near as much in it. Our people-process-technology tripod tries to walk with one leg much more developed than the other two, and tips over.

The upshot of all this is that we're back where we started. Oversized teams (and teams of teams), micromanaged in deep command-and-control heirarchies, measured by arcane heavyweight processes. All the emphasis is back on "Did we do it the right way?", and "Did we do the right thing?" is as immaterial as it was when the Unified Process ruled the waves.

Or one of my comments:

It was this. It’s a great example of why the Agile story model is entirely inappropriate for planning anything that requires a large, underlying infrastructure in order to facilitate the ‘stories’.

Story - something actionable and small enough to fit in a sprint. These are story pointed and defined using INVEST criteria. Stories should deliver a vertical slice of functionality to the customer that is valuable and complete by the end of an iteration. Stories are usually created throughout product development, more so leading up to iteration planning and also during higher level product roadmapping.

Great if you’re writing a web applet, or a small android app, using pre-existing infrastructure and tools, where you deal with self-contained ‘stories’ like this:

'As an employee I want to see the latest news from HR so that I am informed of what is going on.'

Not so great if the story is:

‘As a passenger, I want to be able to watch YouTube videos from my seat on the aeroplane, to alleviate the boredom’

Requires a huge underlying infrastructure.
Cannot be realised in a single sprint. Probably not even in multiple Epics.
Utterly useless for planning the development of an infrastructure-dependent programme.

Imagine if computers didn’t exist.
Imagine if the internet didn’t exist.
Imagine if Ethernet didn’t exist.
Imagine if http, html, CSS, javascript, etc, didn’t exist.
Now imagine that you had to develop them all before you could implement the ‘I want to see the latest news from HR’ story.
Now how do we plan the programme, based on the simplistic user stories?