06 November 2015, ©. Leeham Co: COMAC’s C919 was rolled out in the week. We got to see a new shiny aircraft which looked ready to fly. The nicely curved fuselage and wings were immaculate, the paint was shiny and the CFM LEAP-1C engines were ready to go.
Yet many ask, when will it fly for the first time? It used to be that when the airframe was finished and the engines ran reliably it was time to fly. No longer! Today the most challenging part of an aircraft program is the integration of all the complex systems which hide under the skin. This is what kept the Bombardier CSeries on ground longer than it should and the Boeing 787 and Airbus A380 had the same flu (the latter also had to short wires).
It is the part of the aircraft which takes longest to get to work reliably. The A380 is known for its long period of nuisance warnings from the complex avionics system after entry into service and the reliability work for the 787 has to a large extent been one of software tuning of its system side.
As the system function of modern aircraft has grown more complex the whole architecture of how it was built had to be changed. Here’s how.
The classical aircraft system architecture
The classical system architecture of an aircraft was that each important function like flight controls, engine control, air-conditioning, radios, navigation equipment, etc., had its own computers sitting in dedicated boxes. These boxes communicated with other boxes via point-to-point slow speed (0.1Mbit/s ARINC429) network links. A typical page in a system description for pilots of a 1980’s aircraft like the Airbus A320 looks like:
In the picture, the cabin pressure in the aircraft is controlled by eight boxes talking to each other. The aircraft’s altitude had to be given from the ADIRS (Air Data Inertial Reference Unit) boxes to the air conditioning controllers so they could regulate the cabin outflow valve to get the correct cabin pressure. As the Flight Management Guidance System (FMGS) computers should be able to control the pressure, they needed a link as well as did the cockpit control panel. In all, eight boxes have eight point-to-point links (and therefore wires) to control one outflow valve.
New architecture needed
As the functionality increased, the number of boxes and their point-to-point networking needs exploded. It got complex and heavy. It was time for a smarter concept, the Integrated Modular Avionics (IMA) architecture. This can simplest be described as an airborne adaptation of a modern industrial real time computer network.
Instead of discrete boxes per function and point-to-point networking, many general purpose computing units (part of an IMA system) are connected by a double 100Mbit Ethernet network (called AFDX or ARINC 664part7). Functions like the cabin’s air conditioning system and its control of the outflow valves were now an application running in this network of standardized computers and talking over Ethernet with the FMS, ADIRS and cockpit display/control applications. Its commands to the outflow valves were over standardized input/output modules which all applications could use.
It is obvious that such an architecture is more efficient and flexible than the classical one. The change started in smaller scale on the Boeing 777, then went full circle for the A380, which based the whole avionics system on IMA. Boeing’s 787, Airbus’ A350 and Bombardier’s CSeries then followed and now COMAC’s C919 is the latest aircraft to implement IMA around an AFDX network.
This means that system suppliers are now delivering software applications that run on standardized hardware and talk through standardized interfaces rather than each system supplier developing his computers with software + interfaces to the rest of the world.
The are many advantages of an IMA architecture. The number of unique boxes with their hardware and software that needs to be kept current and kept as spares by the operators goes down. Now they keep stock of a few IMA modules and ADFX network controllers.
Instead of updating boxes to fix problems, software can be updated. In the case of the 787, this is through a special maintenance laptop which connects to the aircraft via Wi-Fi. Should an IMA module stop working, the system shuffles the applications automatically to functioning modules, therefore the aircraft’s avionics system can still function despite hardware failures.
The IMA system can also partition the network into different virtual computer networks so that non system-critical functions can’t affect critical functions like Autopilot or Navigation. For very critical functions like Fly-By-Wire core functions and engine control, these are still handled by dedicated computers but these communicate with the IMA system via the Ethernet.
Today’s airliners integrate complex system functions in a number of areas. We have described the aircraft’s core system functions. We didn’t need to start there.
We could have started with the In Flight Entertainment (IFE) system. This is today a separate network of hundreds of computers where each seat has its own Android computer with local hard disk so the movies and other applications shall be quickly controlled and loaded from the servers that are the heart of the network.
On top of the IFE system comes more and more often the Wi-Fi-based In Flight Internet Connectivity system. Add to that, the cabin crew’s cabin control system and the pilots’ electronics flight bag (EFB) system.
Note that we are not even at the core aircraft systems yet and we have probably described something like 200 to 700 computers (dependent on aircraft) that need to talk to each other over different networks.
No wonder the system side of a modern airliner is the real nut to crack before the aircraft can be certified and delivered.
The major problem with software defined systems is that “you can fix it later” ( supposedly at low cost : just an update, easy )
This takes away pressure from getting it right in the first place.
If there is room there is movement possible. This slack will invariably be taken hostage by management.
(un)Surprisingly late fixes to software are on par costwise to any kind of hardware fixing.
And now the broken record part:
Fred Brooks : “The mythical man month” 😉
Exactly. A recent example would be Windows 10, released in late July when it was not finalized. Since then, Microsoft has been releasing updates that not only fix bugs, but also change the interface, add functions, and modify how certain actions are done.
Don’t spend money yet on a comprehensive Windows 10 reference manual, as the beast will prove to be a moving target for a long while, thus obsoleting the manual as time goes on.
Just look at the time it takes to upload new software on the 787. Some new Aircrafts did not go full ARINC 664part7 but intermix with old ARINC 1553 boxes, Boeing has its own ARINC 664part7 version with its additions. Some interesting side effects is that the central maintenance computer (FIM) sometimes forces you to work thru the troubleshooting step by step and not just replace the broken box as system errors will remain and not clear properely.
A good summary of the change from federated avionics systems to IMA.
A nit-picking point however: Airbus IMA does not do dynamic reconfiguration, in the sense that software that was working on a now-failed IMA module being re-loaded onto a functioning/spare module. There are multiple processing channels operating simultaneously throughout the flight, and if the one in active command should fail then that command transfers automatically to an alternative channel.
Thanks Dan, didn’t know that. It seems every major vendor of the IMA infrastructure have their own variant when it comes to the details, doesn’t detract from the fact that it’s a major advancement though.
“dynamic reconfiguration, in the sense that software that was working on a now-failed IMA module being re-loaded onto a functioning/spare module. ”
Any idea how they certfied that?
I’ve only had my hands in safety relevant rail ( actually maglev ) stuff. I’m aware that rail ( low energy state is safest ) is not comparable to flight ( least energy loss is safest so to speak).
Most difficult aspect was afair to prove that you actually had a fully introspect able _finite_ ( and thus fully analyzable ) state machine at hand.
I absolutely see the attractiveness of autohealing.
But it is a proving nightmare.
Your point about the state machine is correct. The biggest challenge with any kind of ‘dynamically reconfigurable’ system is determinism – being able to prove that you will always know what state the system will be in, regardless of the conditions.
Airbus IMA configuration is determined at start-up, and does not change afterwards. This is non-optimal for the overall aircraft design, because software such as that for landing gear functions continues to operate during all flight phases when the computing hardware could be used for other purposes, but means that you don’t have to worry about if the avionics will reconfigure themselves correctly as the flight phase changes or when a fault occurs.
(There are other reasons too for this static configuration, such as systems being reliant on very specific IO types which may not be replicated across all IMA modules).
Yes, the development redundant fail safe systems is extremely complex. It’s a special environment, no opportunity to press CRT/ALT/DEL and reboot. Unfortunately the tools and methods for developing such systems are only being developed and maturing along side the actual task. Software development always seems to fall behind schedule for that fact. A disconcerting fact: software systems are never 100% error free. Flaws are revealed as a function of time in use in as exponentially decreasing function.
What is the process by which software changes are validated? Is there a standardized methodology? I wonder because I have difficulty understanding how something like the a400m engine software issue that lead to a crash could happen.
that apparently was some kind of misconfiguration/mismatch and not a software bug.
IMHO this should have “blown up” on power on and not during use.
One way is to check for a version “magic”.
This is the first I heard of the A380 fault problems. All other published information (once they got the wire gap closed) indicated it was a flawless entry (at least those I had access to)
So the 787 comes along and it gets beat to death for the same issues (fault limits set so tight that what was either a warning or a non existent issue stepped a flight or caused them to land)
Very interesting contrast.
Bad reporting not to differentiate between a real problem for the 787 like the battery and the others (initially you are better off with too tight than too loose)
In my world (building computer controls) I work to find what I call Rubber Meets the Road alarms, i.e. ones that really do mean you have a problem and are not just nuisances ). I was asked to alarm a critical building and what ultimately turned out to be the easiest was to have a stand alone alarm for one room, if that went off you knew a system had really failed, not a switch or a transitory one.
Frankly if something simple like that is so hard then it must be an insane program world for aircraft.
entry into service for the A380 was not “flawless”.
but it was well managed. i.e. the process was closely watched by Airbus personnel and all issues seem to have been fixed asap.
afair there were some real hardware issues. but not from core functionality. One major issue were nuisance warnings.
Overall EIS appears to have been a rather fair process.
This seems to have been buoyed by the A380 mostly performing beyond sales promises.
Compare to the 787 process were the hurtful screams only stopped after Boeing had set up a watertight news embargo.
My take is that the ones that took the A380 wanted no bad publicity and were willing to cover it all up.
Boeing got reamed, not that they did not in part set themselves up for it but the headlines were screaming all the time on simple software alarm issues.
Lets not rewrite history. The A380 is and always was the target of critics. Too big, too expensive, too heavy, too crowdy, too much engines, passengers don’t mind, too.. not from here.
The 787 the opposite, smart, advanced, beautifull, lean, comfortable, unprecendented PR for Dreamliner.
Things worked out slightly different. The A380 was a euro disaster, heavy, nearly two years delayed. Then came the Dreamliner..
Integrating and Certification is going to ge interresting on the C919. And bringing the bad news equally interesting. The Chinese hyrarchy works totally different.
Maybe over sensitive on both sides?
Other than some engine problems and wing skin separation technically nothing wrong with the A380. My question always was and continues to be, is it viable? So far evidence says not.
Like the Japanese cars being in the right place at the right time with the right product that can change in a short time. I don’t see it but then I haven’t seen a lot of things coming (that light in the tunnel was really bad)
As far as the C919 goes, what people don’t get is its a project that is all government sponsored, lots of money thrown at it but its also a hung over top heavy organization not a private endeavor. Getting things done is extremely cumbersome in that sort of thing.
What always amazed me about Airbus was that it functioned at all let alone bringing out competitive products. I don’t see that in China, the Airbus members were all accomplished in their area as well as democracies with freedom of thought.
“What always amazed me about Airbus was that it functioned at all let alone bringing out competitive products”
? Europe was inventing & producing a variety of big jets for 30 years when Airbus emerged as a concentration of existing entities. In the late sixties / early seventies the guys pushing boundaries in the late thirties / forties / fifties were still doing office hours.
Alas, as happens far too often, we’ve ended up once more on this road of partisan battles.
Without going into details, Airbus has certainly had it’s problems with maturity following A380 EIS, both those well reported in the external press and otherwise. What has been the bigger problem for the A380 however has been the lack of sales, and hence this is what is reported more heavily in the press. The 787 certainly doesn’t have this issue, and so the technical problems (which are multiplied due to the much greater fleet size) take precedence in the reporting.
What really matters for both Airbus and Boeing is how quickly they provide fixes for these problems and so restore credibility for future programmes – more and more so, the airlines are backing away from being new aircraft launch customers (unless they receive *extremely* deep discounting) due to the problems that they have seen happen on older programmes.
I have read somewhere that the 787 uses the Wind River System operating system. This is a commercial real-time-operating system that had its roots in the embedded control system industry. I wonder if there is any feedback on the performance of this system in the 787.
It’s highly unlikely that the operating system has any influence on the overall reliability of the systems. WindRiver’s VxWork, and its spin on Linux, and other RTOSes such as INTEGRITY, QNX, etc. basically just work. At their core these OS are highly reliable, with varying amounts of paperwork to prove it. Board support packages are another matter, though bugs in those tend to get ironed out early on in a project.
No, the problems always come in the application software that Boeing or Airbus or whoever write and run on top of the OS.
The Wikipedia article on INTEGRITY-178B says it’s used on the A380.
Having successfully developed embedded systems using VxWorks, Linux and INTEGRITY I think I’m reasonably qualified to say that they’re all good from a technical point of view, though the real key to their success is properly good vendor support (WindRiver also ship and support their own take on Linux; I don’t mean Ubuntu). Their histories differ. INTEGRITY was written from the ground up for this kind of application, the others have been massaged into the role.
Thanks for the info. Embedded systems software have really come a long way since the days of writing code in Assembly, and later, in C.
The whole point of having an OS is that it provides a standardised set of services such as process scheduling, device drivers, libraries, etc. That saves having to develop them specifically for a project. The US DoD mandates use of the POSIX API for their software projects, and all major operating systems (apart from Windows) do indeed provide those services through the POSIX API.
This means that you can still write your system software in C, or even assembler (though that would be a highly unusual choice these days), but there’s a lot less of it because of what the OS already provides.
It’s actually more usual to write critical avionics software in Ada, rather than C or C++. Ada is the language commonly perceived as being necessary for a safety critical system. POSIX is still in play; the Ada runtime, which is probably written in C or C++, will use it to create the Ada runtime environment. The same goes for Java runtime environments, etc.
Although Ada is the normal choice for safety critical software, it is by no means essential. Ada is rarely taught in universities and there is a dearth of Ada programmers. Planning to use it in a new piece of avionics first means getting a load of programmers from somewhere if they are not already on the company staff. However there just aren’t that many around and they’ve all got jobs already.
This has lead to some projects that were traditionally written in Ada (but weren’t necesarily quite so safety critical – e.g. combat systems on ships) migrating to using Windows and C++.
Even companies like Facebook and Twitter, who started off using PHP and Ruby, ended up severely regretting it. The really big companies (Google, Amazon, Ebay) all did the core of their companies’ servers in C++ from the very beginning. In that industry there’s the phrase “getting in the grey beards” to get things put together properly…
So to get a complex embedded systems development off the ground these days a company practically has to train all the programmers from scratch, and quell a lot of grumblings about having to use this “ancient old technology”… The alternative, which is getting in expensive contractors, is more or less unavoidable, but it can lead to significant code maintenance problems.
For the record, Airbus IMA modules use a bespoke RTOS written by Thales.
Leadership is key, always has been.
A friend’s first software job was at one of the “Digital” named companies. He tells that the project he was on finished on time and budget whereas another project at the time definitely did not. It was a good education for a new software person, watching leadership and what worked.