Boeing: 787 fire caused by FOD, design changes coming, no word on delay

Boeing issued this press release at 3pm PST Nov. 24:

Boeing Initiates Changes to 787 Power Panel, Updates to Software

EVERETT, Wash., Nov. 24, 2010 – Boeing [NYSE: BA] is developing minor design changes to power distribution panels on the 787 and updates to the systems software that manages and protects power distribution on the airplane. These changes come as the result of what has been learned from the investigation of an onboard electrical fire on a test airplane, ZA002, earlier this month in Laredo, Texas.
“We have successfully simulated key aspects of the onboard event in our laboratory and are moving forward with developing design fixes,” said Scott Fancher, vice president and general manager of the 787 program. “Boeing is developing a plan to enable a return to 787 flight test activities and will present it to the U.S. Federal Aviation Administration (FAA) as soon as it is complete.”
Engineers have determined that the fault began as either a short circuit or an electrical arc in the P100 power distribution panel, most likely caused by the presence of foreign debris. The design changes will improve the protection within the panel.  Software changes also will be implemented to further improve fault protection.
The P100 panel is one of five major power distribution panels on the 787. It receives power from the left engine and distributes it to an array of systems.
The 787 team is now assessing the time required to complete the design changes and software updates that are being developed. A revised 787 program schedule is expected to be finalized in the next few weeks.
“Our team is focused on developing these changes and moving forward with the flight test program,” said Fancher. “The team in Laredo is also well along in preparing to return ZA002 to Seattle.”

37 Comments on “Boeing: 787 fire caused by FOD, design changes coming, no word on delay

  1. well so far- three causes have been promoted by various supposed knowledgeable sources – not in order

    1) an aluminum washer
    2) a tool left in panel
    3) some sort of FOD in the panel cooling system

    In any case, if my toyota and ford have caps over the 12 volt positive terminals to prevent inadvertent shorting, and ditto for marine voltage terminals, one has to wonder why such provisions may not have been part of the specifications . . .

    maybe it was on page 6 of the power point presentation to the cheapest bidder to the cheapest company to specify, design, develop, and fly . . .

    or perhaps lost in the translation between french and english ?

  2. Good news, it is not. It is less than terrible as it is some “foreign” source that cannot even be identified but is not a “domestic” cause.

    The result will be another delay and more rework for the growing fleet of planes parked around Boeing Field.

    Since there is no timeline yet proposed we are not even sure what lies ahead. We have only been given a Thanksgiving morsal that it is “minor” in nature.

    “Things like this happen during a testing period”. Boeing should have been cognizant of this when it proposed an “acelerated” testing period. It does not reinforce confidence nor assure that another exogenous event will follow.

    Seven delays have left many of us discouraged and terribly disappointed at this pattern.

  3. I’m skeptical. How to ‘simulate’ a FOD ?
    A FOD by its nature is NOT simulatable.
    Only known things are simulatable.
    Without hard evidence, one cannot safely conclude
    the cause to be a FOD.
    I guess, the mentioned ‘simulation’ went like this:
    “As our design is fine and checking back everything
    didn’t reveal any (hitherto unknown) flaw, the cause
    must have come from somewhere external to our system!”
    It’s a conclusion by exclusion, and that’s NO proof.

    • You design in a way that you can’t bridge contacts with objects that are
      actually large enough to bridge the free air distances.
      Forex you place isolating fins between contacts.
      Large DC voltages are unpleasant to work with because they sustain arcing
      and the voltage drop off from arcing may be nonobvious.
      In a large MagLev project I worked on the switching logic that attached the different stator elements to the drive inverter was augmented with contact position sensing and optolelectronic arc detection to avoid driving stator elements, that were supposed to be offline, via arcing over the switching contacts.

  4. Evin Ormond :
    I’m skeptical. How to ‘simulate’ a FOD ?
    A FOD by its nature is NOT simulatable.
    Only known things are simulatable.
    Without hard evidence, one cannot safely conclude
    the cause to be a FOD.
    I guess, the mentioned ‘simulation’ went like this:
    “As our design is fine and checking back everything
    didn’t reveal any (hitherto unknown) flaw, the cause
    must have come from somewhere external to our system!”
    It’s a conclusion by exclusion, and that’s NO proof.

    +++

    parsing words aside- some types of FOD are simulateable. For an extreme example, take the power panel enclosure, put it on a test stand, dump in small and large nuts, bolts, washers, wire segments, etc. Shake it, rotate it with power on. Properly shielded terminals and relays etc should not be shorted out.

  5. Perhaps I am reading too much into this, but I get the impression the FAA ordered the grounding of the 787 test fleet and perhaps even demanded a design change to the panel. Otherwise why would Boeing keep having to ask the FAA if they can fly their planes back to Seattle and if the new design is ok?
    As I said, I may be reading too much into this but I don’t remember Boeing mentioning the FAA so much before, not even at the beginning of the flight test program.

    • Boeing did so many little insinuations (FAA silent ~= true ) in the last years in respect to what the FAA does, thinks or wants for Dreamliner certification.
      My impression is they may have reached the end of their leash, the FAA putting a foot down.

      • Like most big corporations, Boeing is generally quite arrogant. Most times they treat the FAA as, frankly, a nuisance. I believe they now realize that the FAA could make a decision that will lead to big delays for the 787 – perhaps as much as a year. See my note below. So they are being a bit more deferential.

  6. The key statement, I think, is this: “Boeing is developing a plan to enable a return to 787 flight test activities and will present it to the U.S. Federal Aviation Administration (FAA) as soon as it is complete.”

    It points strongly to being a validation and certification problem. We understand the backup power failed to deploy as it should have done. Redundancy of power supply at all times is an absolute requirement. Boeing will have had to demonstrate the power fail-safe mechanism to the FAA via a series of tests before first flight. So it might not be plane didn’t operate as required, but rather plane didn’t operate as Boeing had validated it

    A question mark over any aspect of certification will lower confidence in the certification in general. Can Boeing convince the FAA that the failure in the backup systems was due to a well understood and genuinely extraordinary set of circumstances that the validation tests couldn’t be expected to cover?

    If not, they’ll be back to square one. Remember that Airbus had to get the entire engine control software rewritten for the A400M because the previous version hadn’t followed procedure, even though there was no concrete issue with it – unlike the 787 following this fire.

      • That is a strange read imho.

        Mr Tinseth talking about the “road” as
        an objective in itself?
        Expounding on longtime continuity ( which was destroyed while going from 777 to 787 ), the meek shall rule?

        He is preparing the public for 2 years of delays at least 😉

  7. Seems we have some skeptics here about Boeing, the FAA, and the cause of the P-100 panel fire. The fact is every airplane in the world is not “FOD-proof”, nor are the components and parts it shares with other airplane designs, or even newly designed parts and components. FOD has been a major aviation problem since Kitty Hawk and continues to be so today. When I worked at DFW Airport, it was not uncommon to find tools, safety wire spools, and other forms of FOD that fell from aircraft on runways and taxiways. It was the same during my USAF career flying the KC-135. FOD is not just an engine or tire problem. It can effect any aircraft system, including the electrical, avionic, and hydraulic systems.

    The goal has always been to reduce incidents of FOD to the lowest level possible. The industry would like that level to be “zero”, as with other aviation problems. But almost every aspect of aviation is subjected to creating FOD, from manufacturing, to daily dispatch, to weather conditions, to bugs and animals, etc. However most incidents are caused by human interaction.

    In this case, it does not matter what caused the FOD to be left in the P-100 panel, it does matter it was there and not found before hand. At least not at the moment. It does matters how long the FOD was there. Was it there since the P-100 panel was manufactured and dislodged later due to normal flight activity (vibrations, etc.). Was there some maintenance done on the panel since it was built? Was it an istallation problem that caused the FOD? Also was the tool, or washer, screw, nut, etc. that was “missing” or loose known at some point before the fire? If so, was it reported and someone try to find it? Was it documented? Or was it not noticed at all?

    The FAA (and EASA) has been involved in the B-787 program from the beginning. The FAA needed to approve every flight before hand, just as they need to approve the return to flight testing now. It is up to Boeing to prove to the FAA the B-787 is safe enough to return to flying. It looks to me that is what Boeing is trying to do.

    • You showcase why Germans use compound words.
      They are more difficult to twist around to a different meaning.

      The issue is neither the type of FOD nor Fod no noFOD.

      The issue is FOD and its admissability in an electrics enclosure.

      If you cannot avoid something you must be prepared for the consequences.

      In a high reliability environment that includes a planned staged takedown
      and switchover to backup systems.

      This seems to not have happened as expected. And it looks like no amount of
      sweet talking will reopen the FAA clam in a timely fashion.

  8. That it will be a few weeks before Boeing will offer the solution suggests a significant further delay.
    The points made above by FF make a lot sense.

  9. Absent cold hard proof, Boeing has now speculated on FOD. It’s managed to scapegoat a theory.

    Think about the magic of that. The wordsmithing. The art and craft of deflection. Boeing exhibits such profound mastery at it that it simply takes one breath away.

    The vomiting begins when the press, especially the aviation press, chugs down this ipicac of breezy excuse making.

    • “The wordsmithing. The art and craft of deflection.”

      Right, my absolute respect for Boeing in that direction.
      And it is a cultural thing too me seemeth, the BushII
      administration excelled on a similar turf.

      Unfortunately the mindset to achieve this is absolutely
      incompatible with the reasoning processes that are the
      foundation for a good engineer or scientist.

  10. Unless the whole contacter or breaker totally melted, there will usually be signs of the arcing where it occurred, judgment can be made and a similar sized washer be placed or hung at the spot, then the equipment energized. If an arc occurred with similar damage it can be pretty certain what happened, BA indicated they did replicate it.

    What is bad is where the systems did not respond as planned to this event. That will take a lot of sorting out.

    • My interpretation from a glimpse of the images is that
      one arc burned from a contact in front of the P100 carrier
      to the (metal) casing of another component
      _enveloping and burning of the insulation_ from ?control?
      wiring to that part.
      Another one must have been raging between carrierpane and
      the rear wall of the enclosure removing more than a sqft
      from that rear wall exposing the insulation matt on the
      inner hull.

  11. onemancrew, what makes you think Boeing has no hard proof of FOD within, or around the P-100 panel. If you think this is only a Boeing theory (developed for PR reasons to show Boeing is not at fault), please prove it. Otherwise your assumption is worthless.

    What is your theory on the cascading electrical failures? Boeing has already said they are developing/designing software and hardware changes because of these.

    BTW, what is your theory of the 54 systems failures on the A-380 and the uncontained failure of the RR Trent-900?

    • Have a look at the picture labeled “rearside of panel”

      most of the energy must have been spent behind the plate
      that holds most installed parts. having no requirements
      for much structural strength the rear of the enclosure
      “passed away” first.
      Would be nice to have a “before” picture from the
      same viewing positions.

      Quality of 54 messages pertaining to defects.

      A great thing. In earlier times
      the first to have a working picture of what failed
      were the crash investigators wading through the
      debris in the ground.
      Here the crew had adequate time to access the information
      and decide, while the autopilot still managed the basic airplane,
      rather unhurried on how to proceed, incl “a brake to meet”
      firefighters in the landing returning all passengers unhurt to
      the ground. A rather remarkable technological achievement.

  12. Why do I think Boeing has no proof of FOD?

    Because they haven’t got any, that’s why.
    No, the theory wasn’t developed just for PR, ( I give them the benefit of the doubt on that) but they do use it as the “probable cause” speaking of this theory as if it were a definitive answer.

    And just how accurate “probable” is is debatable. Is it a 51% likelihood or 91%?

    That might matter to an engineer, but not Boeing corperate, who just shrug and move on.

    • For the resulting repercussions on Dreamliner devel path
      the actual initial trigger is moot ( even if we could with
      some certainty say that it was sabotage or some other evil
      source : The devil ? quite popular currently ;-).

      The relevant part is the exposure of the fact that
      the electrical system as currently installed does
      _not_ meet basic certfication requirements.

      • Well, it seems to me that FOD is the most likely culprit here, based on Boeing’s explaination. I base this on my own flying experience and my 40 years in aviation, including accident investigations.

        So, here is my theory;

        1. ZA-002 was assembled with this unknown FOD already inside the P-100 panel.

        2. ZA-002 had flown for more than 500 hours in flight testing before the fire on nearly 100 flights.

        3. The culmimitive effect of turbulance, CAT, landing gear cycling, changes in engine thrust changes, spoiler cycling, braking, and aircraft vibrations dislodged the FOD on the test flight to Texas, dropping it into a position where a short circuit and arcing occured.

        4. The FOD was, most likely, made from steel or aluminum.

        5. The FOD created arcing as hot enough and long enough to start a very hot fire.

        6. The fire also consumed some of the steel backing plate of the P-100 panel, as well as steel and alunimum covers of some other electrical components.

        7. The fire ran out of “fuel” very quickly, thus “self extingushed”.

        8. The burned materials were “rich” enough in the fuel it consumed to produce a lot of black smoke and soot. As evident in the pictures. Both melting steel and melting aluminum produce this type of black soot.

        9. The fire consumed the FOD.

        10. The fire was, partially or totally, contained by the insulation blanket.

        I have not seem the actual damage, only the pictures I posted at 6:22 am today. Nor, am I in any way connected to this investigation or connected to Boeing or the FAA.

  13. KC135TopBoom :
    Well, it seems to me that FOD is the most likely culprit here, based on
    …. lots of known knowns snipped ….
    Boeing or the FAA.

    Now Topboom, what grave news did you want to tell us about?

    humpty dumpty is broken and all Boeings media men can’t fix it 😉

    • Actually only the factual observation is correct, but there is a
      missconception in your post TopBoom re the interpretation of those facts.

      There was no “fire” in the sense that (much) energy was released
      from combustion of combustible materials.

      Burning aluminium per se will not leave soot ( no carbon in Al )
      But excess consumption of oxygen would burn everything else in
      a low oxygen environment : result coke and soot ( partial combustion )
      You get about similar results from reaching decomposition/burning temps
      in a no air exchange environment.

      My accessment is that the P100 panel compartment was used
      as a 50++kW arc furnace for 30 seconds.
      It was unfortunately not designed for this use case 😉

  14. RE top boom- great analysis given the info available. The first thing I noticed on the photos sort of shocked me- but perhaps I misread what appears to be uncovered/bare terminals at the same height or above the partitions between the three terminals on the adjoining relay/contactor. (lower right in first photo with blue tinted base ) seems to me that is a short waiting to happen with any FOD larger than about 1/2 to 3/4 inch in length or diameter. Seems to me some sort of vented snap on cover – fire and heat resistant would be in order – and be required ! Why put an isolation ‘ fin” between the three terminals if one can easily bridge the terminals/nuts/posts above ??

    MY guess is that an aluminum or copper ” washer ” would likely have been vaporized faster than a steel ” washer ” and a little less likely to have started the resultant fire .

  15. Uwe :
    Actually only the factual observation is correct, but there is a
    missconception in your post TopBoom re the interpretation of those facts.
    There was no “fire” in the sense that (much) energy was released
    from combustion of combustible materials.
    Burning aluminium per se will not leave soot ( no carbon in Al )

    +++++

    GROOOANNNNN –

    1) True that pure aluminum does not in itself produce soot when burning

    2) NOT true that pure aluminum would likely be found in or around any structure or in washers, bolts, wire, etc. Aerospace/Aircraft aluminum is almost never pure aluminum, but actually a alloy of various compositions typically similar to that of 7075 (7075 aluminum alloy’s composition includes 5.1-6.1% zinc, 2.1-2.9% magnesium, 1.2-2.0% copper, and less than half a percent of silicon, iron, manganese, titanium, chromium, and other metals. It is commonly produced in several heat temper grades, 7075-O, 7075-T6, 7075-T651.) Pure aluminim is to ‘ soft’ for almost anything like a washer or bolt. Closest to pure aluminum would be the aluminum foil used in the kitchen- not as part of an aircraft.

    3) true that most of soot probably came from burning insulation, ‘ plastic ‘ shells, etc.

    4) Aluminum conducts heat much better than steel, and melts at a lower temperature, so it remains to see if it would sustain an arc long enough to overheat surrounding area.

    A redesign suggestion would be for Boeing/Suppliers to get and use other than Radio shack relays- cheaper is not always better 😛

    • (Don, did you move directly from detail draftsman to union politics?)

      * see any Carbon mentioned in your alloy listings?
      The black cauliflowery stuff you see is incompletely combusted plastic
      (phenolic, epoxy, silicone or polyester based and reinforce or not).

      * there is a posting (further up/other thread?) on how much energy you
      need to melt Al ( from me 😉

      * Radio Shack or not. having high DC voltages sources with a major bit of
      source “ompff” is a difficult to mitigate risk.
      Isolating loaded HV DC busses is a PITA, believe me!

  16. looking at the arrangement of power terminals and ‘ control’ wiring, cant help but wonder what the effects of a major magnetic solar storm might do to not only airplanes in flight, but cars, communications, computer systems, etc.

    Military aircraft are generally well protected for the magnetic effects of high alt nuke explosions nearby ( within say 100 to 300 miles ), but these effects are orders of magnitude smaller than a good solar flare – magneic storm

    http://www.telegraph.co.uk/science/space/7819201/Nasa-warns-solar-flares-from-huge-space-storm-will-cause-devastation.html

    back to diesel engines . . in cars, etc

    • Narry a modern diesel without electronics, injectors/pumpjets for state of the art specimen are piezo driven.

      The potential for damage comes via induction into physically long
      above earth conductors ( like the HVAC Grid, antenna, pipelines, … )

      You won’t get the sharp field gradients that characterize an EMP.

      My guess is crafty people will try to leverage this into another
      hoarding spree like the Year2000buG. ( Got a very cheap generator
      out of this in late 2000 😉

  17. Rather glum article in the Chicago Tribune.
    http://articles.chicagotribune.com/2010-12-04/business/ct-biz-1205-787-delay-20101204_1_dreamliner-teal-group-richard-aboulafia

    Even without the fire, Boeing almost certainly would have postponed …

    .. nagging problems like “rain in the plane,” condensation that dripped and pooled on some flights.

    Software failures caused most of the 787’s flight-deck displays to go dark …

    …………

    One can’t be more damning in such an objective report.

  18. From All Things 787

    If true this would be a four and a half month delay. The newspaper reports that fixes should be done by the end of December with test flights resuming in January. This all, of course, is dependent on FAA approval of the required fixes. Click here for the Les Echos article. The translation is below (by Google Translate):

    “The fire was caused by the presence of particles in the closet – and not a forgotten tool as has been mentioned. The short circuit would result in a blackout then the electricity network, despite the security software. Boeing and its partners to the electrical system, Hamilton Sunstrand and Zodiac, have therefore set about rewriting the computer program key.”

    Hmmmmm

Leave a Reply

Your email address will not be published. Required fields are marked *