National Security

Error Report II

GAO's defense of its F-35 investigation worries me even more.

It is important and proper that the Government Accountability Office (GAO) responded to my criticisms of its March 2013 report on the Defense Department's F-35 Joint Strike Fighter. Any watchdog agency should jealously defend its competence and independence, especially when the foundations of its oversight come under attack from those with experience, and sources, inside the agency.

The GAO took issue with my article, titled "Error Report," on three matters.

First, I am said to have ignored the "numerous quantitative indicators of development and production progress" that the GAO says substantiated its conclusion that "Overall, the F-35 Joint Strike Fighter program is now moving in the right direction after a long, expensive, and arduous learning process." And yet, much of this "progress" "in the right direction" remains behind even the revised -- and much relaxed -- program schedule, and it is occurring just as the F-35 is entering the more strenuous part of its flight testing. More to the point, the favorable conclusion flies in the face of expert reports, such as that of the Defense Department's own director of operational test and evaluation, who has reported a growing, not diminishing, number of problems.

The GAO's implication that the F-35's more difficult years are behind it, not ahead, is, in effect, a recommendation to proceed with the F-35's highly "concurrent" (buy it before you finish testing it) acquisition schedule. Having criticized concurrency in the past, the GAO would surely protest it is doing no such thing now, but that is the effect. If you doubt my judgment, read the assessment of Lockheed Martin's hired consultant praising the GAO report. (Lockheed-Martin produces the F-35.)

Second, I am said to have mischaracterized the GAO's evidence of the F-35 program making "considerable progress" in resolving the serious problems in the all-important helmet-mounted display of essential flight and threat information for the pilot. The GAO's substantiation of the "progress" is the argument that "DOD is pursuing a dual-design approach, essentially creating an alternative to the display's original design." In others words, the fix for the failure in one design is to try a second ("while working to fix the first"). The "considerable progress" is not in the functioning of the helmet but in the management decision to try an alternate approach. However, nothing in the GAO report gives credence to material progress in either helmet. All we have is the assurance that "program and contractor officials told us that they have increased confidence that the helmet will be fixed." As I said in my article, when I worked in the GAO division specializing in program evaluation and methodology, we used to laugh derisively at such data-free "officials told us" reports.

Nonetheless, we are assured by the GAO that "Pursuing an alternative is an appropriate way to reduce risk." Indeed, if that is truly the case, and the GAO believes its own prescription, where is the alternative design for the ever-problematic F-35 itself, and where is the GAO recommendation that such a management approach should be imposed on DOD? The F-35 program is riddled with remaining risks (even according to GAO), and an alternative design for new, better, cheaper fighters and ground-attack aircraft would vastly inform the debate and give decision-makers real choices.

Finally, the GAO stoutly defends its independence and competence, saying that formal discussions with agency officials about the facts in a report "before a draft report is sent for official comments" are necessary and appropriate "to ensure that the facts are not in dispute." I could not agree more; that's not what I alleged.

The practice in certain defense-related sections of the GAO today is to go beyond holding formal discussions to ensure the accuracy of facts. Instead, they send a written draft-document to agency officials well before the formal submission of an officially approved draft report for official agency comments. This is done for the purpose of acquiring comments from DOD officials not just on the accuracy of facts but also on the overall content. When those various comments from DOD officials are provided, they are received in the context of a GAO management climate that warns staff that "non-concurs" from (that is to say, disputes with) DOD are very much discouraged. In short, at this early stage of a draft GAO report, DOD has the opportunity to modify or even scrub out any unwanted findings, and it is done before the formal exchange of an official draft report and receipt of official agency comments that are publicly recorded in GAO reports. This process is all done in the absence of public disclosure.

When GAO managers impress on staff their desire to reduce to an absolute minimum those "non-concurs" with DOD, a "love letter" like that I describe in my article -- stating "The Department will continue to be supportive of the annual GAO review of the F-35 program" -- can be the result. This is not a reassuring relationship of an executive branch agency under investigation with its legislative branch overseer; it has all the hallmarks of the students interacting with the professors to influence the grade on their final exam.

In arguing against my central accusation that DOD exercised an unseen influence on the GAO's March 2013 F-35 report, the GAO sought to prove the independence and quality of its work by stating "GAO's quality assurance processes have been assessed on three separate occasions by teams of international auditors -- most recently in 2010 -- and in each case those processes were found to be effective and reliable."

But that's not exactly the whole story. Not only did those three outside reviews not include the specific GAO report in question, at least one of them did not give GAO a complete bill of health, insiders tell me. I am informed that one of the international peer reviews critiqued the GAO on the selection of evaluation criteria (i.e., issues such as the standards by which assertions like "considerable progress" should be measured). In response, the GAO has started an in-house training course to address the deficiency. But you would never know that by reading the GAO's retort to my article. (The GAO did not even name the three separate international auditors.)

I further wonder what the peer reviewers would have to say if they were allowed access -- including to all staff and their work papers -- to a GAO report that conveyed its draft content to the agency under investigation before the official agency comments were formally requested. I further wonder what peer reviewers would think of a management climate that admonished staff to eliminate areas of non-concurrence with the agency under investigation. (When I was at the GAO, they called it "having good relations with the agency.") Is this the relationship we want a watchdog agency to have with the people and programs under investigation?

All is not well in the GAO. The agency's retort to my criticisms is flawed; it seems to validate my concerns, and actually, it increases them.

Readers of the GAO's reports should be asking, is the GAO telling me all the facts I need to know, and is this report coming from an agency that disciplines itself to be strictly independent and at arm's length from those under inquiry -- not currying favor with them?

Wikipedia

National Security

Error Report

Is there a government conspiracy to save the F-35?

Until recently, the F-35 Joint Strike Fighter had been having a pretty rough time.

In 2012, its estimated average "program acquisition unit cost" was reported to have doubled, from the $81 million per copy anticipated in 2001 to $161 million, flight tests revealed deficiencies in achieving the F-35's modest performance requirements, and scheduled full-rate production was delayed to 2019.

In 2013, the pace of negative events and reports has only increased. So far, the F-35 has been grounded not once but twice (with different components showing signs of failure). Also, not one but two reports from the Defense Department's director of operational test and evaluation (DOT&E) documented serious problems: Existing deficiencies, such as the inability to land safely on aircraft carriers, have not been resolved, and new issues, such as lower than predicted acceleration, are cropping up -- just as the more challenging flight tests are beginning.

Even the F-35 program manager, Lieutenant General Christopher Bogdan, was quoted in Australia being volubly caustic toward the F-35's two largest contractors (Lockheed-Martin and Pratt & Whitney), saying they were trying to "squeeze every nickel" out of the program, not controlling their costs. Then this week, the Government Accountability Office (GAO) released this year's annual review of the program. It reported that the first four F-35 production contracts have overrun their targets by $1.2 billion; above that, the F-35s built before 2016 will need an additional $1.7 billion to fix the problems uncovered, so far, by testing. In addition, the amount needed to sustain the F-35 will climb to almost $14 billion in 2018, while the Pentagon is supposed to absorb cuts required by budget deals. As GAO also points out, the program faces a conundrum: Reducing production to save money will actually increase the unit cost of each F-35 built, thereby making the already unaffordable program that much more unsustainable.

And yet, it would seem that in other respects, the F-35 defenders are circling their wagons.

Lt. Gen. Bogdan is a more complicated personality in the F-35 saga than his highly quotable statements about greedy contractors might make him out to be. On inspection, he is an uncompromising defender of the F-35 -- he has trashed unwelcome reports on the F-35's ineffectiveness and, more importantly, tried to intimidate subordinates who have unflattering things to say about the plane.

Speaking at a conference of DOD bigwigs funded by Credit Suisse and McAleese & Associates, Bogdan dismissed one of the DOT&E reports as "premature," and then he went even further. One of the pilots quoted in the report had the strength of character and intellect (based on a few hundred hours of operational aircraft flight experience) to express concern that the design of the F-35 cockpit restricted pilots' ability to see threats to the rear, saying "aft visibility will get the pilot gunned [down] every time." The comment addressed a real flaw in the F-35's airframe design, and it came from an experienced operator. What was Bogdan's response? He told the press at the bankers' conference that "we can always put that pilot in a C-2 [cargo aircraft]...where [he] won't worry about getting gunned down."

Sure, general, that's just what DOD needs more of: Washington-based bureaucrats talking down to servicemen in the field with the character to express their experience-based concerns about defective equipment. Donald Rumsfeld did the same thing in 2004 when a reservist in Iraq had the gall to suggest that soldiers should not have to scrounge in scrap heaps to improvise "hillbilly armor" to survive in their vehicles in combat. Saying, "You go to war with the Army you have," Rumsfeld told the soldier to suck it up.

Lt. Gen. Bogdan, welcome to your own personal alcove in Donald Rumsfeld's hall of conceit.

However, beyond arrogance from "milicrats" at financiers' conferences in Washington, something else -- subtle, but perhaps more sinister -- has been going on.

For years, the GAO has been reporting on the F-35. While some of the reports have not probed as deeply or as completely as others, they have all been a valuable, informative resource. In some respects, the new March 11 report is no exception: F-35 costs continue to grow; capabilities are being achieved late or moved out of the program altogether; after 10 years, most capabilities are yet to be verified, or even tested; what some call the "death spiral" (smaller numbers of aircraft at increased cost) is a major threat to the program.

However, this new GAO report, titled "F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern," makes certain optimistic statements that it does not substantiate, suggesting the Pentagon may have influenced its findings.

Twice in the report, GAO asserts that the F-35 program has made "considerable progress" addressing the problems the F-35's high-tech helmet-mounted display. DOT&E reported that pilots in training frequently saw misaligned horizons, inoperative or flickering displays, and double, unfocused, jittery, washed-out and/or latent images in the helmet display, but the GAO report did not explain the problems, let alone any solutions. Then, GAO stated that "program and contractor officials told us that they have increased confidence that the helmet deficiencies will be fixed." In other words, contractors and DOD officials told GAO the helmet was on the mend, and without meaningful evaluation or evidence, GAO simply regurgitated the assurance in its report.

In another section labeled "F-35 Program Met Most of Its 2012 Key Management Objectives," GAO presented a matrix supporting inferred progress: Seven of ten objectives were declared met. GAO failed to note that several of the achieved objectives were almost impossible to fail. For example, GAO gave the F-35 credit for passing the objective titled "Begin Lab Testing" of the problematic helmet display and for receiving a go-ahead decision for the entire program from DOD's Defense Acquisition Board. These "accomplishments" are tantamount to students giving themselves a passing grade for starting a course, not completing it -- let alone with a passing grade. Also, the F-35 was found to have passed the objective to "Begin F-35A and F-35B Pilot Training," even though the DOT&E report raised serious questions about whether the F-35A was qualified to begin such training.

Interestingly, two of the objectives declared to have been failed were ones that are readily quantifiable: delivering the promised number of aircraft and achieving the Defense Contracting Management Agency's criteria for an audit. When pass or fail was quantifiable, there were problems.

Such vapid but positive findings gave one of the F-35's biggest boosters, Lockheed consultant Loren Thompson, license to author a paean to the GAO report, "GAO Gives F-35 Fighter Its Best Grades Ever," citing the above-mentioned and other supposed signs of progress.

Having worked in the GAO for nine years, I am familiar with some of the problems this GAO report contains.

In the lunch room in the GAO headquarters, we used to crack jokes about weak, "officials told us" reporting. It may be accurate that contractors and officials said the helmet would be fixed, but where is the empirical -- not rhetorical -- evidence of the "considerable progress"? What has actually been improved, what is the proof (from test data) of that, and why would anyone simply repeat the advocates' assurances as the primary evidence? How sad that such fluff suffices as evidence for managers in today's GAO.

While there are other examples in the report of poorly supported assertions, GAO's conclusion is the real stunner: "Overall, the F-35 Joint Strike Fighter program is now moving in the right direction after a long, expensive, and arduous learning process." Other than the weak examples cited, there is no substantiation of this sweeping statement. Indeed, reporting from others, such as DOD's own DOT&E, would seem to indicate the problems in the F-35 program may be growing, especially as it enters the more difficult part of its flight testing.

One has to wonder where GAO got the idea to print such a reassuring, but vapid, conclusion. I think I might know the answer.

Every GAO report should be, and is widely assumed to be, completely independent of any influence from outsiders, especially those who are the subject of investigation. The way I saw GAO operate during my tenure there as an assistant director and as former colleagues and other GAO contacts continue to tell me, that essential independence is sometimes compromised.

A basic element of the GAO report process is to send a final, manager-approved draft of a report to the agency being investigated for its official comments. These comments are routinely published in the report, and any time the agency under investigation finds what it considers to be errors of fact or judgment, it states them -- publically and on the record. GAO has the option, on its own re-examination, to modify the report accordingly or to stick to its guns, citing its reasons why.

However, for many years this important process has been abused by some in GAO. Repeatedly, I would observe others at GAO providing early, not final, draft reports to DOD officials asking for their informal -- not publically recorded -- comments, plus any useful data to substantiate their claims. DOD would even edit report language. The agency's "suggestions" would always come back accompanied by the implication that "if you say it this way, we won't raise any serious objections to your report."

For reasons that always seemed bizarre -- at best -- to me, many top GAO managers preferred findings, conclusions, and recommendations that gave DOD only limited, if any, heartburn. (I devoted a chapter in my first book, The Wastrels of Defense, to these and other serious problems in GAO research. However, to its great credit, the GAO division in which I worked, which specialized in program evaluation and methodology, refused to engage in this under-the-table agency-editing and report-correcting process.)

This problematic behavior has apparently taken deeper root in GAO -- all of it unseen by the readers of GAO reports. DOD is now routinely given a written "statement of fact" that conveys the substantive parts of a report at an early, incomplete stage before the research is closed out. The DOD responses, including edits and other suggestions, are taken as a regular part of the report-writing process. Moreover, staff are admonished that a "good" report is one that DOD does not obstreperously object to: As few "non-concurs" as possible -- if any -- are highly desirable in the final, written agency comments that appear in the end of a published report.

In short, DOD appears to have an unseen hand in influencing the text of GAO reports, and the management guidance as perceived by the GAO staff is to accept the DOD guidance to reduce as much as possible any areas of disagreement. The differences may only be subtle in a final GAO report, or they could appear in the form of strangely unsubstantiated assertions and conclusions -- the sort of vapid statements that appear in GAO's new F-35 report.

Indeed, the only other explanation for the data-free assertions might be that the work at GAO, especially the review process, was sloppy and did not merit what used to be the highly prized declaration (known as the "GAGAS statement") that all the work for the report was performed "in accordance with generally accepted government auditing standards," which -- justified or not -- appears on page two of the report.

My knowledge of this sort of practice led me to view as highly suspicious Appendix III in this latest GAO F-35 report: "Comments from the Department of Defense." It is a one-page letter stating, "The DOD values the GAO's analysis of the F-35 program. We agree with GAO's concluding observations and appreciate the recognition of the Department's responsiveness to previous recommendations. The Department will continue to be supportive of the annual GAO review of the F-35 program."

With a love letter like that from DOD to the agency that is supposed to be Congress's watchdog, it is time to worry how far self-interested interventions have penetrated a respected and heretofore sacrosanct agency. All may not be so well with the F-35 -- or with GAO.

Samuel King/DVIDS