Conduct Learning-Based Reviews (part 2 of 2)

This is the second of a two-part story. (Read part 1 here.)

Changing one’s mindset to consider reviews as a vital learning opportunity is not easy. While the reviewing organization often refuses to abandon the “review as control” perspective, the subject of the review may be equally resistant to changing its approach, due to overconfidence or skepticism. In the following two examples, winning or losing a project was largely determined by the contractors’ willingness (or lack thereof) to learn from project reviews.

The first example comes from Jenny Baer-Riedhart, a NASA program manager. In 1994, NASA initiated the Environmental Research Aircraft and Sensor Technology (ERAST) program, which focused on converting high-altitude, long-endurance unmanned aerial vehicles (UAVs) into research platforms. Because of the difficulty in controlling the risks involved, UAV industry development lagged behind the interest in and knowledge of how to improve the technology.

To mitigate the risks and stimulate the industry, NASA adopted a radically different approach and formed a joint sponsored research program with four of the main players in the industry. For the ERAST reviews, NASA brought in people with experience in a particular area of aircraft development and testing. The companies not open to NASA’s advice did not fare well. Although one of these companies was superb on paper, with genius employees, it crashed its UAV twice. Reflecting on the case, Jenny concluded that had the company been open to NASA’s advice during reviews, it might have prevented the crashes.¹

Similarly, Terry Little the U.S. Air Force’s project manager of Joint Air-to-Surface Standoff Missile (JASSM) concluded that learning-based reviews played a major role in distinguishing between the two finalists. Although the losing company had good engineers and disciplined processes, it failed to listen to feedback, which led to its downfall.²

Learning-based reviews are also highly regarded in industries that engage in more traditional projects, such as the design and construction of manufacturing facilities. One approach that naturally facilitates a learning rather than a control focus in the review is to establish review panels composed of peers, rather than senior managers or experts, who are expected to report to senior managers following the review. In praise of the peer review practice employed at Proctor and Gamble, Scott Cameron, Global Project Management Technology Process Owner at P&G, asserts: “The most successful method we have found to improve project performance is to conduct anywhere between 1-5 peer reviews throughout the life of a project.” ³

Peer Review Practice

Purpose of the Peer Review

To gain the most valuable input in the shortest amount of time to improve the chances for a successful project and avoid disasters.

Whom to Invite

Just peers, no hierarchy. A diverse group of ten to twenty people consisting of technical engineers, project managers, construction managers, purchasing managers, finance managers, research and development personnel, and contractors. 

What Protocol to Use

The project team and the project manager concisely communicate their technical and execution strategies. They then invite peers to comment, critique, and ask clarifying questions. Pre-work can be sent out to the peers to review prior to the meeting. Peers should be open, honest, and engaged, or should not bother to attend.

How Long Should It Be?

A maximum of 6-8 hours, including lunch and breaks.

How to Summarize the Discussion

Take copious notes and display them on the wall. In the last peer review I attended, there must have been 30-40 pages of flip-chart paper capturing the ideas on a $50MM project. These were then typed and distributed to all the participants with a note thanking them for helping improve the success of the project.

What to Expect of a Peer Review

Out of the notes, there were only 5-10 “nuggets” that the project team used to improve the project. Implementing these nuggets more than made up for the cost of the peer review. As we have conducted more peer reviews, we’ve noted that the invited peers are taking “nuggets” they had not considered back to their projects and programs.⁴

1.    Laufer, A., Hoffman  E., and Cohen, D. (2012). “Flying Solar-Powered Airplanes: Soaring High on Spirit and Systems,” Chapter 3 in Laufer, A., Mastering the Leadership Role in Project Management: Practices that Deliver Remarkable Results. NJ: FT Press.

2.    Laufer, A., Ward, D., and Cockburn, A. (2012). “Developing a Missile: The Power of Autonomy and Learning,”  Chapter 1 in Laufer, A., Mastering the Leadership Role in Project Management: Practices that Deliver Remarkable Results. NJ: FT Press.

3.    The Hour Glass and the Project Manager,” W. Scott Cameron, Procter & Gamble. 2001. Ask Magazine 4 (July): 27-8.  http://appel.nasa.gov/ask/about/overview/index.html

4.    ibid.