They operated from a clandestine apartment in Harlem, a block from Columbia University at 401 West 118th Street. A cell comprising 18 of the most respected American mathematicians and statisticians spirited datasets up the stairs, analyzed them, and stole to Washington DC on military aircraft to present the results of their rumination to the admirals of the Navy and the Marines, the generals of the Army, Marines, and Air Force during the Second World War. Allen Wallis, Director of SRG said of his team, “This was surely the most extraordinary group of statisticians ever organized.”

And their impact is hard to overstate.

During the Battle of the Bulge in December 1944, several high-ranking army officers flew to Washington from the battle, spent the day discussing the best settings on proximity fuses for air bursts of artillery shells against ground troops and flew back to the battle so they could put into affected vice from, among others, Milton Friedman whose earlier studies of the fuses had given him extensive and accurate knowledge of the way the fuses actually performed.

Milton Friedman’s memoirs, Two Lucky People

Many of these men relocated their families abruptly to Manhattan, called to serve their country. Chief among them, Abraham Wald, a Rumanian prodigy whose intellectual parents had home-schooled him. A Jew, Wald respected the sabbath on Saturday and wasn’t permitted attend Rumanian school on Saturdays. His parents homeschooled him instead and Abraham graduated with a doctorate in mathematics from the University of Vienna in 1931. In 1938, he fled Nazism in Austria for New York. He began to teach statistics at Columbia University.

Soon thereafter, the SRG recruited Wald to join them. The US Air Force needed desperately to solve a key problem. Allied planes in the European and Pacific theaters were being shot down at astonishing rate. During the course of the war, more than 43,581 planes would be lost to flak cannons, the Luftwaffe and Japanese Zeros. In a single 376 plane raid in August 1943, 60 B-17s were shot down. The loss rates were so great that it was statistically impossible for a serviceman to survive a 25 mission tour of duty in Europe.

With airmen facing impossible odds, the SRG tasked Wald to design and develop a new armor layout for aircraft flying in enemy territory. The Air Force wanted to reinforce planes, but could only add armor to a small section of the plane. Each pound of armor reduced the bomb payload capacity by one pound, diminishing the attack capability of the fleet.

To inform the analysis, the Air Force supplied Wald with data from the front. Servicemen recorded the number and placement of bullet holes on the fuselage, wings, and tails of all the planes that returned to base. Returning bombers bore scars concentrated around the tail gunner and the wings. So, the leaders of the Air Force suggested reinforcing these bullet-riddled regions with armor.

Wald vehemently disagreed.

The returning planes’ pockmarks showed where the birds were strongest, the places they could suffer injury yet still return to base. The fuselage and wings least needed the armor to survive. These planes suffered injuries in these regions and return to base regardless. Wald argued this proved these areas weren’t critical.

There were tens of thousands of planes that hadn’t returned home because they’d been struck elsewhere: the engines, the tail, and cockpit. Those areas should be armored more heavily.

The oversight by the U.S. Air Force, dubbed survivorship bias, is a cognitive bias that plagues data analysts in all fields.

This is an excerpt from a book I co-authored called Winning with Data, and it’s one of the most memorable stories I came across in the use of statistics and the importance of understanding statistical bias.