Codebreakers' Victory: How the Allied Cryptographers Won World War II
Click on image to enlarge.
by Hervie Haufler
Category: General Nonfiction
Description: For years, the story of the World War II codebreakers was kept a crucial state secret. Even Winston Churchill, himself a great advocate of Britain's cryptologic program, purposefully minimized their achievements in his history books. Now though, after decades have passed, the true scope of the British and American cryptographers' role in the war has come to light. It was a role key to the Allied victory. From the Battle of Britain to the Pacific front to the panzer divisions in Africa, superior cryptography gave the Allies a decisive advantage over the Axis generals. Military intelligence made a significant difference in battle after battle. In CODEBREAKERS' VICTORY, veteran cryptographer Hervie Haufler takes readers behind the scenes in this fascinating underground world of ciphers and decoders. This broad view represents the first comprehensive account of codebreaking during World War II. Haufler pulls together years of research, exclusive access to top secret files and personal interviews to craft a captivating must-read for anyone interested in the behind-the-front intellect and perseverance that went into beating the Nazis and Japan.
eBook Publisher: E-Reads/E-Reads, 2003
eBookwise Release Date: March 2011
Available eBook Formats: OEBFF Format (IMP) [638 KB]
Reading time: 384-538 min.
Belligerents: Choose Your Code Machines
* * * *
Down through military and diplomatic history, where there has been an adversary there has been cryptology. It is a two-sided art. On one side are the methods used to prevent communications with colleagues and allies from being read by rivals. On the other are the technologies for penetrating shielded messages and extracting their meaning for information that can provide an advantage in acting against the opposition.
Historians of cryptology, such as David Kahn, can trace the beginnings of this art back to some of the earliest civilizations of recorded history, including the ancient Egyptians, Mesopotamians, Babylonians, Hebrews, Greeks, and Romans. The Greek Herodotus tells us, as a celebrated example, how a Spartan, Demaratus, scratched on a wooden tablet the letters warning the Greek nations that the Persians under Xerxes were about to invade and then covered the letters with wax in order to conceal them from guards along the way. So alerted, the Greeks beat off the Persians and ended the threat of conquest.
In those early days, the simplest encoding was enough to throw off would-be codebreakers. The most-used systems were ciphers, in which the original letters of a message--its plaintext--were transposed or replaced by other letters. One of Julius Caesar's ciphers, as an instance, consisted of substituting plaintext letters with those that were three places farther along in the Roman alphabetic sequence. Even that was too complicated for his great-nephew Augustus, the first emperor of Rome, who simply substituted the very next letter in the alphabet. Use of these single-alphabet--or monoalphabetic--ciphers could convert the plaintext order attack into the meaningless jumble taktac by the transposition of letters, or buubdl by applying Augustus's substitution method.
Over time, a second type of cryptographic system that came into widespread use was that of codes. For key words he wanted to communicate, the codemaker would create lists of unintelligible equivalents--letters, numbers or symbols--and assemble them in codebooks to be held both by senders and receivers. To signal attack the sender would look up the equivalent--1502, say--in his book. The receiver would, in turn, track down that number in his codebook, read its plaintext meaning and know what his commander was ordered to do.
Those primitive types of codes were easy prey for cryptanalysts who discovered the technique of frequency analysis. By counting the number of times a given letter appeared in a message, the analyst could take a good guess that the most frequently used code letter stood for the most common plaintext letter in a given language--e, in English for example--and thus gain a lever for opening up the remainder of the plaintext.
It was essential that codemakers come up with new ways to outwit the cryptanalysts. One way to foil frequency analysis was to treat the consonants of a code alphabet as usual, but to add several different variants for e and each of the other vowels. An even better system was invented by Leon Battista Alberti, a Florentine born in 1404. Alberti developed a cipher disk, with the plaintext alphabet on the outer ring and a cipher alphabet on the inner ring. By prearrangement between sender and receiver, the sender would set his cipher ring at, say, G under the plaintext A and encode several words. But then he would turn the cipher ring so that Y would fall under A. In effect he would bring a whole new cipher alphabet into play. With other shifts, he would employ additional alphabets. This was the start of the multi-alphabet, or polyalphabetic, substitution system that, in ever more advanced forms, remained a cryptographer's mainstay for centuries.
So it went, down through the ages. In medieval England, both Roger Bacon and Geoffrey Chaucer were ardent users of cryptography. During the Renaissance, the city-states of Italy raised the art to high levels of sophistication. Napoleon Bonaparte's neglect of more secure forms of code-making led to his first great defeat. In a preview of a much later cryptanalytic triumph, the Russians read Napoleon's dispatches and combined these disclosures with the rigors of the Russian winter to turn back the invaders at the gates of Moscow.
New technologies necessitated new security safeguards for communications. Invention of the telegraph, for example, led Union commanders in the American Civil War to seek better ways to prevent their messages from being betrayed to Confederate generals. For Union generals McClellan and Grant, cryptographers devised codebooks in which user-friendly ordinary words substituted for plaintext. Colonel became Venus, Neptune was Richmond, and Adam was President Lincoln.
Superior cryptography gave Union leaders an advantage over their Confederate opponents. The South relied on ciphers that included a version of the venerable Vigenere system that Kahn has described as "probably the most famous cipher system of all time." Its origination is attributed to Blaise de Vigenere, who lived in the sixteenth century. He used a square bounded on the left side by a vertical alphabet and across the top by a horizontal one. Within the square, each horizontal row is another alphabet, which begins with the letter of the left-hand column. To replace a plaintext letter with a cipher one, the cryptographer traced the column beneath the top horizontal letter down to its row on the vertical alphabet. In other words, this was a creative complication of Alberti's polyalphabetic ciphers. When combined with a key word or phrase and such complications as reversed alphabets, Vigenere confounded cryptanalysts for a long time. But as early as the 1840s, America's Charles Babbage demonstrated how to solve the Vigenere system, and Union analysts' unraveling of the Confederates' use of them contributed to the South's defeat.
Another cipher invented to assure secrecy in telegraphic messages was named Playfair, after a British baron, although it was devised not by him but by his friend Charles Wheatstone. This cipher also uses a square, with a key word rather than an alphabet across the top. Instead of yielding just letter transpositions, though, it delivers "digraphs," in which two letters of a message are enciphered together. Since there are 26 letters but 676 digraphs, the use of digraphs overcame the limitations of alphabets and sharply complicated life for the cryptanalyst.
* * * *
Great War Debacles Demand Cryptologic Change
As the Great War of 1914-18 began, military communications faced a formidable new technological challenge, that of radio. Marconi's not-yet-twenty-year-old invention put telegraphy on the air. With streams of Morse code sprayed out from their command center, deskbound admirals could direct warships far out at sea, and generals were able to better control highly mobile gasoline-driven armies.
Along with this positive came a negative. Skulduggery was no longer necessary to secure an enemy's messages; interceptors as well as intended recipients could pluck them out of the ether by monitoring message-transmitting frequencies. Cryptanalysts were given masses of enemy communications to work with.
The situation called for groundbreaking new codes. They weren't forthcoming. Military communicators were still relying on pen-and-paper ciphers left over from the previous century, often no more than variations of Vigenere or Playfair systems.
Cryptanalysts of warring nations were presented with opportunities they moved quickly to exploit. The French were best prepared, with a group of codebreakers who had been working together since well before the war began. They also had in place both a line of intercept stations and the beginnings of sites for the direction-finding of enemy transmitters. In London, the British organized the now famous Room 40, where some of the nation's best minds concentrated on messages fetched in by a new line of coastal intercept centers. Germany launched into the conflict without a single cryptanalyst on the western front, but then strove mightily to catch up.
With all this emphasis on codebreaking, the Great War soon became a codemaker's nightmare. Cryptanalysts held the upper hand. Everyone was breaking everyone else's codes.
The Germans were the first to reap a major victory from their opponent's cryptographic failures. They did this against the Russians pressing in on them from the east. The French tried to help their more primitively equipped allies by supplying them with codebooks, but the czarist government and military were so corrupt that the code was quickly betrayed, for a payoff, to the Germans. Efforts by the Russian commanders to introduce a new code came to nought. In August 1914, as they approached the Battle of Tannenberg, the decisive struggle on the eastern front, the Russian leaders ran short of the wire and wire-laying equipment to communicate by telephone. Trying to coordinate their huge two-pronged pincer movement, they had no choice but to use radio--and to send their messages unenciphered. The Germans intercepted them and translated them. They revealed the Russians' entire plan of attack.
Intercepted messages in hand, the Germans knew how to counter the offensive. Aware that the Russians' northern wing, after an initial victory against the Germans, was pausing to reorganize, Generals Hindenburg and Ludendorff held that front with a thin screen of cavalry and concentrated their main forces to fall on the southern wing. They enveloped the Russian armies, killed some thirty thousand troops and captured one hundred thousand others, setting Russia on the long downward slide that ended in 1917 with the Bolshevik revolution and the Russian withdrawal from the war.
Early on, Britain's Room 40 began breaking German naval codes. The decrypts led to two relatively inconsequential British forays, but then were used with great effect on May 31, 1916, when decrypted messages warned that German navy commanders were massing their ships for a major offensive in the North Sea. The result was the climactic Battle of Jutland. "Without the cryptographic department," Winston Churchill wrote, "there would have been no Battle of Jutland." Although both navies were badly battered, the surviving German ships retreated into their home ports and did not again take on the Royal Navy throughout the rest of the war. With Room 40's aid, the British navy also ended the threat of German U-boats in their attempt to choke off Britain's Atlantic supply line.
Britain's breaking of another German message, the infamous Zimmermann telegram, brought the U.S. into the war. Wanting the U.S. to be a mediator for peace rather than a belligerent in the war, President Woodrow Wilson maintained American neutrality even after the Germans lifted their embargo on submarine attacks against neutral ships and sank the Cunard liner Lusitania, with the loss of 128 American lives. The U.S. public reacted with such fury that the Germans reconsidered and, for four months, suspended their U-boat campaign. But then Arthur Zimmermann, the German foreign minister, hatched what he considered an inspired scheme, one that would keep the U.S. so occupied with troubles close to home that American leaders would be unable to think about involvement in Europe. His idea was to induce Mexico to join with Germany in an alliance that would provide German financial backing for the Mexican army to cross its northern borders and reclaim its lost territories in Texas, New Mexico and Arizona. Moreover, he proposed that the Mexican president persuade the Japanese to attack the American West Coast. He sent encrypted instructions via a cablegram to the German ambassador in Washington, who was to forward them to the German ambassador in Mexico City and thence to the Mexican president.
The British, however, were tapping Atlantic cable communications. They intercepted the telegram and deciphered it. But they weren't sure what to do next. To reveal its contents to the Americans would give away the fact that they were breaking the German codes, which was an unacceptable disclosure. Yet they knew that Zimmermann's plan would infuriate the Americans and would likely draw them into the war. They came up with a bright solution. The German's Washington ambassador would have to strip off the instructions meant just for him before the relay to Mexico. Consequently, the version arriving in Mexico City would vary from the intercept as well as have a different transmission date. Germany's ambassador to Mexico would deliver a deciphered version of the message to the Mexican president. The British scheme was to have one of their agents in Mexico City obtain a copy of this second message and then have that turned over to the Americans. It was all done so skillfully that the Germans blamed treachery in Mexico rather than suspecting the British.
Once the telegram was leaked to the press, headlines across the U.S. blared the incredible news. Any doubts about the message's authenticity were dispelled when Zimmermann admitted that he had sent it. With that, Wilson could no longer withstand the storm of rage the telegram stirred up; the U.S. declared war on Germany.
In March 1918, came the foremost cryptanalytic victory of the war. The German armies were closing in on Paris, preparing for the push that would seize the capital and drive France to make peace. German generals had been launching devastating surprise attacks because their cryptographers had devised a new cipher the French were unable to break. Known as the ADFGVX, it used only these letters of the alphabet because their Morse code equivalents were distinct from one another and less liable to be garbled in transmission. With German salients only thirty miles from Paris, close enough that the city was being bombarded by long-range Big Bertha artillery, French commanders were desperate to know where the next assault would fall. The task of breaking ADFGVX was left to France's most able cryptanalyst, young Lieutenant Georges Painvin. In an incredible feat of sleepless concentration, he broke the cipher and revealed when and where the Germans would strike. This time the French were ready for the German advance. The assault was beaten back and France was saved from defeat.
When the U.S. sent the American Expeditionary Force to France, Herbert O. Yardley, organizer of the first serious U.S. cryptographic program, went along to help with code work at the headquarters of the AEF commander, General John Pershing. Yardley was horrified to find the American forces relying on "schoolboy codes and ciphers" that, he was sure, the Germans were decoding as quickly as American operators. Nonetheless, the doughboys turned the course of the war toward triumph by their fresh vitality and overwhelming numbers, despite having their leaders' orders almost instantly known to the enemy.
The Great War was, indeed, a cryptographer's nightmare.
Well before the war's end, it was evident that a new order of military communications was required. The gasoline-powered mobility of modern armed forces needed radio to coordinate and direct their movements, which, in turn, called for faster and more secure methods of encryption and decryption than were possible with manual systems out of the past. It was time for machines to take on the tasks of cryptology.
* * * *
Inventors Concentrate on Rotor Machines
Late in the war, the British put forth a code machine, the work of J. St. Vincent Pletts, that they recommended for immediate use by Allied commands. To convince their U.S. allies, they sent over a sample to be tested. The machine was delivered to American cryptanalyst William Friedman, along with five encoded messages the British were sure would prove undecipherable. Friedman broke them in three hours, ending this early try at machine encoding.
The need for mechanical systems was so evident, however, that almost simultaneously inventors in four different countries began work on machines, each of which relied on the same idea. This was the application of the electric-powered rotor, a revolvable code wheel.
Of the four inventors, the one whose development was to have the greatest consequence in World War II was the German, Arthur Scherbius. His work on a rotor device gained a boost when the Dutch inventor who had received the first patent on the machine, Hugo Alexander Koch, assigned the rights to Scherbius a year before he himself died. After going through several transformations, the Scherbius machine emerged as a device resembling an ungainly typewriter housed in a wooden box. It had a keyboard like a typewriter, but with only three rows of keys for the twenty-six letters, and none left over for numbers, punctuation or other extras. Atop the machine to the rear of the keyboard was a plate in which twenty-six round glass apertures were labeled with the letters of the alphabet and positioned above glow lamps. When the operator pressed down a key, rather than a skeletal arm rising to print an impression on paper, one of the glow lamps would illuminate a letter.
The trick was that the lighted letter--the cipher letter--was never the same as the depressed key--the plaintext letter. Pushing down a key fed a battery-powered electrical impulse into the machine's interior, and thereby hangs a tale of clever complexity.
The Scherbius machine depended primarily on three rotors on a single shaft to do the encoding and decoding. Each rotor was a small hockey-puck-like disk of insulating material. Around its rim were double rows of electrical contacts, twenty-six in number, representing letters of the alphabet. The contacts on one side of a rotor were wired in a random internal arrangement to those on the other side. As a result, the plaintext letters of the message delivered to one side emerged on the other side as different letters, transposed and scrambled. Thus, if the plaintext letter entered the right-hand rotor as A, it might exit it as Q. Then, entering the second rotor as Q, it emerged as W. And entering the third rotor as W, it came out the far side as X.
On the left-hand wall of the machine was a fourth scrambling element: a fixed half rotor with thirteen contacts only on one side. This was the reflector, which the Germans called "the turnaround wheel." It bounced the electrical impulse back once again through the three rotors, rescrambling the order in the passage through each one.
The electrical surge did something else as well: it caused that first rotor to rotate one space, one twenty-sixth of a revolution. Otherwise, each plaintext letter entering the right side of the disk would invariably activate the same ciphertext letter on the other side--easy prey for cryptanalysts. By edging forward a notch each time a key was pressed, the entry letter's current flowed through a different contact on the cipher side. As a result, when a plaintext letter--B, say--was hit a second time--BB--the repeated plaintext letter became a different cipher letter. That is, with the first B enciphered as, say, M, the second would be different--X, say. When the first rotor completed its twenty-six-letter cycle, it triggered the second rotor to move forward a notch and, after its twenty-six-letter rotation, to activate the third rotor. In this way, the machine was constantly changing the interconnections, additionally altering the plaintext inputs.
As if that amount of letter scrambling weren't enough to foil crypt-analysis, Scherbius and his colleagues added a further complexity: the order of the rotors on their shaft could be changed. What had been the right-hand rotor could be switched to the left-hand slot or the middle one, and so on.
Another important feature of Scherbius's machine was the reciprocity of its lettering. If plaintext A lighted the glow lamp for ciphertext X, then on the deciphering side of the cycle, X invariably equaled A. It was this reversibility that allowed the receiver to instantly decipher what the sender had transmitted.
Scherbius called his machine the Enigma. Ironically, considering its subsequent history, he is said to have derived the name from a musical composition, Enigma Variations, in which the British composer Edward Elgar used melodic codes in characterizing some of his friends.
In seeking customers for his Enigma, Scherbius pointed out a critical advantage: the machine itself could be captured, but unlike a purloined codebook, it would still be useless to the captor. The reason was that in order to decode a message on the Enigma, it was necessary to know the starting positions of the rotors. This essential information was called the key. With the multirotor Enigma, the number of the key variations ran into the billions. To determine even one key, he argued, would take cryptanalysts years of effort.
His first attempts, in the 1920s, to market the Enigma to business customers as well as military chiefs met with rebuffs. The German navy considered the machine but turned him down. So did the commercial prospects he approached. But then English writers on the war, including Winston Churchill, gave Scherbius a lift. In Churchill's book The World Crisis, he revealed how British successes against the German fleet in the Great War stemmed in part from the breaking of German naval codes. His disclosures prompted German navy officers of the twenties to have second thoughts. They bought the Enigma and decided it was their cryptographic answer. The navy began using Enigmas in 1925. The army followed suit in 1928 and the newly reborn air force in 1935.
To make their Enigmas even more secure against cryptanalysis, the Germans introduced two major changes. The first was an increase in the number of rotors. They had their Enigmas built with slots to store two extra rotors. Their machines continued to operate with just three rotors, but the operator's ability to vary the sequence among the five available code wheels enormously increased the difficulties facing the would-be analyst. Later the navy upped the ante by adding an extra rotor and altering their machines to operate with four instead of three.
The second change was the introduction of an entirely new scrambling element, the plugboard, which looked like a miniature telephone switchboard. It included cables to facilitate the pairings of plugs and sockets for twenty-six letters. The operator could change these pairings to send current through the machine by entirely different paths.
With these changes, the Germans could instruct their Enigma operators on the sequence of rotors to insert, the start-up position of each rotor and the order of plugboard pairings. Now, when a German operator pressed down a key of his Enigma, the electrical impulse ran a most tortuous course. First it went through the plugboard maze of wiring, then proceeded one way through the rotors. At the end it was bounced back by that fixed-wheel reflector and returned by a different route through the rotors. Only then did it light the glow lamp.
In peacetime, changes in the settings were first made at quarterly intervals, then once a month and, later, once a week. When the war came, changes were made once a day or, in some cases, every eight hours.
The Enigma required at least two operators, one to strike the plaintext keys, the other to read and copy down the lighted ciphertext letters. For the fastest operation, extra operators were used, the final one transmitting the gobbledygook letter groups over the air.
Progressively altering and improving the Enigma, the Germans made it their all-purpose code machine. It was selected by the security police organizations, railroads and other governmental departments, in addition to the military services.
Thousands of Enigmas were put into use. During the course of the war the number of different keys rose to nearly two hundred, and at some stages of the war the various German networks employed fifty different keys simultaneously.
The Germans had good reason to believe their Enigmas were secure against cryptanalysis. Dr. Ray Miller, a computer scientist at the U.S. National Security Agency, has calculated the exact number of key settings faced by Enigma codebreakers. The possible permutations for the plugboards alone, he has determined, run to more than 500 trillion. And that was just one of the machine's five variable components. All the variables together multiply out to 3 x 10114. That number compares with only 1080 as the estimated number of atoms in the entire observable universe. "No wonder," he concluded, "the German cryptographers had confidence in their machine!"
Did the Germans ever suspect that their enemies were reading their Enigma-encoded messages? At times when the Allies seemed to benefit from what seemed like amazing coincidences and incredible streaks of good fortune, questions were raised. Investigations were conducted. Always, at least until the very end of the war, the answers came back uniformly: the Enigma-based communications systems were inviolate. It was inconceivable that human minds could cope with such astronomical numbers of variations. If the Allies fared better than could be explained by brilliance or luck, the cause had to be secret agents, not penetration of the Enigma.