Cryptology: Empirical Values for Natural Languages

Cryptology

Empirical Values for Natural Languages

a7Hzq .#5r< kÜ\as TâÆK$ ûj(Ö2 ñw%h: Úk{4R f~`z8 ¤˜Æ+Ô „&¢Dø

Here are some additional explicit examples.

Conclusion

These examples show some trends:

[For solid statistical inferences of course we should take much more samples. See the next subsections.]

The typical coincidence index of two English texts is about 0.06.
The typical coincidence index of two German texts is about 0.08.
The typical coincidence index of a German and an English text is about 0.06 to 0.07.
The typical coincidence index of a plaintext and a ciphertext is about 0.03 bis 0.05—that is near the »random« value 1/26 = 0.0385.
Using the same key for polyalphabetically encrypting two different texts is revealed by a coincidence index near the value for two palintexts.

This last statement represents the first application of coincidence indices. No matter whether the encryption is periodic or not—if the cryptanalyst gets several ciphertexts encrypted by the same procedure with the same key, they can arrange them in parallel as rows and get monoalphabetically encrypted columns that offer a handle for successful cryptanalysis. In a military context where a key often was in use for a complete day the enemy cryptanalyst eventually could collect sufficiently many messages for applying this method.

Historical Example

The Polish cryptanalyst Rejewski was the first who successfully broke early versions of the German cipher machine Enigma, see »Cryptanalysis of rotor machines«. He detected that ciphertexts were »in phase« by coincidence counts. It is unknown whether he knew Friedman's approach, or whether he found it for himself.

Friedman's early publications were not classified and published even in France.

For example he noted that the two ciphertexts

RFOWL DOCAI HWBGX EMPTO BTVGG INFGR OJVDD ZLUWS JURNK KTEHM RFOWL DNWEL SCAPX OAZYB BYZRG GCJDX NGDFE MJUPI MJVPI TKELY

besides having the initial six letters identical also had an increased coincidence between the remaining 44 letters.

Exercise: How many coincidences would you expect for independent texts?

Rejewski concluded that the first six letters denoted a »message key« that was identical for the two messages, and from this that the Enigma operators prefixed their messages by a six letter message key. (Later on he even detected that in fact they used a repeated three letter key.)

[Source: F. L. Bauer: Mathematik besiegte in Polen die unvernünftig gebrauchte ENIGMA. Informatik Spektrum 1. Dezember 2005, 493 -497.]

More Empirical Observations

Author: Klaus Pommerening, 2002-May-20; last change: 2014-Jan-23.