In cryptography and steganography, plausibly deniable encryption describes encryption techniques where the existence of an encrypted file or message is deniable in the sense that an adversary cannot prove that the plaintext data exists.
The users may convincingly deny that a given piece of data is encrypted, or that they are able to decrypt a given piece of encrypted data, or that some specific encrypted data exists. Such denials may or may not be genuine. For example, it may be impossible to prove that the data is encrypted without the cooperation of the users. If the data is encrypted, the users genuinely may not be able to decrypt it. Deniable encryption serves to undermine an attacker's confidence either that data is encrypted, or that the person in possession of it can decrypt it and provide the associated plaintext.
Function
Deniable encryption makes it impossible to prove the existence of the plaintext message without the proper decryption key. This may be done by allowing an encrypted message to be decrypted to different sensible plaintexts, depending on the key used. This allows the sender to have plausible deniability if compelled to give up their encryption key. The notion of "deniable encryption" was used by Julian Assange and Ralf Weinmann in the Rubberhose filesystem and explored in detail in a paper by Ran Canetti, Cynthia Dwork, Moni Naor, and Rafail Ostrovsky in 1996.
Scenario
Deniable encryption allows the sender of an encrypted message to deny sending that message. This requires a trusted third party. A possible scenario works like this:
Bob suspects his wife Alice is engaged in adultery. That being the case, Alice wants to communicate with her secret lover Carl. She creates two keys, one intended to be kept secret, the other intended to be sacrificed. She passes the secret key (or both) to Carl. Alice constructs an innocuous message M1 for Carl (intended to be revealed to Bob in case of discovery) and an incriminating love letter M2 to Carl. She constructs a cipher-text C out of both messages, M1 and M2, and emails it to Carl. Carl uses his key to decrypt M2 (and possibly M1, in order to read the fake message, too). Bob finds out about the email to Carl, becomes suspicious and forces Alice to decrypt the message. Alice uses the sacrificial key and reveals the innocuous message M1 to Bob. Since it is impossible for Bob to know for sure that there might be other messages contained in C, he might assume that there are no other messages (alternatively, Bob may not be familiar with the concept of plausible encryption in the first place, and thus may not be aware it is even possible for C to contain more than one message).
Another possible scenario involves Alice sending the same ciphertext (some secret instructions) to Bob and Carl, to whom she has handed different keys. Bob and Carl are to receive different instructions and must not be able to read each other's instructions. Bob will receive the message first and then forward it to Carl.
Alice constructs the ciphertext out of both messages, M1 and M2, and emails it to Bob. Bob uses his key to decrypt M1 and isn't able to read M2. Bob forwards the ciphertext to Carl. Carl uses his key to decrypt M2 and isn't able to read M1.
Forms of deniable encryption
Normally, ciphertexts decrypt to a single plaintext that is intended to be kept secret. However, one form of deniable encryption allows its users to decrypt the ciphertext to produce a different (innocuous but plausible) plaintext and plausibly claim that it is what they encrypted. The holder of the ciphertext will not be able to differentiate between the true plaintext, and the bogus-claim plaintext. In general, one ciphertext cannot be decrypted to all possible plaintexts unless the key is as large as the plaintext, so it is not practical in most cases for a ciphertext to reveal no information whatsoever about its plaintext. However, some schemes allow decryption to decoy plaintexts that are close to the original in some metric (such as edit distance).
Modern deniable encryption techniques exploit the fact that without the key, it is infeasible to distinguish between ciphertext from block ciphers and data generated by a cryptographically secure pseudorandom number generator (the cipher's pseudorandom permutation properties).
This is used in combination with some decoy data that the user would plausibly want to keep confidential that will be revealed to the attacker, claiming that this is all there is. This is a form of steganography.