We define the erasure entropy of a collection of random variables as the sum of entropies of the individual variables conditioned on all the rest. The erasure entropy measures the information content carried by each symbol knowing its context.
In the setup of a source observed through an erasure channel, we offer an operational characterization of erasure entropy rate as the minimal amount of bits per erasure required to recover the erased information in the limit of small erasure probability.
When we allow recovery of the erased symbols within a prescribed degree of distortion, the fundamental tradeoff is described by the erasure rate-distortion function which we characterize. Connections between the erasure rate-distortion function and the regular one are established.
We derive a lower bound on the erasure rate-distortion function which is analogous to the Shannon Lower Bound on the regular rate-distortion function. We show that the erasure rate-distortion function is derivable in closed form for various sources whose regular rate-distortion function is not known in closed form. This includes the class of stationary binary source and random fields (under Hamming loss).
When no additional encoded information is available, the erased information is reconstructed solely on the basis of its context by a denoiser. Connections between erasure entropy and discrete denoising are developed.
Based on joint work with Tsachy Weissman.