Information sources with different costscales and the principle of conservation of entropy. Probability Theory and Related Fields. 1969;12(3):185-222. Abstract
Information sources with different costscales and the principle of conservation of entropy
The aim of this paper is to provide a mathematically rigorous and sufficiently general treatment of the basic information-theoretic problems concerning sources with symbols of different costs and noiseless coding in a general sense. The main new concepts defined in this paper are the entropy rate (entropy per unit cost) of a source with respect to a stochastic cost scale and the encoding (in particular decodable encoding) of a source in a general sense. On the basis of these concepts, we prove some general theorems on the relation of entropy rates with respect to different cost scales and on the effect of encoding to the entropy rate. In particular, the ldquoprinciple of conservation of entropyrdquo and the ldquonoiseless coding theoremrdquo are proved under very general conditions.The main results of this research have been reported at the Second Congress of Bulgarian Mathematicians, Varna, August 1967, at the International Symposium on Information Theory, San Remo, September 1967 and at the Colloquium on Information Theory, Debrecen, September 1967.