Abstract: Conventionally, posterior matching is investigated in channel coding and block encoding contexts – the source symbols are equiprobably distributed and are entirely known by the encoder before the transmission. In this paper, we consider a streaming source, whose symbols progressively arrive at the encoder at a sequence of deterministic times. We derive the joint source-channel coding (JSCC) reliability function for streaming over a discrete memoryless channel (DMC) with feedback. We propose a novel instantaneous encoding phase that operates during the symbol arriving period and achieves the JSCC reliability function for streaming when followed by a block encoding scheme that achieves the JSCC reliability function for a classical source whose symbols are fully accessible before the transmission. During the instantaneous encoding phase, the evolving message alphabet is partitioned into groups whose priors are close to the capacity-achieving distribution, and the encoder determines the group index of the actual sequence of symbols arrived so far and applies randomization to exactly match the distribution of the transmitted index to the capacity-achieving one. Surprisingly, the JSCC reliability function for streaming is equal to that for a fully accessible source, implying that the knowledge of the entire symbol sequence before the transmission offers no advantage in terms of the reliability function. For streaming over a symmetric binary-input DMC, we propose a one-phase instantaneous small-enough difference (SED) code that not only achieves the JSCC reliability function, but also, thanks to its single-phase time-invariant coding rule, can be used to stabilize an unstable linear system over a noisy channel. For equiprobably distributed source symbols, we design low complexity algorithms to implement both the instantaneous encoding phase and the instantaneous SED code. The algorithms group the source sequences into sets we call types, which enable the encoder and the decoder to track the priors and the posteriors of source sequences jointly, leading to a log-linear complexity in time. While the reliability function is derived for non-degenerate DMCs, i.e., DMCs whose transition probability matrix has all positive entries, for degenerate DMCs, we design a code with instantaneous encoding that achieves zero error for all rates below Shannon’s joint source-channel coding limit.

Publication: IEEE Transactions on Information Theory Vol.: 69 No.: 4 ISSN: 0018-9448

ID: CaltechAUTHORS:20230502-16744400.1

]]>

Abstract: We consider the basic problem of querying an expert oracle for labeling a dataset in machine learning. This is typically an expensive and time consuming process and therefore, we seek ways to do so efficiently. The conventional approach involves comparing each sample with (the representative of) each class to find a match. In a setting with N equally likely classes, this involves N/2 pairwise comparisons (queries per sample) on average. We consider a k-ary query scheme with k ≥ 2 samples in a query that identifies (dis)similar items in the set while effectively exploiting the associated transitive relations. We present a randomized batch algorithm that operates on a round-by-round basis to label the samples and achieves a query rate of O(N/k²). In addition, we present an adaptive greedy query scheme, which achieves an average rate of ≈0.2N queries per sample with triplet queries. For the proposed algorithms, we investigate the query rate performance analytically and with simulations. Empirical studies suggest that each triplet query takes an expert at most 50% more time compared with a pairwise query, indicating the effectiveness of the proposed k-ary query schemes. We generalize the analyses to nonuniform class distributions when possible.

Publication: IEEE Transactions on Pattern Analysis and Machine Intelligence Vol.: 44 No.: 11 ISSN: 0162-8828

ID: CaltechAUTHORS:20221031-572094900.1

]]>

Abstract: We consider an unstable scalar linear stochastic system, X_(n+1) = aX_n + Z_n − U_n , where a ≥ 1 is the system gain, Z_n's are independent random variables with bounded α-th moments, and U_n's are the control actions that are chosen by a controller who receives a single element of a finite set {1,…,M} as its only information about system state Xi. We show new proofs that M > a is necessary and sufficient for β-moment stability, for any β < α. Our achievable scheme is a uniform quantizer of the zoom-in/zoom-out type that codes over multiple time instants for data rate efficiency; the controller uses its memory of the past to correctly interpret the received bits. We analyze the performance of our scheme using probabilistic arguments. We show a simple proof of a matching converse using information-theoretic techniques. Our results generalize to vector systems, to systems with dependent Gaussian noise, and to the scenario in which a small fraction of transmitted messages is lost.

Publication: IEEE Transactions on Automatic Control Vol.: 67 No.: 10 ISSN: 0018-9286

ID: CaltechAUTHORS:20211217-98215000

]]>

Abstract: Consider the following distributed optimization scenario. A worker has access to training data that it uses to compute the gradients while a server decides when to stop iterative computation based on its target accuracy or delay constraints. The server receives all its information about the problem instance from the worker via a rate-limited noiseless communication channel. We introduce the principle we call differential quantization (DQ) that prescribes compensating the past quantization errors to direct the descent trajectory of a quantized algorithm towards that of its unquantized counterpart. Assuming that the objective function is smooth and strongly convex, we prove that differentially quantized gradient descent (DQ-GD) attains a linear contraction factor of $\max \{\sigma _{\mathrm {GD}}, \rho _{n} 2^{-R}\}$ , where $\sigma _{\mathrm {GD}}$ is the contraction factor of unquantized gradient descent (GD), $\rho _{n} \geq 1$ is the covering efficiency of the quantizer, and $R$ is the bitrate per problem dimension $n$ . Thus at any $R\geq \log _{2} \rho _{n} /\sigma _{\mathrm {GD}}$ bits, the contraction factor of DQ-GD is the same as that of unquantized GD, i.e., there is no loss due to quantization. We show a converse demonstrating that no algorithm within a certain class can converge faster than $\max \{\sigma _{\mathrm {GD}}, 2^{-R}\}$ . Since quantizers exist with $\rho _{n} \to 1$ as $n \to \infty $ (Rogers, 1963), this means that DQ-GD is asymptotically optimal. In contrast, naively quantized GD where the worker directly quantizes the gradient barely attains $\sigma _{\mathrm {GD}} + \rho _{n}2^{-R}$ . The principle of differential quantization continues to apply to gradient methods with momentum such as Nesterov’s accelerated gradient descent, and Polyak’s heavy ball method. For these algorithms as well, if the rate is above a certain threshold, there is no loss in contraction factor obtained by the differentially quantized algorithm compared to its unquantized counterpart, and furthermore, the differentially quantized heavy ball method attains the optimal contraction achievable among all (even unquantized) gradient methods. Experimental results on least-squares problems validate our theoretical analysis.

Publication: IEEE Transactions on Information Theory Vol.: 68 No.: 9 ISSN: 0018-9448

ID: CaltechAUTHORS:20220909-232702000

]]>

Abstract: We consider the following communication scenario. An encoder causally observes the Wiener process and decides when and what to transmit about it. A decoder estimates the process using causally received codewords in real time. We determine the causal encoding and decoding policies that jointly minimize the mean-square estimation error, under the long-term communication rate constraint of R bits per second. We show that an optimal encoding policy can be implemented as a causal sampling policy followed by a causal compressing policy. We prove that the optimal encoding policy samples the Wiener process once the innovation passes either √1/R or −√1/R and compresses the sign of innovation (SOI) using a 1-bit codeword. The SOI coding scheme achieves the operational distortion-rate function, which is equal to D^(op)(R)=1/6R. Surprisingly, this is significantly better than the distortion-rate tradeoff achieved in the limit of infinite delay by the best noncausal code. This is because the SOI coding scheme leverages the free timing information supplied by the zero-delay channel between the encoder and the decoder. The key to unlocking that gain is the event-triggered nature of the SOI sampling policy. In contrast, the distortion-rate tradeoffs achieved with deterministic sampling policies are much worse: we prove that the causal informational distortion-rate function in that scenario is as high as D_(DET)(R)=5/6R. It is achieved by the uniform sampling policy with the sampling interval 1/R. In either case, the optimal strategy is to sample the process as fast as possible and to transmit 1-bit codewords to the decoder without delay. We show that the SOI coding scheme also minimizes the mean-square cost of a continuous-time control system driven by the Wiener process and controlled via rate-constrained impulses.

Publication: IEEE Transactions on Automatic Control Vol.: 67 No.: 4 ISSN: 0018-9286

ID: CaltechAUTHORS:20210412-071230573

]]>

Abstract: Consider the following communication scenario. An encoder observes a stochastic process and causally decides when and what to transmit about it, under a constraint on the expected number of bits transmitted per second. A decoder uses the received codewords to causally estimate the process in real time. The encoder and the decoder are synchronized in time. For a class of continuous Markov processes satisfying regularity conditions, we find the optimal encoding and decoding policies that minimize the end-to-end estimation mean-square error under the rate constraint. We show that the optimal encoding policy transmits a 1-bit codeword once the process innovation passes one of two thresholds. The optimal decoder noiselessly recovers the last sample from the 1-bit codewords and codeword-generating time stamps, and uses it to decide the running estimate of the current process, until the next codeword arrives. In particular, we show the optimal causal code for the Ornstein-Uhlenbeck process and calculate its distortion-rate function. Furthermore, we show that the optimal causal code also minimizes the mean-square cost of a continuous-time control system driven by a continuous Markov process and controlled by an additive control signal.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 12 ISSN: 0018-9448

ID: CaltechAUTHORS:20211222-322820000

]]>

Abstract: An n-dimensional source with memory is observed by K isolated encoders via parallel channels, who compress their observations to transmit to the decoder via noiseless rate-constrained links while leveraging their memory of the past. At each time instant, the decoder receives K new codewords from the observers, combines them with the past received codewords, and produces a minimum-distortion estimate of the latest block of n source symbols. This scenario extends the classical one-shot CEO problem to multiple rounds of communication with communicators maintaining the memory of the past. We extend the Berger-Tung inner and outer bounds to the scenario with inter-block memory, showing that the minimum asymptotically (as n→∞) achievable sum rate required to achieve a target distortion is bounded by minimal directed mutual information problems. For the Gauss-Markov source observed via K parallel AWGN channels, we show that the inner bound is tight and solve the corresponding minimal directed mutual information problem, thereby establishing the minimum asymptotically achievable sum rate. Finally, we explicitly bound the rate loss due to a lack of communication among the observers; that bound is attained with equality in the case of identical observation channels. The general coding theorem is proved via a new nonasymptotic bound that uses stochastic likelihood coders and whose asymptotic analysis yields an extension of the Berger-Tung inner bound to the causal setting. The analysis of the Gaussian case is facilitated by reversing the channels of the observers.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 12 ISSN: 0018-9448

ID: CaltechAUTHORS:20211222-495165300

]]>

Abstract: This paper presents finite-blocklength achievability bounds for the Gaussian multiple access channel (MAC) and random access channel (RAC) under average-error and maximal-power constraints. Using random codewords uniformly distributed on a sphere and a maximum likelihood decoder, the derived MAC bound on each transmitter’s rate matches the MolavianJazi-Laneman bound (2015) in its first- and second-order terms, improving the remaining terms to ½ log n/n + O(1/n) bits per channel use. The result then extends to a RAC model in which neither the encoders nor the decoder knows which of K possible transmitters are active. In the proposed rateless coding strategy, decoding occurs at a time n_t that depends on the decoder’s estimate t of the number of active transmitters k. Single-bit feedback from the decoder to all encoders at each potential decoding time n_i, I ≤ t , informs the encoders when to stop transmitting. For this RAC model, the proposed code achieves the same first-, second-, and third-order performance as the best known result for the Gaussian MAC in operation.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 11 ISSN: 0018-9448

ID: CaltechAUTHORS:20211122-171053652

]]>

Abstract: A key result of classical information theory states that if the rate of a randomly generated codebook is less than the mutual information between the channel’s input and output, then the probability that that codebook has negligible error goes to one as the blocklength goes to infinity. In an attempt to bridge the gap between the probabilistic world of classical information theory and the combinatorial world of zero-error information theory, this work derives necessary and sufficient conditions on the rate so that the probability that a randomly generated codebook operated under list decoding (for any fixed list size) has zero error probability goes to one as the blocklength goes to infinity. Furthermore, this work extends the classical birthday problem to an information-theoretic setting, which results in the definition of a “noisy” counterpart of Rényi entropy, analogous to how mutual information can be considered a noisy counterpart of Shannon entropy.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 9 ISSN: 0018-9448

ID: CaltechAUTHORS:20210825-143502652

]]>

Abstract: This paper provides a precise error analysis for the maximum likelihood estimate â_(ML)(uⁿ₁) of the parameter a given samples uⁿ₁ = (u₁, ... , u_n)ʹ drawn from a nonstationary Gauss-Markov process U_i = aU_(i-1) + Z)_i, i ≥ 1, where U₀ = 0, a > 1, and Zi ’s are independent Gaussian random variables with zero mean and variance σ². We show a tight nonasymptotic exponentially decaying bound on the tail probability of the estimation error. Unlike previous works, our bound is tight already for a sample size of the order of hundreds. We apply the new estimation bound to find the dispersion for lossy compression of nonstationary Gauss-Markov sources. We show that the dispersion is given by the same integral formula that we derived previously for the asymptotically stationary Gauss-Markov sources, i.e., |a|<1 . New ideas in the nonstationary case include separately bounding the maximum eigenvalue (which scales exponentially) and the other eigenvalues (which are bounded by constants that depend only on a) of the covariance matrix of the source sequence, and new techniques in the derivation of our estimation error bound.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 4 ISSN: 0018-9448

ID: CaltechAUTHORS:20210211-151615781

]]>

Abstract: Consider a random access communication scenario over a channel whose operation is defined for any number of possible transmitters. As in the model recently introduced by Polyanskiy for the Multiple Access Channel (MAC) with a fixed, known number of transmitters, the channel is assumed to be invariant to permutations on its inputs, and all active transmitters employ identical encoders. Unlike the Polyanskiy model, in the proposed scenario, neither the transmitters nor the receiver knows which transmitters are active. We refer to this agnostic communication setup as the Random Access Channel (RAC). Scheduled feedback of a finite number of bits is used to synchronize the transmitters. The decoder is tasked with determining from the channel output the number of active transmitters, k, and their messages but not which transmitter sent which message. The decoding procedure occurs at a time n_t depending on the decoder’s estimate, t, of the number of active transmitters, k, thereby achieving a rate that varies with the number of active transmitters. Single-bit feedback at each time n_(i,i) ≤ t , enables all transmitters to determine the end of one coding epoch and the start of the next. The central result of this work demonstrates the achievability on a RAC of performance that is first-order optimal for the MAC in operation during each coding epoch. While prior multiple access schemes for a fixed number of transmitters require 2^k−1 simultaneous threshold rules, the proposed scheme uses a single threshold rule and achieves the same dispersion.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 4 ISSN: 0018-9448

ID: CaltechAUTHORS:20210113-163504947

]]>

Abstract: We study the stabilization of a linear control system with an unbounded random system gain where the controller must act based on a rate-limited observation of the state. More precisely, we consider the system X_(n+1)=A_nX_n+W_n−U_n, where the A_n’s are drawn independently at random at each time n from a known distribution with unbounded support, and where the controller receives at most R bits about the system state at each time from an encoder. We provide a time-varying achievable strategy to stabilize the system in a second-moment sense with fixed, finite R. While our previous result provided a strategy to stabilize this system using a variable-rate code, this work provides an achievable strategy using a fixed-rate code. The strategy we employ to achieve this is time-varying and takes different actions depending on the value of the state. It proceeds in two modes: a normal mode (or zoom-in), where the realization of A_n is typical, and an emergency mode (or zoom-out), where the realization of A_n is exceptionally large. To analyze the performance of the scheme we construct an auxiliary sequence that bounds the state X_n, and then bound auxiliary sequence in both the zoom-in and zoom-out modes.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 4 ISSN: 0018-9448

ID: CaltechAUTHORS:20210205-093044649

]]>

Abstract: We propose a two-layer coding architecture for communication of multiple users over a shared slotted medium enabling joint collision resolution and decoding. Each user first encodes its information bits with an outer code for reliability, and then transmits these coded bits with possible repetitions over transmission time slots of the access channel. The transmission patterns are dictated by the inner collision-resolution code and collisions with other users’ transmissions may occur. We analyze two types of codes for the outer layer: long-blocklength LDPC codes, and short-blocklength algebraic codes. With LDPC codes, a density evolution analysis enables joint optimization of both outer and inner code parameters for maximum throughput. With algebraic codes, we invoke a similar analysis by approximating their average erasure correcting capability while assuming a large number of active transmitters. The proposed low-complexity schemes operate at a significantly smaller gap to capacity than the state of the art. Our schemes apply both to a multiple access scenario where the number of users within a frame is known a priori, and to a random access scenario where that number is known only to the decoder. In the latter case, we optimize an outage probability due to the variability in user activity.

Publication: IEEE Transactions on Wireless Communications Vol.: 19 No.: 12 ISSN: 1536-1276

ID: CaltechAUTHORS:20191004-135310393

]]>

Abstract: This work studies point-to-point, multiple access, and random access lossless source coding in the finite-blocklength regime. In each scenario, a random coding technique is developed and used to analyze third-order coding performance. Asymptotic results include a third-order characterization of the Slepian-Wolf rate region with an improved converse that relies on a connection to composite hypothesis testing. For dependent sources, the result implies that the independent encoders used by Slepian-Wolf codes can achieve the same third-order-optimal performance as a single joint encoder. The concept of random access source coding is introduced to generalize multiple access (Slepian-Wolf) source coding to the case where encoders decide independently whether or not to participate and the set of participating encoders is unknown a priori to both the encoders and the decoder. The proposed random access source coding strategy employs rateless coding with scheduled feedback. A random coding argument proves the existence of a single deterministic code of this structure that simultaneously achieves the third-order-optimal Slepian-Wolf performance for each possible active encoder set.

Publication: IEEE Transactions on Information Theory Vol.: 66 No.: 11 ISSN: 0018-9448

ID: CaltechAUTHORS:20200723-131543186

]]>

Abstract: Consider a control problem with a communication channel connecting the observer of a linear stochastic system to the controller. The goal of the controller is to minimize a quadratic cost function in the state variables and control signal, known as the linear quadratic regulator (LQR). We study the fundamental tradeoff between the communication rate r bits/sec and the expected cost b. We obtain a lower bound on a certain rate-cost function, which quantifies the minimum directed mutual information between the channel input and output that is compatible with a target LQR cost. The rate-cost function has operational significance in multiple scenarios of interest: among others, it allows us to lower-bound the minimum communication rate for fixed and variable length quantization, and for control over noisy channels. We derive an explicit lower bound to the rate-cost function, which applies to the vector, non-Gaussian, and partially observed systems, thereby extending and generalizing an earlier explicit expression for the scalar Gaussian system, due to Tatikonda el al. [2]. The bound applies as long as the differential entropy of the system noise is not −∞ . It can be closely approached by a simple lattice quantization scheme that only quantizes the innovation, that is, the difference between the controller's belief about the current state and the true state. Via a separation principle between control and communication, similar results hold for causal lossy compression of additive noise Markov sources. Apart from standard dynamic programming arguments, our technical approach leverages the Shannon lower bound, develops new estimates for data compression with coding memory, and uses some recent results on high resolution variablelength vector quantization to prove that the new converse bounds are tight.

Publication: IEEE Transactions on Automatic Control Vol.: 64 No.: 11 ISSN: 0018-9286

ID: CaltechAUTHORS:20190425-103602105

]]>

Abstract: In successive refinement of information, the decoder refines its representation of the source progressively as it receives more encoded bits. The rate-distortion region of successive refinement describes the minimum rates required to attain the target distortions at each decoding stage. In this paper, we derive a parametric characterization of the rate-distortion region for successive refinement of abstract sources. Our characterization extends Csiszár’s result to successive refinement, and generalizes a result by Tuncel and Rose, applicable for finite alphabet sources, to abstract sources. This characterization spawns a family of outer bounds to the rate-distortion region. It also enables an iterative algorithm for computing the rate-distortion region, which generalizes Blahut’s algorithm to successive refinement. Finally, it leads a new nonasymptotic converse bound. In all the scenarios where the dispersion is known, this bound is second-order optimal. In our proof technique, we avoid Karush–Kuhn–Tucker conditions of optimality, and we use basic tools of probability theory. We leverage the Donsker–Varadhan lemma for the minimization of relative entropy on abstract probability spaces.

Publication: IEEE Transactions on Information Theory Vol.: 65 No.: 10 ISSN: 0018-9448

ID: CaltechAUTHORS:20190613-142206819

]]>

Abstract: The Gauss-Markov source produces U_i = aU_(i–1) + Z_i for i ≥ 1, where U_0 = 0, |a| < 1 and Z_i ~ N(0; σ^2) are i.i.d. Gaussian random variables. We consider lossy compression of a block of n samples of the Gauss-Markov source under squared error distortion. We obtain the Gaussian approximation for the Gauss-Markov source with excess-distortion criterion for any distortion d > 0, and we show that the dispersion has a reverse waterfilling representation. This is the first finite blocklength result for lossy compression of sources with memory. We prove that the finite blocklength rate-distortion function R(n; d; ε) approaches the rate-distortion function R(d) as R(n; d; ε) = R(d)+ √ V(d)/n Q–1(ε)+o(1√n), where V (d) is the dispersion, ε ε 2 (0; 1) is the excess-distortion probability, and Q^(-1) is the inverse Q-function. We give a reverse waterfilling integral representation for the dispersion V (d), which parallels that of the rate-distortion functions for Gaussian processes. Remarkably, for all 0 < d ≥ σ^2 (1+|σ|)^2, R(n; d; ε) of the Gauss-Markov source coincides with that of Z_i, the i.i.d. Gaussian noise driving the process, up to the second-order term. Among novel technical tools developed in this paper is a sharp approximation of the eigenvalues of the covariance matrix of n samples of the Gauss-Markov source, and a construction of a typical set using the maximum likelihood estimate of the parameter a based on n observations.

Publication: IEEE Transactions on Information Theory Vol.: 65 No.: 10 ISSN: 0018-9448

ID: CaltechAUTHORS:20190610-093125985

]]>

Abstract: We consider the problem of controlling an unstable linear plant with Gaussian disturbances over an additive white Gaussian noise channel with an average transmit power constraint, where the signaling rate of communication may be different from the sampling rate of the underlying plant. Such a situation is quite common since sampling is done at a rate that captures the dynamics of the plant and that is often lower than the signaling rate of the communication channel. This rate mismatch offers the opportunity of improving the system performance by using coding over multiple channel uses to convey a single control action. In a traditional, separation-based approach to source and channel coding, the analog message is first quantized down to a few bits and then mapped to a channel codeword whose length is commensurate with the number of channel uses per sampled message. Applying the separation-based approach to control meets its challenges: first, the quantizer needs to be capable of zooming in and out to be able to track unbounded system disturbances, and second, the channel code must be capable of improving its estimates of the past transmissions exponentially with time, a characteristic known as anytime reliability. We implement a separated scheme by leveraging recently developed techniques for control over quantized-feedback channels and for efficient decoding of anytime-reliable codes. We further propose an alternative, namely, to perform analog joint source–channel coding, by this avoiding the digital domain altogether. For the case where the communication signaling rate is twice the sampling rate, we employ analog linear repetition as well as Shannon–Kotel’nikov maps to show a significant improvement in stability margins and linear-quadratic costs over separation-based schemes. We conclude that such analog coding performs better than separation, and can stabilize all moments as well as guarantee almost-sure stability.

Publication: IEEE Transactions on Automatic Control Vol.: 64 No.: 9 ISSN: 0018-9286

ID: CaltechAUTHORS:20190425-110709737

]]>

Abstract: We consider the problem of tracking the state of Gauss–Markov processes over rate-limited erasure-prone links. We concentrate first on the scenario in which several independent processes are seen by a single observer. The observer maps the processes into finite-rate packets that are sent over the erasure-prone links to a state estimator, and are acknowledged upon packet arrivals. The aim of the state estimator is to track the processes with zero delay and with minimum mean square error (MMSE). We show that, in the limit of many processes, greedy quantization with respect to the squared error distortion is optimal. That is, there is no tension between optimizing the MMSE of the process in the current time instant and that of future times. For the case of packet erasures with delayed acknowledgments, we connect the problem to that of compression with side information that is known at the observer and may be known at the state estimator—where the most recent packets serve as side information that may have been erased, and demonstrate that the loss due to a delay by one time unit is rather small. For the scenario where only one process is tracked by the observer–state estimator system, we further show that variable-length coding techniques are within a small gap of the many-process outer bound. We demonstrate the usefulness of the proposed approach for the simple setting of discrete-time scalar linear quadratic Gaussian control with a limited data-rate feedback that is susceptible to packet erasures.

Publication: IEEE Transactions on Control of Network Systems Vol.: 6 No.: 2 ISSN: 2325-5870

ID: CaltechAUTHORS:20180629-113827049

]]>

Abstract: We derive a lower bound on the differential entropy of a log-concave random variable X in terms of the p-th absolute moment of X. The new bound leads to a reverse entropy power inequality with an explicit constant, and to new bounds on the rate-distortion function and the channel capacity. Specifically, we study the rate-distortion function for log-concave sources and distortion measure d(x,x^)=|x−x^|r , with r ≥ 1 , and we establish that the difference between the rate-distortion function and the Shannon lower bound is at most log(√(πe)) ≈ 1.5 bits, independently of r and the target distortion d. For mean-square error distortion, the difference is at most log(√((πe)/2)) ≈ 1 bit, regardless of d. We also provide bounds on the capacity of memoryless additive noise channels when the noise is log-concave. We show that the difference between the capacity of such channels and the capacity of the Gaussian channel with the same noise power is at most log(√((πe)/2)) ≈ 1 bit. Our results generalize to the case of a random vector X with possibly dependent coordinates. Our proof technique leverages tools from convex geometry.

Publication: Entropy Vol.: 20 No.: 3 ISSN: 1099-4300

ID: CaltechAUTHORS:20180430-101112645

]]>

Abstract: This paper considers lossy source coding of n-dimensional memoryless sources and shows an explicit approximation to the minimum source coding rate required to sustain the probability of exceeding distortion d no greater than ϵ, which is simpler than known dispersion-based approximations. Our approach takes inspiration in the celebrated classical result stating that the Shannon lower bound to rate-distortion function becomes tight in the limit d → 0. We formulate an abstract version of the Shannon lower bound that recovers both the classical Shannon lower bound and the rate-distortion function itself as special cases. Likewise, we show that a nonasymptotic version of the abstract Shannon lower bound recovers all previously known nonasymptotic converses. A necessary and sufficient condition for the Shannon lower bound to be attained exactly is presented. It is demonstrated that whenever that condition is met, the rate-dispersion function is given simply by the varentropy of the source. Remarkably, all finite alphabet sources with balanced distortion measures satisfy that condition in the range of low distortions. Most continuous sources violate that condition. Still, we show that lattice quantizers closely approach the nonasymptotic Shannon lower bound, provided that the source density is smooth enough and the distortion is low. This implies that fine multidimensional lattice coverings are nearly optimal in the rate-distortion sense even at finite . The achievability proof technique is based on a new bound on the output entropy of lattice quantizers in terms of the differential entropy of the source, the lattice cell size, and a smoothness parameter of the source density. The technique avoids both the usual random coding argument and the simplifying assumption of the presence of a dither signal.

Publication: IEEE Transactions on Information Theory Vol.: 63 No.: 7 ISSN: 0018-9448

ID: CaltechAUTHORS:20170308-161418854

]]>

Abstract: This paper quantifies the fundamental limits of variable-length transmission of a general (possibly analog) source over a memoryless channel with noiseless feedback, under a distortion constraint. We consider excess distortion, average distortion and guaranteed distortion (d-semifaithful codes). In contrast to the asymptotic fundamental limit, a general conclusion is that allowing variable-length codes and feedback leads to a sizable improvement in the fundamental delay-distortion tradeoff. In addition, we investigate the minimum energy required to reproduce k source samples with a given fidelity after transmission over a memoryless Gaussian channel, and we show that the required minimum energy is reduced with feedback and an average (rather than maximal) power constraint.

Publication: IEEE Transactions on Information Theory Vol.: 63 No.: 6 ISSN: 0018-9448

ID: CaltechAUTHORS:20170524-171749779

]]>

Abstract: This paper shows new general nonasymptotic achievability and converse bounds and performs their dispersion analysis for the lossy compression problem in which the compressor observes the source through a noisy channel. While this problem is asymptotically equivalent to a noiseless lossy source coding problem with a modified distortion function, nonasymptotically there is a noticeable gap in how fast their minimum achievable coding rates approach the common rate-distortion function, as evidenced both by the refined asymptotic analysis (dispersion) and the numerical results. The size of the gap between the dispersions of the noisy problem and the asymptotically equivalent noiseless problem depends on the stochastic variability of the channel through which the compressor observes the source.

Publication: IEEE Transactions on Information Theory Vol.: 62 No.: 11 ISSN: 0018-9448

ID: CaltechAUTHORS:20160726-080006578

]]>

Abstract: This paper studies the fundamental limits of the minimum average length of lossless and lossy variable-length compression, allowing a nonzero error probability ε, for lossless compression. We give nonasymptotic bounds on the minimum average length in terms of Erokhin’s rate-distortion function and we use those bounds to obtain a Gaussian approximation on the speed of approach to the limit, which is quite accurate for all but small blocklengths: (1 − ε)kH(S) − ((kV(S)/2π))^1/2 exp[−((Q−1 (ε))^(2)/2)], where Q^−1 (·) is the functional inverse of the standard Gaussian complementary cumulative distribution function, and V(S) is the source dispersion. A nonzero error probability thus not only reduces the asymptotically achievable rate by a factor of 1 − ε, but this asymptotic limit is approached from below, i.e., larger source dispersions and shorter blocklengths are beneficial. Variablelength lossy compression under an excess distortion constraint is shown to exhibit similar properties.

Publication: IEEE Transactions on Information Theory Vol.: 61 No.: 8 ISSN: 0018-9448

ID: CaltechAUTHORS:20150729-151803634

]]>

Abstract: This paper shows the strong converse and the dispersion of memoryless channels with cost constraints and performs a refined analysis of the third-order term in the asymptotic expansion of the maximum achievable channel coding rate, showing that it is equal to (1/2)((log n)/n) in most cases of interest. The analysis is based on a nonasymptotic converse bound expressed in terms of the distribution of a random variable termed the mathsf b -tilted information density, which plays a role similar to that of the mathsf d -tilted information in lossy source coding. We also analyze the fundamental limits of lossy joint-source-channel coding over channels with cost constraints.

Publication: IEEE Transactions on Information Theory Vol.: 61 No.: 5 ISSN: 0018-9448

ID: CaltechAUTHORS:20150428-084727561

]]>

Abstract: Convexity properties of error rates of a class of decoders, including the maximum-likelihood/min-distance one as a special case, are studied for arbitrary constellations, bit mapping, and coding. Earlier results obtained for the additive white Gaussian noise channel are extended to a wide class of noise densities, including unimodal and spherically invariant noise. Under these broad conditions, symbol and bit error rates are shown to be convex functions of the signal-to-noise ratio (SNR) in the high-SNR regime with an explicitly determined threshold, which depends only on the constellation dimensionality and minimum distance, thus enabling an application of the powerful tools of convex optimization to such digital communication systems in a rigorous way. It is the decreasing nature of the noise power density around the decision region boundaries that ensures the convexity of symbol error rates in the general case. The known high/low-SNR bounds of the convexity/concavity regions are tightened and no further improvement is shown to be possible in general. The high-SNR bound fits closely into the channel coding theorem: all codes, including capacity-achieving ones, whose decision regions include the hardened noise spheres (from the noise sphere hardening argument in the channel coding theorem), satisfy this high-SNR requirement and thus has convex error rates in both SNR and noise power. We conjecture that all capacity-achieving codes have convex error rates. Convexity properties in signal amplitude and noise power are also investigated. Some applications of the results are discussed. In particular, it is shown that fading is convexity-preserving and is never good in low dimensions under spherically invariant noise, which may also include any linear diversity combining.

Publication: IEEE Transactions on Information Theory Vol.: 59 No.: 10 ISSN: 0018-9448

ID: CaltechAUTHORS:20190213-084137169

]]>

Abstract: This paper finds new tight finite-blocklength bounds for the best achievable lossy joint source-channel code rate, and demonstrates that joint source-channel code design brings considerable performance advantage over a separate one in the nonasymptotic regime. A joint source-channel code maps a block of k source symbols onto a length- n channel codeword, and the fidelity of reproduction at the receiver end is measured by the probability ε that the distortion exceeds a given threshold d . For memoryless sources and channels, it is demonstrated that the parameters of the best joint source-channel code must satisfy nC - kR( d ) ≈ √( nV + k V ( d )) Q^(-1) (ε), where C and V are the channel capacity and channel dispersion, respectively; R ( d ) and V ( d ) are the source rate-distortion and rate-dispersion functions; and Q is the standard Gaussian complementary cumulative distribution function. Symbol-by-symbol (uncoded) transmission is known to achieve the Shannon limit when the source and channel satisfy a certain probabilistic matching condition. In this paper, we show that even when this condition is not satisfied, symbol-by-symbol transmission is, in some cases, the best known strategy in the nonasymptotic regime.

Publication: IEEE Transactions on Information Theory Vol.: 59 No.: 5 ISSN: 0018-9448

ID: CaltechAUTHORS:20190213-090748243

]]>

Abstract: This paper studies the minimum achievable source coding rate as a function of blocklength n and probability ϵ that the distortion exceeds a given level d. Tight general achievability and converse bounds are derived that hold at arbitrary fixed blocklength. For stationary memoryless sources with separable distortion, the minimum rate achievable is shown to be closely approximated by R(d) + √V(d)/(n) Q^(-1)(ϵ), where R(d) is the rate-distortion function, V(d) is the rate dispersion, a characteristic of the source which measures its stochastic variability, and Q-1(·) is the inverse of the standard Gaussian complementary cumulative distribution function.

Publication: IEEE Transactions on Information Theory Vol.: 58 No.: 6 ISSN: 0018-9448

ID: CaltechAUTHORS:20140910-113526494

]]>

Abstract: Several instantaneous optimization strategies for rate and/or power allocation in the coded V-BLAST are studied analytically. Outage probabilities and system capacities of these strategies in a spatial multiplexing system are compared under generic settings. The conventional waterfilling algorithm is shown to be suboptimal for the coded V-BLAST and a new algorithm ("fractional water-filling") is proposed, which simultaneously maximizes the system capacity and minimizes the outage probability. Closed-form performance analysis of the considered algorithms is given, and the fractional water-filling algorithm is shown to attain the full MIMO channel diversity, significantly outperforming other strategies. Many of the results also apply to generic multi-stream transmission systems (e.g. spatial multiplexing on the channel eigenmodes, OFDM) or the systems relying on successive interference cancelation (multi-user detection, channel equalization).

Publication: IEEE Transactions on Communications Vol.: 59 No.: 10 ISSN: 0090-6778

ID: CaltechAUTHORS:20140910-105022320

]]>

Abstract: An analytical framework for performance analysis and optimization of coded V-BLAST is developed. Average power and/or rate allocations to minimize the outage probability as well as their robustness and dual problems are investigated. Compact, closed-form expressions for the optimum allocations and corresponding system performance are given. The uniform power allocation is shown to be near optimum in the low outage regime in combination with the optimum rate allocation. The average rate allocation provides the largest performance improvement (extra diversity gain), and the average power allocation offers a modest SNR gain limited by the number of transmit antennas but does not increase the diversity gain. The dual problems are shown to have the same solutions as the primal ones. All these allocation strategies are shown to be robust. The reported results also apply to coded multiuser detection and channel equalization systems relying on successive interference cancellation.

Publication: IEEE Transactions on Communications Vol.: 59 No.: 3 ISSN: 0090-6778

ID: CaltechAUTHORS:20140910-074750052

]]>

Abstract: Motivated by a recent surge of interest in convex optimization techniques, convexity/concavity properties of error rates of the maximum likelihood detector operating in the AWGN channel are studied and extended to frequency-flat slow-fading channels. Generic conditions are identified under which the symbol error rate (SER) is convex/concave for arbitrary multidimensional constellations. In particular, the SER is convex in SNR for any one- and two-dimensional constellation, and also in higher dimensions at high SNR. Pairwise error probability and bit error rate are shown to be convex at high SNR, for arbitrary constellations and bit mapping. Universal bounds for the SER first and second derivatives are obtained, which hold for arbitrary constellations and are tight for some of them. Applications of the results are discussed, which include optimum power allocation in spatial multiplexing systems, optimum power/time sharing to decrease or increase (jamming problem) error rate, an implication for fading channels (Â¿fading is never good in low dimensionsÂ¿) and optimization of a unitary-precoded OFDM system. For example, the error rate bounds of a unitary-precoded OFDM system with QPSK modulation, which reveal the best and worst precoding, are extended to arbitrary constellations, which may also include coding. The reported results also apply to the interference channel under Gaussian approximation, to the bit error rate when it can be expressed or approximated as a nonnegative linear combination of individual symbol error rates, and to coded systems.

Publication: IEEE Transactions on Information Theory Vol.: 56 No.: 4 ISSN: 0018-9448

ID: CaltechAUTHORS:20140910-093556659

]]>

Abstract: A unified analytical framework for optimum power allocation in the unordered V-BLAST algorithm and its comparative performance analysis are presented. Compact closed-form approximations for the optimum power allocation are derived, based on average total and block error rates. The choice of the criterion has little impact on the power allocation and, overall, the optimum strategy is to allocate more power to lower step transmitters and less to higher ones. High-SNR approximations for optimized average block and total error rates are given. The SNR gain of optimization is rigorously defined and studied using analytical tools, including lower and upper bounds, high and low SNR approximations. The gain is upper bounded by the number of transmit antennas, for any modulation format and type of fading channel. While the average optimization is less complex than the instantaneous one, its performance is almost as good at high SNR. A measure of robustness of the optimized algorithm is introduced and evaluated. The optimized algorithm is shown to be robust to perturbations in individual and total transmit powers. Based on the algorithm robustness, a pre-set power allocation is suggested as a low-complexity alternative to the other optimization strategies, which exhibits only a minor loss in performance over the practical SNR range.

Publication: IEEE Transactions on Communications Vol.: 56 No.: 6 ISSN: 0090-6778

ID: CaltechAUTHORS:20140909-152126131

]]>