Abstract: Conventionally, posterior matching is investigated in channel coding and block encoding contexts – the source symbols are equiprobably distributed and are entirely known by the encoder before the transmission. In this paper, we consider a streaming source, whose symbols progressively arrive at the encoder at a sequence of deterministic times. We derive the joint source-channel coding (JSCC) reliability function for streaming over a discrete memoryless channel (DMC) with feedback. We propose a novel instantaneous encoding phase that operates during the symbol arriving period and achieves the JSCC reliability function for streaming when followed by a block encoding scheme that achieves the JSCC reliability function for a classical source whose symbols are fully accessible before the transmission. During the instantaneous encoding phase, the evolving message alphabet is partitioned into groups whose priors are close to the capacity-achieving distribution, and the encoder determines the group index of the actual sequence of symbols arrived so far and applies randomization to exactly match the distribution of the transmitted index to the capacity-achieving one. Surprisingly, the JSCC reliability function for streaming is equal to that for a fully accessible source, implying that the knowledge of the entire symbol sequence before the transmission offers no advantage in terms of the reliability function. For streaming over a symmetric binary-input DMC, we propose a one-phase instantaneous small-enough difference (SED) code that not only achieves the JSCC reliability function, but also, thanks to its single-phase time-invariant coding rule, can be used to stabilize an unstable linear system over a noisy channel. For equiprobably distributed source symbols, we design low complexity algorithms to implement both the instantaneous encoding phase and the instantaneous SED code. The algorithms group the source sequences into sets we call types, which enable the encoder and the decoder to track the priors and the posteriors of source sequences jointly, leading to a log-linear complexity in time. While the reliability function is derived for non-degenerate DMCs, i.e., DMCs whose transition probability matrix has all positive entries, for degenerate DMCs, we design a code with instantaneous encoding that achieves zero error for all rates below Shannon’s joint source-channel coding limit.

Publication: IEEE Transactions on Information Theory Vol.: 69 No.: 4 ISSN: 0018-9448

ID: CaltechAUTHORS:20230502-16744400.1

]]>

Abstract: We consider the basic problem of querying an expert oracle for labeling a dataset in machine learning. This is typically an expensive and time consuming process and therefore, we seek ways to do so efficiently. The conventional approach involves comparing each sample with (the representative of) each class to find a match. In a setting with N equally likely classes, this involves N/2 pairwise comparisons (queries per sample) on average. We consider a k-ary query scheme with k ≥ 2 samples in a query that identifies (dis)similar items in the set while effectively exploiting the associated transitive relations. We present a randomized batch algorithm that operates on a round-by-round basis to label the samples and achieves a query rate of O(N/k²). In addition, we present an adaptive greedy query scheme, which achieves an average rate of ≈0.2N queries per sample with triplet queries. For the proposed algorithms, we investigate the query rate performance analytically and with simulations. Empirical studies suggest that each triplet query takes an expert at most 50% more time compared with a pairwise query, indicating the effectiveness of the proposed k-ary query schemes. We generalize the analyses to nonuniform class distributions when possible.

Publication: IEEE Transactions on Pattern Analysis and Machine Intelligence Vol.: 44 No.: 11 ISSN: 0162-8828

ID: CaltechAUTHORS:20221031-572094900.1

]]>

Abstract: We consider an unstable scalar linear stochastic system, X_(n+1) = aX_n + Z_n − U_n , where a ≥ 1 is the system gain, Z_n's are independent random variables with bounded α-th moments, and U_n's are the control actions that are chosen by a controller who receives a single element of a finite set {1,…,M} as its only information about system state Xi. We show new proofs that M > a is necessary and sufficient for β-moment stability, for any β < α. Our achievable scheme is a uniform quantizer of the zoom-in/zoom-out type that codes over multiple time instants for data rate efficiency; the controller uses its memory of the past to correctly interpret the received bits. We analyze the performance of our scheme using probabilistic arguments. We show a simple proof of a matching converse using information-theoretic techniques. Our results generalize to vector systems, to systems with dependent Gaussian noise, and to the scenario in which a small fraction of transmitted messages is lost.

Publication: IEEE Transactions on Automatic Control Vol.: 67 No.: 10 ISSN: 0018-9286

ID: CaltechAUTHORS:20211217-98215000

]]>

Abstract: Consider the following distributed optimization scenario. A worker has access to training data that it uses to compute the gradients while a server decides when to stop iterative computation based on its target accuracy or delay constraints. The server receives all its information about the problem instance from the worker via a rate-limited noiseless communication channel. We introduce the principle we call differential quantization (DQ) that prescribes compensating the past quantization errors to direct the descent trajectory of a quantized algorithm towards that of its unquantized counterpart. Assuming that the objective function is smooth and strongly convex, we prove that differentially quantized gradient descent (DQ-GD) attains a linear contraction factor of $\max \{\sigma _{\mathrm {GD}}, \rho _{n} 2^{-R}\}$ , where $\sigma _{\mathrm {GD}}$ is the contraction factor of unquantized gradient descent (GD), $\rho _{n} \geq 1$ is the covering efficiency of the quantizer, and $R$ is the bitrate per problem dimension $n$ . Thus at any $R\geq \log _{2} \rho _{n} /\sigma _{\mathrm {GD}}$ bits, the contraction factor of DQ-GD is the same as that of unquantized GD, i.e., there is no loss due to quantization. We show a converse demonstrating that no algorithm within a certain class can converge faster than $\max \{\sigma _{\mathrm {GD}}, 2^{-R}\}$ . Since quantizers exist with $\rho _{n} \to 1$ as $n \to \infty $ (Rogers, 1963), this means that DQ-GD is asymptotically optimal. In contrast, naively quantized GD where the worker directly quantizes the gradient barely attains $\sigma _{\mathrm {GD}} + \rho _{n}2^{-R}$ . The principle of differential quantization continues to apply to gradient methods with momentum such as Nesterov’s accelerated gradient descent, and Polyak’s heavy ball method. For these algorithms as well, if the rate is above a certain threshold, there is no loss in contraction factor obtained by the differentially quantized algorithm compared to its unquantized counterpart, and furthermore, the differentially quantized heavy ball method attains the optimal contraction achievable among all (even unquantized) gradient methods. Experimental results on least-squares problems validate our theoretical analysis.

Publication: IEEE Transactions on Information Theory Vol.: 68 No.: 9 ISSN: 0018-9448

ID: CaltechAUTHORS:20220909-232702000

]]>

Abstract: We consider the feedback capacity of a MIMO channel whose channel output is given by a linear state-space model driven by the channel inputs and a Gaussian process. The generality of our state-space model subsumes all previous studied models such as additive channels with colored Gaussian noise, and channels with an arbitrary dependence on previous channel inputs or outputs. The main result is a computable feedback capacity expression that is given as a convex optimization problem subject to a detectability condition. We demonstrate the capacity result on the auto-regressive Gaussian noise channel, where we show that even a single time-instance delay in the feedback reduces the feedback capacity significantly in the stationary regime. On the other hand, for large regression parameters (in the non-stationary regime), the feedback capacity can be approached with delayed feedback. Finally, we show that the detectability condition is satisfied for scalar models and conjecture that it is true for MIMO models.

Publication: arXiv
ID: CaltechAUTHORS:20221222-234257392

]]>

Abstract: We consider the feedback capacity of a MIMO channel whose channel output is given by a linear state-space model driven by the channel inputs and a Gaussian process. The generality of our state-space model subsumes all previous studied models such as additive channels with colored Gaussian noise, and channels with an arbitrary dependence on previous channel inputs or outputs. The main result is a computable feedback capacity expression that is given as a convex optimization problem subject to a detectability condition. We demonstrate the capacity result on the auto-regressive Gaussian noise channel, where we show that even a single time-instance delay in the feedback reduces the feedback capacity significantly in the stationary regime. On the other hand, for large regression parameters, the feedback capacity can be achieved with delayed feedback. Finally, we show that the detectability condition is satisfied for scalar models and conjecture that it is true for MIMO models.

ID: CaltechAUTHORS:20220804-765679000

]]>

Abstract: Conventionally, posterior matching is investigated in channel coding and block encoding contexts – the source symbols are equiprobably distributed and are entirely known by the encoder before the transmission. In this paper, we consider a streaming source, whose symbols progressively arrive at the encoder at a sequence of deterministic times. We derive the joint source-channel coding (JSCC) reliability function for streaming over a discrete memoryless channel (DMC) with feedback under regularity conditions. We propose a novel instantaneous encoding phase that operates during the symbol arriving period and that achieves the JSCC reliability function for streaming when followed by a block encoding scheme that achieves the JSCC reliability function for a classical source whose symbols are fully accessible before the transmission. The instantaneous encoding phase partitions the evolving message alphabet into groups whose priors are close to the capacity-achieving distribution, and randomizes the group indices to ensure that the transmitted group index has the capacity-achieving distribution. Surprisingly, the JSCC reliability function for streaming is equal to that for a fully accessible source, implying that the knowledge of the entire symbol sequence before the transmission offers no advantage in terms of the reliability function.

ID: CaltechAUTHORS:20220804-765703000

]]>

Abstract: The channel coding problem in the moderate deviations regime is studied; here, the error probability sub-exponentially decays to zero, and the rate approaches the capacity slower than O(1/√n). The main result refines Altuğ and Wagner’s moderate deviations result by deriving lower and upper bounds on the third-order term in the asymptotic expansion of the maximum achievable message set size. The third-order term of the expansion employs a new quantity called the channel skewness. For the binary symmetric channel and most practically important (n,ϵ) pairs, including n ∈ [100, 500] and ϵ ∈ [10⁻¹⁰,10⁻¹], an approximation up to the channel skewness is the most accurate among several expansions in the literature.

ID: CaltechAUTHORS:20220804-765685000

]]>

Abstract: In this paper, we are interested in the performance of a variable-length stop-feedback (VLSF) code with m optimal decoding times for the binary-input additive white Gaussian noise channel. We first develop tight approximations to the tail probability of length-n cumulative information density. Building on the work of Yavas et al., for a given information density threshold, we formulate the integer program of minimizing the upper bound on average blocklength over all decoding times subject to the average error probability, minimum gap and integer constraints. Eventually, minimization of locally optimal upper bounds over all thresholds yields the globally minimum upper bound and the above method is called the two-step minimization. Relaxing to allow positive real-valued decoding times activates the gap constraint. We develop gap-constrained sequential differential optimization (SDO) procedure to find the optimal, gap-constrained, real-valued decoding times. In the error regime of practical interest, Polyanskiy's scheme of stopping at zero does not help. In this region, the achievability bounds estimated by the two-step minimization and gap-constrained SDO show that Polyanskiy’s achievability bound for VLSF codes can be approached with a small number of decoding times.

ID: CaltechAUTHORS:20220804-765691000

]]>

Abstract: This paper focuses on the numerical evaluation of the maximal achievable rate of variable-length stop-feedback (VLSF) codes with m decoding times at a given message size and error probability for binary-input additive white Gaussian noise channel, binary symmetric channel, and binary erasure channel (BEC). Leveraging the Edgeworth and Petrov expansions, we develop tight approximations to the tail probability of length-n cumulative information density that are accurate for any blocklength n. We reduce Yavas et al.'s non-asymptotic achievability bound on VLSF codes with m decoding times to an integer program of minimizing the upper bound on the average blocklength subject to the average error probability, minimum gap, and integer constraints. We develop two distinct methods to solve this program. Numerical evaluations show that Polyanskiy's achievability bound for VLSF codes, which assumes m = ∞, can be approached with a relatively small m in all of the three channels. For BEC, we consider systematic transmission followed by random linear fountain coding. This allows us to obtain a new achievability bound stronger than a previously known bound and new VLSF codes whose rate further outperforms Polyanskiy's bound.

Publication: arXiv
ID: CaltechAUTHORS:20220804-201311712

]]>

Abstract: We consider the following communication scenario. An encoder causally observes the Wiener process and decides when and what to transmit about it. A decoder estimates the process using causally received codewords in real time. We determine the causal encoding and decoding policies that jointly minimize the mean-square estimation error, under the long-term communication rate constraint of R bits per second. We show that an optimal encoding policy can be implemented as a causal sampling policy followed by a causal compressing policy. We prove that the optimal encoding policy samples the Wiener process once the innovation passes either √1/R or −√1/R and compresses the sign of innovation (SOI) using a 1-bit codeword. The SOI coding scheme achieves the operational distortion-rate function, which is equal to D^(op)(R)=1/6R. Surprisingly, this is significantly better than the distortion-rate tradeoff achieved in the limit of infinite delay by the best noncausal code. This is because the SOI coding scheme leverages the free timing information supplied by the zero-delay channel between the encoder and the decoder. The key to unlocking that gain is the event-triggered nature of the SOI sampling policy. In contrast, the distortion-rate tradeoffs achieved with deterministic sampling policies are much worse: we prove that the causal informational distortion-rate function in that scenario is as high as D_(DET)(R)=5/6R. It is achieved by the uniform sampling policy with the sampling interval 1/R. In either case, the optimal strategy is to sample the process as fast as possible and to transmit 1-bit codewords to the decoder without delay. We show that the SOI coding scheme also minimizes the mean-square cost of a continuous-time control system driven by the Wiener process and controlled via rate-constrained impulses.

Publication: IEEE Transactions on Automatic Control Vol.: 67 No.: 4 ISSN: 0018-9286

ID: CaltechAUTHORS:20210412-071230573

]]>

Abstract: Conventionally, posterior matching is investigated in channel coding and block encoding contexts -- the source symbols are equiprobably distributed and are entirely known by the encoder before the transmission. In this paper, we consider a streaming source, whose symbols arrive at the encoder at a sequence of deterministic times. We derive the joint source-channel coding (JSCC) reliability function for streaming over a discrete memoryless channel (DMC) with feedback. We propose a novel instantaneous encoding phase that operates during the symbol arriving period and achieves the JSCC reliability function for streaming when followed by a block encoding scheme that achieves the JSCC reliability function for a classical source whose symbols are fully accessible before the transmission. During the instantaneous encoding phase, the evolving message alphabet is partitioned into groups, and the encoder determines the index of the group that contains the symbols arrived so far and applies randomization to match the distribution of the transmitted index to the capacity-achieving one. Surprisingly, the JSCC reliability function for streaming is equal to that for a fully accessible source, implying that the knowledge of the entire symbol sequence before the transmission offers no advantage regarding the reliability function. For streaming over a symmetric 2-input DMC, we propose an instantaneous small-enough difference (SED) code that not only achieves the JSCC reliability function but also can be used to stabilize an unstable linear system over a noisy channel. We design low complexity algorithms to implement both the instantaneous encoding phase and the instantaneous SED code. While the reliability function is derived for non-degenerate DMCs, for degenerate DMCs we design a code with instantaneous encoding that achieves zero error for all rates below Shannon's JSCC limit.

Publication: arXiv
ID: CaltechAUTHORS:20230504-968774000.1

]]>

Abstract: Consider the following communication scenario. An encoder observes a stochastic process and causally decides when and what to transmit about it, under a constraint on the expected number of bits transmitted per second. A decoder uses the received codewords to causally estimate the process in real time. The encoder and the decoder are synchronized in time. For a class of continuous Markov processes satisfying regularity conditions, we find the optimal encoding and decoding policies that minimize the end-to-end estimation mean-square error under the rate constraint. We show that the optimal encoding policy transmits a 1-bit codeword once the process innovation passes one of two thresholds. The optimal decoder noiselessly recovers the last sample from the 1-bit codewords and codeword-generating time stamps, and uses it to decide the running estimate of the current process, until the next codeword arrives. In particular, we show the optimal causal code for the Ornstein-Uhlenbeck process and calculate its distortion-rate function. Furthermore, we show that the optimal causal code also minimizes the mean-square cost of a continuous-time control system driven by a continuous Markov process and controlled by an additive control signal.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 12 ISSN: 0018-9448

ID: CaltechAUTHORS:20211222-322820000

]]>

Abstract: An n-dimensional source with memory is observed by K isolated encoders via parallel channels, who compress their observations to transmit to the decoder via noiseless rate-constrained links while leveraging their memory of the past. At each time instant, the decoder receives K new codewords from the observers, combines them with the past received codewords, and produces a minimum-distortion estimate of the latest block of n source symbols. This scenario extends the classical one-shot CEO problem to multiple rounds of communication with communicators maintaining the memory of the past. We extend the Berger-Tung inner and outer bounds to the scenario with inter-block memory, showing that the minimum asymptotically (as n→∞) achievable sum rate required to achieve a target distortion is bounded by minimal directed mutual information problems. For the Gauss-Markov source observed via K parallel AWGN channels, we show that the inner bound is tight and solve the corresponding minimal directed mutual information problem, thereby establishing the minimum asymptotically achievable sum rate. Finally, we explicitly bound the rate loss due to a lack of communication among the observers; that bound is attained with equality in the case of identical observation channels. The general coding theorem is proved via a new nonasymptotic bound that uses stochastic likelihood coders and whose asymptotic analysis yields an extension of the Berger-Tung inner bound to the causal setting. The analysis of the Gaussian case is facilitated by reversing the channels of the observers.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 12 ISSN: 0018-9448

ID: CaltechAUTHORS:20211222-495165300

]]>

Abstract: This paper presents finite-blocklength achievability bounds for the Gaussian multiple access channel (MAC) and random access channel (RAC) under average-error and maximal-power constraints. Using random codewords uniformly distributed on a sphere and a maximum likelihood decoder, the derived MAC bound on each transmitter’s rate matches the MolavianJazi-Laneman bound (2015) in its first- and second-order terms, improving the remaining terms to ½ log n/n + O(1/n) bits per channel use. The result then extends to a RAC model in which neither the encoders nor the decoder knows which of K possible transmitters are active. In the proposed rateless coding strategy, decoding occurs at a time n_t that depends on the decoder’s estimate t of the number of active transmitters k. Single-bit feedback from the decoder to all encoders at each potential decoding time n_i, I ≤ t , informs the encoders when to stop transmitting. For this RAC model, the proposed code achieves the same first-, second-, and third-order performance as the best known result for the Gaussian MAC in operation.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 11 ISSN: 0018-9448

ID: CaltechAUTHORS:20211122-171053652

]]>

Abstract: This paper investigates variable-length feedback codes for discrete memoryless channels in point-to-point, multiple access, and random access communication. The proposed nested code employs L decoding times n₁,n₂,…,n_L for the point-to-point and multiple access channels and KL decoding times {n_(k,ℓ):1 ≤ k ≤ K,1 ≤ ℓ ≤ L} for the random access channel with at most K active transmitters; in the latter case, decoding times n_(k,ℓ), 1 ≤ ℓ ≤ L are reserved for decoding in the scenario where the decoder believes that the number of active transmitters is k. The code has a nested structure, i.e., codewords used to decode messages from k active transmitters are prefix of codewords used to decode messages from k+1 active transmitters. The code employs single-bit, scheduled feedback from the receiver to the transmitters at each potential decoding time to inform the transmitters whether or not it is able to decode. Transmitters cease transmission, thereby truncating their codewords, when no further transmissions are required by the decoder. The choice of decoding times is optimized to minimize the expected decoding time subject to an error probability constraint, and second order achievability bounds are derived.

ID: CaltechAUTHORS:20220105-510075300

]]>

Abstract: We consider the basic problem of querying an expert oracle for labeling a dataset in machine learning. This is typically an expensive and time consuming process and therefore, we seek ways to do so efficiently. The conventional approach involves comparing each sample with (the representative of) each class to find a match. In a setting with N equally likely classes, this involves N/2 pairwise comparisons (queries per sample) on average. We consider a k-ary query scheme with k ≥ 2 samples in a query that identifies (dis)similar items in the set while effectively exploiting the associated transitive relations. We present a randomized batch algorithm that operates on a round-by-round basis to label the samples and achieves a query rate of O(N/k²). In addition, we present an adaptive greedy query scheme, which achieves an average rate of ≈0.2N queries per sample with triplet queries. For the proposed algorithms, we investigate the query rate performance analytically and with simulations. Empirical studies suggest that each triplet query takes an expert at most 50% more time compared with a pairwise query, indicating the effectiveness of the proposed k-ary query schemes. We generalize the analyses to nonuniform class distributions when possible.

Publication: arXiv
ID: CaltechAUTHORS:20220804-201317566

]]>

Abstract: We study the linear quadratic Gaussian (LQG) control problem, in which the controller's observation of the system state is such that a desired cost is unattainable. To achieve the desired LQG cost, we introduce a communication link from the observer (encoder) to the controller. We investigate the optimal trade-off between the improved LQG cost and the consumed communication (information) resources, measured with the conditional directed information, across all encoding-decoding policies. The main result is a semidefinite programming formulation for that optimization problem in the finite-horizon scenario, which applies to time-varying linear dynamical systems. This result extends a seminal work by Tanaka et al., where the only information the controller knows about the system state arrives via a communication channel, to the scenario where the controller has also access to a noisy observation of the system state. As part of our derivation to show the optimiality of an encoder that transmits a memoryless Gaussian measurement of the state, we show that the presence of the controller's observations at the encoder can not reduce the minimal directed information. For time-invariant systems, where the optimal policy may be time-varying, we show in the infinite-horizon scenario that the optimal policy is time-invariant and can be computed explicitly from a solution of a finite-dimensional semidefinite programming. The results are demonstrated via examples that show that even low-quality measurements can have a significant impact on the required communication resources.

Publication: arXiv
ID: CaltechAUTHORS:20220804-201321456

]]>

Abstract: A key result of classical information theory states that if the rate of a randomly generated codebook is less than the mutual information between the channel’s input and output, then the probability that that codebook has negligible error goes to one as the blocklength goes to infinity. In an attempt to bridge the gap between the probabilistic world of classical information theory and the combinatorial world of zero-error information theory, this work derives necessary and sufficient conditions on the rate so that the probability that a randomly generated codebook operated under list decoding (for any fixed list size) has zero error probability goes to one as the blocklength goes to infinity. Furthermore, this work extends the classical birthday problem to an information-theoretic setting, which results in the definition of a “noisy” counterpart of Rényi entropy, analogous to how mutual information can be considered a noisy counterpart of Shannon entropy.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 9 ISSN: 0018-9448

ID: CaltechAUTHORS:20210825-143502652

]]>

Abstract: Consider the following distributed optimization scenario. A worker has access to training data that it uses to compute the gradients while a server decides when to stop iterative computation based on its target accuracy or delay constraints. The only information that the server knows about the problem instance is what it receives from the worker via a rate-limited noiseless communication channel. We introduce the technique we call differential quantization (DQ) that compensates past quantization errors to make the descent trajectory of a quantized algorithm follow that of its unquantized counterpart. Assuming that the objective function is smooth and strongly convex, we prove that differentially quantized gradient descent (DQ-GD) attains a linear convergence rate of max{σ_(GD), ρ_n2^(-R)}, where σ_(GD) is the convergence rate of unquantized gradient descent (GD), ρ_n is the covering efficiency of the quantizer, and R is the bitrate per problem dimension n. Thus at any R ≥ log₂ρ_n/σ_(GD), the convergence rate of DQ-GD is the same as that of unquantized GD, i.e., there is no loss due to quantization. We show a converse demonstrating that no GD-like quantized algorithm can converge faster than max{σ_(GD), 2^(-R)}. Since quantizers exist with ρ_n → 1 as n → ∞ (Rogers, 1963), this means that DQ-GD is asymptotically optimal. In contrast, naively quantized GD where the worker directly quantizes the gradient attains only σ_(GD) + ρ_n2^(-R). The technique of differential quantization continues to apply to gradient methods with momentum such as Nesterov's accelerated gradient descent, and Polyak's heavy ball method. For these algorithms as well, if the rate is above a certain threshold, there is no loss in convergence rate obtained by the differentially quantized algorithm compared to its unquantized counterpart. Experimental results on both simulated and realworld least-squares problems validate our theoretical analysis.

Publication: arXiv
ID: CaltechAUTHORS:20200214-105624458

]]>

Abstract: Finding a computable expression for the feedback capacity of channels with non-white Gaussian, additive noise is a long standing open problem. In this paper, we solve this problem in the scenario where the channel has multiple inputs and multiple outputs (MIMO) and the noise process is generated as the output of a state-space model (a hidden Markov model). The main result is a computable characterization of the feedback capacity as a finite-dimensional convex optimization problem. Our solution subsumes all previous solutions to the feedback capacity including the auto-regressive moving-average (ARMA) noise process of first order, even if it is a non-stationary process. The capacity problem can be viewed as the problem of maximizing the measurements' entropy rate of a controlled (policy-dependent) state-space subject to a power constraint. We formulate the finite-block version of this problem as a sequential convex optimization problem, which in turn leads to a single-letter and computable upper bound. By optimizing over a family of time-invariant policies that correspond to the channel inputs distribution, a tight lower bound is realized. We show that one of the optimization constraints in the capacity characterization boils down to a Riccati equation, revealing an interesting relation between explicit capacity formulae and Riccati equations.

Publication: arXiv
ID: CaltechAUTHORS:20210719-210210078

]]>

Abstract: In this paper, we propose a novel code for transmitting a sequence of n message bits in real time over a discrete-memoryless channel (DMC) with noiseless feedback, where the message bits stream into the encoder one by one at random time instants. Similar to existing posterior matching schemes with block encoding, the encoder in our work takes advantage of the channel feedback to form channel inputs that contain the information the decoder does not yet have, and that are distributed close to the capacity-achieving input distribution, but dissimilar to the existing posterior matching schemes, the encoder performs instantaneous encoding - it immediately weaves the new message bits into a continuing transmission. A posterior matching scheme by Naghshvar et al. partitions the source messages into groups so that the group posteriors have a small-enough difference (SED) to the capacity-achieving distribution, and transmits the group index that contains the actual message. Our code adopts the SED rule to apply to the evolving message alphabet that contains all the possible variable-length strings that the source could have emitted up to that time. Our instantaneous SED code achieves better delay-reliability tradeoffs than existing feedback codes over 2-input DMCs: we establish this dominance both by simulations and via an analysis comparing the performance of the instantaneous SED code to Burnashev’s reliability function. Due to the message alphabet that grows exponentially with time t, the complexity of the instantaneous SED code is double-exponential in t. To overcome this complexity barrier to practical implementation, we design a low-complexity code for binary symmetric channels that we name the instantaneous type set SED code. It groups the message strings into sets we call type sets and tracks their prior and posterior probabilities jointly, resulting in the reduction of complexity from double-exponential to O(t⁴). Simulation results show that the gap in performance between the instantaneous SED code and the instantaneous type-set SED code is negligible.

Publication: 2021 IEEE International Symposium on Information Theory
ID: CaltechAUTHORS:20210323-145545445

]]>

Abstract: We investigate variable-length feedback (VLF) codes for the Gaussian point-to-point channel under maximal power, average error probability, and average decoding time constraints. Our proposed strategy chooses K < ∞ decoding times n₁,n₂,…,n_K rather than allowing decoding at any time n = 0,1,2,…. We consider stop-feedback, which is one-bit feedback transmitted from the receiver to the transmitter at times n₁,n₂,… only to inform her whether to stop. We prove an achievability bound for VLF codes with the asymptotic approximation ln M ≈ NC(P)/1−ϵ − √N ln_((K−1))(N)V(P)/1−ϵ, where ln_((K))(⋅) denotes the K-fold nested logarithm function, N is the average decoding time, and C(P) and V(P) are the capacity and dispersion of the Gaussian channel, respectively. Our achievability bound evaluates a non-asymptotic bound and optimizes the decoding times n₁,…,n_K within our code architecture.

Publication: arXiv
ID: CaltechAUTHORS:20210329-152031630

]]>

Abstract: This paper provides a precise error analysis for the maximum likelihood estimate â_(ML)(uⁿ₁) of the parameter a given samples uⁿ₁ = (u₁, ... , u_n)ʹ drawn from a nonstationary Gauss-Markov process U_i = aU_(i-1) + Z)_i, i ≥ 1, where U₀ = 0, a > 1, and Zi ’s are independent Gaussian random variables with zero mean and variance σ². We show a tight nonasymptotic exponentially decaying bound on the tail probability of the estimation error. Unlike previous works, our bound is tight already for a sample size of the order of hundreds. We apply the new estimation bound to find the dispersion for lossy compression of nonstationary Gauss-Markov sources. We show that the dispersion is given by the same integral formula that we derived previously for the asymptotically stationary Gauss-Markov sources, i.e., |a|<1 . New ideas in the nonstationary case include separately bounding the maximum eigenvalue (which scales exponentially) and the other eigenvalues (which are bounded by constants that depend only on a) of the covariance matrix of the source sequence, and new techniques in the derivation of our estimation error bound.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 4 ISSN: 0018-9448

ID: CaltechAUTHORS:20210211-151615781

]]>

Abstract: Consider a random access communication scenario over a channel whose operation is defined for any number of possible transmitters. As in the model recently introduced by Polyanskiy for the Multiple Access Channel (MAC) with a fixed, known number of transmitters, the channel is assumed to be invariant to permutations on its inputs, and all active transmitters employ identical encoders. Unlike the Polyanskiy model, in the proposed scenario, neither the transmitters nor the receiver knows which transmitters are active. We refer to this agnostic communication setup as the Random Access Channel (RAC). Scheduled feedback of a finite number of bits is used to synchronize the transmitters. The decoder is tasked with determining from the channel output the number of active transmitters, k, and their messages but not which transmitter sent which message. The decoding procedure occurs at a time n_t depending on the decoder’s estimate, t, of the number of active transmitters, k, thereby achieving a rate that varies with the number of active transmitters. Single-bit feedback at each time n_(i,i) ≤ t , enables all transmitters to determine the end of one coding epoch and the start of the next. The central result of this work demonstrates the achievability on a RAC of performance that is first-order optimal for the MAC in operation during each coding epoch. While prior multiple access schemes for a fixed number of transmitters require 2^k−1 simultaneous threshold rules, the proposed scheme uses a single threshold rule and achieves the same dispersion.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 4 ISSN: 0018-9448

ID: CaltechAUTHORS:20210113-163504947

]]>

Abstract: We study the stabilization of a linear control system with an unbounded random system gain where the controller must act based on a rate-limited observation of the state. More precisely, we consider the system X_(n+1)=A_nX_n+W_n−U_n, where the A_n’s are drawn independently at random at each time n from a known distribution with unbounded support, and where the controller receives at most R bits about the system state at each time from an encoder. We provide a time-varying achievable strategy to stabilize the system in a second-moment sense with fixed, finite R. While our previous result provided a strategy to stabilize this system using a variable-rate code, this work provides an achievable strategy using a fixed-rate code. The strategy we employ to achieve this is time-varying and takes different actions depending on the value of the state. It proceeds in two modes: a normal mode (or zoom-in), where the realization of A_n is typical, and an emergency mode (or zoom-out), where the realization of A_n is exceptionally large. To analyze the performance of the scheme we construct an auxiliary sequence that bounds the state X_n, and then bound auxiliary sequence in both the zoom-in and zoom-out modes.

Publication: IEEE Transactions on Information Theory Vol.: 67 No.: 4 ISSN: 0018-9448

ID: CaltechAUTHORS:20210205-093044649

]]>

Abstract: We study a linear quadratic Gaussian (LQG) control problem, in which a noisy observation of the system state is available to the controller. To lower the achievable LQG cost, we introduce an extra communication link from the system to the controller. We investigate the trade-off between the improved LQG cost and the consumed communication (information) resources that are measured with the conditional directed information. The objective is to minimize the directed information over all encoding-decoding policies subject to a constraint on the LQG cost. The main result is a semidefinite programming formulation for the optimization problem in the finite-horizion scenario where the dynamical system may have time-varying parameters. This result extends the seminal work by Tanaka et al., where the direct noisy measurement of the system state at the controller is assumed to be absent. As part of our derivation to show the optimality of an encoder that transmits a Gaussian measurement of the state, we show that the presence of the noisy measurements at the encoder can not reduce the minimal directed information, extending a prior result of Kostina and Hassibi to the vector case. Finally, we show that the results in the finite-horizon case can be extended to the infinite-horizon scenario when assuming a time-invariant system, but possibly a time-varying policy. We show that the solution for this optimization problem can be realized by a time-invariant policy whose parameters can be computed explicitly from a finite-dimensional semidefinite program.

ID: CaltechAUTHORS:20210121-152557490

]]>

Abstract: We propose a two-layer coding architecture for communication of multiple users over a shared slotted medium enabling joint collision resolution and decoding. Each user first encodes its information bits with an outer code for reliability, and then transmits these coded bits with possible repetitions over transmission time slots of the access channel. The transmission patterns are dictated by the inner collision-resolution code and collisions with other users’ transmissions may occur. We analyze two types of codes for the outer layer: long-blocklength LDPC codes, and short-blocklength algebraic codes. With LDPC codes, a density evolution analysis enables joint optimization of both outer and inner code parameters for maximum throughput. With algebraic codes, we invoke a similar analysis by approximating their average erasure correcting capability while assuming a large number of active transmitters. The proposed low-complexity schemes operate at a significantly smaller gap to capacity than the state of the art. Our schemes apply both to a multiple access scenario where the number of users within a frame is known a priori, and to a random access scenario where that number is known only to the decoder. In the latter case, we optimize an outage probability due to the variability in user activity.

Publication: IEEE Transactions on Wireless Communications Vol.: 19 No.: 12 ISSN: 1536-1276

ID: CaltechAUTHORS:20191004-135310393

]]>

Abstract: This work studies point-to-point, multiple access, and random access lossless source coding in the finite-blocklength regime. In each scenario, a random coding technique is developed and used to analyze third-order coding performance. Asymptotic results include a third-order characterization of the Slepian-Wolf rate region with an improved converse that relies on a connection to composite hypothesis testing. For dependent sources, the result implies that the independent encoders used by Slepian-Wolf codes can achieve the same third-order-optimal performance as a single joint encoder. The concept of random access source coding is introduced to generalize multiple access (Slepian-Wolf) source coding to the case where encoders decide independently whether or not to participate and the set of participating encoders is unknown a priori to both the encoders and the decoder. The proposed random access source coding strategy employs rateless coding with scheduled feedback. A random coding argument proves the existence of a single deterministic code of this structure that simultaneously achieves the third-order-optimal Slepian-Wolf performance for each possible active encoder set.

Publication: IEEE Transactions on Information Theory Vol.: 66 No.: 11 ISSN: 0018-9448

ID: CaltechAUTHORS:20200723-131543186

]]>

Abstract: Consider the following communication scenario. An n-dimensional source with memory is observed by K isolated encoders via parallel channels, who causally compress their observations to transmit to the decoder via noiseless rate-constrained links. At each time instant, the decoder receives K new codewords from the observers, combines them with the past received codewords, and produces a minimum- distortion estimate of the latest block of n source symbols. This scenario extends the classical one-shot CEO problem to multiple rounds of communication with communicators maintaining memory of the past.We prove a coding theorem showing that the minimum asymptotically (as n → ∞) achievable sum rate required to achieve a target distortion is equal to the directed mutual information from the observers to the decoder minimized subject to the distortion constraint and the separate encoding constraint. For the Gauss-Markov source observed via K parallel AWGN channels, we solve that minimal directed mutual information problem, thereby establishing the minimum asymptotically achievable sum rate. Finally, we explicitly bound the rate loss due to a lack of communication among the observers; that bound is attained with equality in the case of identical observation channels.The general coding theorem is proved via a new nonasymptotic bound that uses stochastic likelihood coders and whose asymptotic analysis yields an extension of the Berger-Tung inner bound to the causal setting. The analysis of the Gaussian case is facilitated by reversing the channels of the observers.

Publication: arXiv
ID: CaltechAUTHORS:20191213-102421150

]]>

Abstract: This paper presents finite-blocklength achievabil- ity bounds for the Gaussian multiple access channel (MAC) and random access channel (RAC) under average-error and maximal-power constraints. Using random codewords uniformly distributed on a sphere and a maximum likelihood decoder, the derived MAC bound on each transmitter’s rate matches the MolavianJazi-Laneman bound (2015) in its first- and second-order terms, improving the remaining terms to ½ log n/n + O(1/n) bits per channel use. The result then extends to a RAC model in which neither the encoders nor the decoder knows which of K possible transmitters are active. In the proposed rateless coding strategy, decoding occurs at a time n t that depends on the decoder’s estimate t of the number of active transmitters k. Single-bit feedback from the decoder to all encoders at each potential decoding time n_i, i ≤ t, informs the encoders when to stop transmitting. For this RAC model, the proposed code achieves the same first-, second-, and third-order performance as the best known result for the Gaussian MAC in operation.

Publication: arXiv
ID: CaltechAUTHORS:20200214-105613893

]]>

Abstract: Consider the following communication scenario. An encoder observes a stochastic process and causally decides when and what to transmit about it, under a constraint on bits transmitted per second. A decoder uses the received codewords to causally estimate the process in real time. The encoder and the decoder are synchronized in time. We aim to find the optimal encoding and decoding policies that minimize the end-to-end estimation mean-square error under the rate constraint. For a class of continuous Markov processes satisfying regularity conditions, we show that the optimal encoding policy transmits a 1-bit codeword once the process innovation passes one of two thresholds. The optimal decoder noiselessly recovers the last sample from the 1-bit codewords and codeword-generating time stamps, and uses it as the running estimate of the current process, until the next codeword arrives. In particular, we show the optimal causal code for the Ornstein-Uhlenbeck process and calculate its distortion-rate function.

Publication: arXiv
ID: CaltechAUTHORS:20200214-105617315

]]>

Abstract: The stabilization of unstable dynamical systems using rate-limited feedback links is investigated. In the scenario of a constant-rate link and a noise with unbounded support, the fundamental limit of communication is known, but no simple algorithm to achieve it exists. The main challenge in constructing an optimal scheme is to fully exploit the communication resources while occasionally signaling the controller that a special operation needs to be taken due to a large noise observation. In this work, we present a simple and explicit algorithm that stabilizes the dynamical system and achieves the fundamental limits of communication. The new idea is to use a constrained quantizer in which certain patterns of sequences are avoided throughout the quantization process. These patterns are preserved to signal the controller that a zoom-out operation should be initiated due to large noise observation. We show that the constrained quantizer has a negligible effect on the rate, so it achieves the fundamental limit of communication. Specifically, the rate-optimal algorithm is shown to stabilize any β-moment of the state if the noise has a bounded absolute (β +ϵ)-moment for some ϵ > 0 regardless of the other noise characteristics.

ID: CaltechAUTHORS:20200831-150243516

]]>

Abstract: Consider a control problem with a communication channel connecting the observer of a linear stochastic system to the controller. The goal of the controller is to minimize a quadratic cost function in the state variables and control signal, known as the linear quadratic regulator (LQR). We study the fundamental tradeoff between the communication rate r bits/sec and the expected cost b. We obtain a lower bound on a certain rate-cost function, which quantifies the minimum directed mutual information between the channel input and output that is compatible with a target LQR cost. The rate-cost function has operational significance in multiple scenarios of interest: among others, it allows us to lower-bound the minimum communication rate for fixed and variable length quantization, and for control over noisy channels. We derive an explicit lower bound to the rate-cost function, which applies to the vector, non-Gaussian, and partially observed systems, thereby extending and generalizing an earlier explicit expression for the scalar Gaussian system, due to Tatikonda el al. [2]. The bound applies as long as the differential entropy of the system noise is not −∞ . It can be closely approached by a simple lattice quantization scheme that only quantizes the innovation, that is, the difference between the controller's belief about the current state and the true state. Via a separation principle between control and communication, similar results hold for causal lossy compression of additive noise Markov sources. Apart from standard dynamic programming arguments, our technical approach leverages the Shannon lower bound, develops new estimates for data compression with coding memory, and uses some recent results on high resolution variablelength vector quantization to prove that the new converse bounds are tight.

Publication: IEEE Transactions on Automatic Control Vol.: 64 No.: 11 ISSN: 0018-9286

ID: CaltechAUTHORS:20190425-103602105

]]>

Abstract: In successive refinement of information, the decoder refines its representation of the source progressively as it receives more encoded bits. The rate-distortion region of successive refinement describes the minimum rates required to attain the target distortions at each decoding stage. In this paper, we derive a parametric characterization of the rate-distortion region for successive refinement of abstract sources. Our characterization extends Csiszár’s result to successive refinement, and generalizes a result by Tuncel and Rose, applicable for finite alphabet sources, to abstract sources. This characterization spawns a family of outer bounds to the rate-distortion region. It also enables an iterative algorithm for computing the rate-distortion region, which generalizes Blahut’s algorithm to successive refinement. Finally, it leads a new nonasymptotic converse bound. In all the scenarios where the dispersion is known, this bound is second-order optimal. In our proof technique, we avoid Karush–Kuhn–Tucker conditions of optimality, and we use basic tools of probability theory. We leverage the Donsker–Varadhan lemma for the minimization of relative entropy on abstract probability spaces.

Publication: IEEE Transactions on Information Theory Vol.: 65 No.: 10 ISSN: 0018-9448

ID: CaltechAUTHORS:20190613-142206819

]]>

Abstract: The Gauss-Markov source produces U_i = aU_(i–1) + Z_i for i ≥ 1, where U_0 = 0, |a| < 1 and Z_i ~ N(0; σ^2) are i.i.d. Gaussian random variables. We consider lossy compression of a block of n samples of the Gauss-Markov source under squared error distortion. We obtain the Gaussian approximation for the Gauss-Markov source with excess-distortion criterion for any distortion d > 0, and we show that the dispersion has a reverse waterfilling representation. This is the first finite blocklength result for lossy compression of sources with memory. We prove that the finite blocklength rate-distortion function R(n; d; ε) approaches the rate-distortion function R(d) as R(n; d; ε) = R(d)+ √ V(d)/n Q–1(ε)+o(1√n), where V (d) is the dispersion, ε ε 2 (0; 1) is the excess-distortion probability, and Q^(-1) is the inverse Q-function. We give a reverse waterfilling integral representation for the dispersion V (d), which parallels that of the rate-distortion functions for Gaussian processes. Remarkably, for all 0 < d ≥ σ^2 (1+|σ|)^2, R(n; d; ε) of the Gauss-Markov source coincides with that of Z_i, the i.i.d. Gaussian noise driving the process, up to the second-order term. Among novel technical tools developed in this paper is a sharp approximation of the eigenvalues of the covariance matrix of n samples of the Gauss-Markov source, and a construction of a typical set using the maximum likelihood estimate of the parameter a based on n observations.

Publication: IEEE Transactions on Information Theory Vol.: 65 No.: 10 ISSN: 0018-9448

ID: CaltechAUTHORS:20190610-093125985

]]>

Abstract: We consider the problem of controlling an unstable linear plant with Gaussian disturbances over an additive white Gaussian noise channel with an average transmit power constraint, where the signaling rate of communication may be different from the sampling rate of the underlying plant. Such a situation is quite common since sampling is done at a rate that captures the dynamics of the plant and that is often lower than the signaling rate of the communication channel. This rate mismatch offers the opportunity of improving the system performance by using coding over multiple channel uses to convey a single control action. In a traditional, separation-based approach to source and channel coding, the analog message is first quantized down to a few bits and then mapped to a channel codeword whose length is commensurate with the number of channel uses per sampled message. Applying the separation-based approach to control meets its challenges: first, the quantizer needs to be capable of zooming in and out to be able to track unbounded system disturbances, and second, the channel code must be capable of improving its estimates of the past transmissions exponentially with time, a characteristic known as anytime reliability. We implement a separated scheme by leveraging recently developed techniques for control over quantized-feedback channels and for efficient decoding of anytime-reliable codes. We further propose an alternative, namely, to perform analog joint source–channel coding, by this avoiding the digital domain altogether. For the case where the communication signaling rate is twice the sampling rate, we employ analog linear repetition as well as Shannon–Kotel’nikov maps to show a significant improvement in stability margins and linear-quadratic costs over separation-based schemes. We conclude that such analog coding performs better than separation, and can stabilize all moments as well as guarantee almost-sure stability.

Publication: IEEE Transactions on Automatic Control Vol.: 64 No.: 9 ISSN: 0018-9286

ID: CaltechAUTHORS:20190425-110709737

]]>

Abstract: We consider the following communication scenario. An encoder causally observes the Wiener process and decides when and what to transmit about it. A decoder makes real-time estimation of the process using causally received codewords. We determine the causal encoding and decoding policies that jointly minimize the mean-square estimation error, under the long-term communication rate constraint of R bits per second. We show that an optimal encoding policy can be implemented as a causal sampling policy followed by a causal compressing policy. We prove that the optimal encoding policy samples the Wiener process once the innovation passes either √(1/R) or −√(1/R), and compresses the sign of the innovation (SOI) using a 1-bit codeword. The SOI coding scheme achieves the operational distortion-rate function, which is equal to D^(op)(R)=1/(6R). Surprisingly, this is significantly better than the distortion-rate tradeoff achieved in the limit of infinite delay by the best non-causal code. This is because the SOI coding scheme leverages the free timing information supplied by the zero-delay channel between the encoder and the decoder. The key to unlock that gain is the event-triggered nature of the SOI sampling policy. In contrast, the distortion-rate tradeoffs achieved with deterministic sampling policies are much worse: we prove that the causal informational distortion-rate function in that scenario is as high as D_(DET)(R)=5/(6R). It is achieved by the uniform sampling policy with the sampling interval 1/R. In either case, the optimal strategy is to sample the process as fast as possible and to transmit 1-bit codewords to the decoder without delay.

Publication: arXiv
ID: CaltechAUTHORS:20191004-133629184

]]>

Abstract: We present a characterization of the Gaussian CEO rate region, in which the operational point at each boundary is characterized by one free parameter. That parameter determines the water level. Only those sensors whose observation noise is below that water level need to compress and transmit their data. Using that characterization, we present a simple (suboptimal) achievable region, expressed in terms of the difference between the noisy and the noiseless rate-distortion functions. Using that achievable region, we can explicitly bound the rate loss due to lack of cooperation among the compressors.

ID: CaltechAUTHORS:20200214-101154368

]]>

Abstract: This paper provides a precise error analysis for the maximum likelihood estimate â (u) of the parameter a given samples u = (u 1 , … , u n )^⊤ drawn from a nonstationary Gauss-Markov process U i = aU i−1 + Z i , i ≥ 1, where a > 1, U 0 = 0, and Z i ’s are independent Gaussian random variables with zero mean and variance σ^2 . We show a tight nonasymptotic exponentially decaying bound on the tail probability of the estimation error. Unlike previous works, our bound is tight already for a sample size of the order of hundreds. We apply the new estimation bound to find the dispersion for lossy compression of nonstationary Gauss-Markov sources. We show that the dispersion is given by the same integral formula derived in our previous work [1] for the (asymptotically) stationary Gauss-Markov sources, i.e., |a| < 1. New ideas in the nonstationary case include a deeper understanding of the scaling of the maximum eigenvalue of the covariance matrix of the source sequence, and new techniques in the derivation of our estimation error bound.

ID: CaltechAUTHORS:20191004-100332992

]]>

Abstract: This paper treats point-to-point, multiple access and random access lossless source coding in the finite-blocklength regime. A random coding technique is developed, and its power in analyzing the third-order coding performance is demonstrated in all three scenarios. Results include a third-order-optimal characterization of the Slepian-Wolf rate region and a proof showing that for dependent sources, the independent encoders used by Slepian-Wolf codes can achieve the same third-order- optimal performance as a single joint encoder. The concept of random access source coding, which generalizes the multiple access scenario to allow for a subset of participating encoders that is unknown a priori to both the encoders and the decoder, is introduced. Contributions include a new definition of the probabilistic model for a random access-discrete multiple source, a general random access source coding scheme that employs a rateless code with sporadic feedback, and an analysis demonstrating via a random coding argument that there exists a deterministic code of the proposed structure that simultaneously achieves the third- order-optimal performance of Slepian-Wolf codes for all possible subsets of encoders.

Publication: arXiv
ID: CaltechAUTHORS:20190402-132221601

]]>

Abstract: We consider the problem of communications over the binary symmetric channel with feedback, where the information sequence is made available in a causal, possibly random, fashion. We develop a real-time variant of the renowned Horstein scheme and provide analytical guarantees for its error-probability exponential decay rate. We further use the scheme to stabilize an unstable control plant over a binary symmetric channel and compare the analytical guarantees with its empirical performance as well as with those of anytime-reliable codes.

ID: CaltechAUTHORS:20191004-100333333

]]>

Abstract: We consider the problem of tracking the state of Gauss–Markov processes over rate-limited erasure-prone links. We concentrate first on the scenario in which several independent processes are seen by a single observer. The observer maps the processes into finite-rate packets that are sent over the erasure-prone links to a state estimator, and are acknowledged upon packet arrivals. The aim of the state estimator is to track the processes with zero delay and with minimum mean square error (MMSE). We show that, in the limit of many processes, greedy quantization with respect to the squared error distortion is optimal. That is, there is no tension between optimizing the MMSE of the process in the current time instant and that of future times. For the case of packet erasures with delayed acknowledgments, we connect the problem to that of compression with side information that is known at the observer and may be known at the state estimator—where the most recent packets serve as side information that may have been erased, and demonstrate that the loss due to a delay by one time unit is rather small. For the scenario where only one process is tracked by the observer–state estimator system, we further show that variable-length coding techniques are within a small gap of the many-process outer bound. We demonstrate the usefulness of the proposed approach for the simple setting of discrete-time scalar linear quadratic Gaussian control with a limited data-rate feedback that is susceptible to packet erasures.

Publication: IEEE Transactions on Control of Network Systems Vol.: 6 No.: 2 ISSN: 2325-5870

ID: CaltechAUTHORS:20180629-113827049

]]>

Abstract: We consider an unstable scalar linear stochastic system, X_(n + 1) = aX_n + Z_n – U_n.; where a ≥ 1 is the system gain, Z_n's are independent random variables with bounded α-th moments, and U_n'S are the control actions that are chosen by a controller who receives a single element of a finite set {1, …, M} as its only information about system state X_i. We show that M = [a] + 1 is necessary and sufficient for ß- moment stability, for any ß < a. Our achievable scheme is a uniform quantizer of the zoom-in / zoom-out type. We analyze its performance using probabilistic arguments. We prove a matching converse using information-theoretic techniques. Our results generalize to vector systems, to systems with dependent Gaussian noise, and to the scenario in which a small fraction of transmitted messages is lost.

ID: CaltechAUTHORS:20190204-124407625

]]>

Abstract: Consider a control problem in which a remote controller chooses its control action based on two kinds of information about the system state: the information it receives from the system via a rate-constrained feedback link, and side information - a noisy measurement of the system state it observes directly. The goal of the controller is to minimize a quadratic cost function in the state variables and control signal, known as the linear quadratic regulator (LQR). We study the fundamental tradeoff between the communication rate, the expected cost b and the quality of side information. Due to a separation principle between estimation and control, we focus on the tracking problem, where the goal is to track the system state rather than to control it. We introduce the causal rate-distortion function with side information at the decoder. It is expressed in terms of directed mutual information, and it extends the classical (noncausal) Wyner-Ziv rate-distortion function to real-time tracking problems with causality constraints and memory of the past at both encoder and decoder. We compute that function in the scalar linear Gaussian setting; we draw a link with the Kalman filter; we show that making side information available also at the encoder does not help to improve the optimal tradeoffs.

ID: CaltechAUTHORS:20190329-091228468

]]>

Abstract: Understanding how a biomolecular system achieves various control objectives via chemical reactions is of crucial importance in cell biology. However, unlike typical control problems where full information about the system is assumed to be known, typically, only a small portion of the entire biomolecular system can be characterized with certainty. In order to gain insights in these situations, we use control and information theory to derive the performance bounds when chemical species implement feedback control via the production rate or the degradation rate of chemical species. We expand the approach of the pioneering work of Lestas et al. to treat more general scenarios and derive explicit lower bounds on the achievable Fano factor of the controlled species. Our results suggest that control and sensing via the degradation rates, compared with those via the production rates, benefit from the additional design freedom to choose degradation efficiencies, in addition to previously considered signal rate, which helps to lower the Fano factor of the controlled species. We compare our lower bounds with achievable performance via simulation of chemical master equations.

ID: CaltechAUTHORS:20190205-081829763

]]>

Abstract: The entropy power inequality (EPI) has a fundamental role in Information Theory, and has deep connections with famous geometric inequalities. In particular, it is often compared to the Brunn-Minkowski inequality in convex geometry. In this article, we further strengthen the relationships between the EPI and geometric inequalities. Specifically, we establish an equivalence between a strong form of reverse EPI and the hyperplane conjecture, which is a long-standing conjecture in high-dimensional convex geometry. We also provide a simple proof of the hyperplane conjecture for a certain class of distributions, as a straightforward consequence of the EPI.

ID: CaltechAUTHORS:20181126-145018270

]]>

Abstract: Consider a random access communication scenario over a channel whose operation is defined for any number of possible transmitters. Inspired by the model recently introduced for the Multiple Access Channel (MAC) with a fixed, known number of transmitters by Polyanskiy, we assume that the channel is invariant to permutations on its inputs, and that all active transmitters employ identical encoders. Unlike Polyanskiy, we consider a scenario in which neither the transmitters nor the receiver know which or how many transmitters are active. We refer to this agnostic communication setup as the Random Access Channel, or RAC. Limited feedback is used to ensure that the collection of active transmitters remains fixed during each epoch. The decoder is tasked with determining from the channel output the number of active transmitters (k) and their messages but not which transmitter sent which message. The central result of this work demonstrates the achievability on a RAC of performance that is first-order optimal for the MAC in operation during each coding epoch. While prior multiple access schemes for a fixed number of transmitters require 2^k - 1 simultaneous threshold rules, the proposed scheme uses a single threshold rule and achieves the same dispersion.

ID: CaltechAUTHORS:20181126-150049870

]]>

Abstract: We study the stabilization of an unpredictable linear control system where the controller must act based on a rate-limited observation of the state. More precisely, we consider the system X_(n+1) = A_n X_n +W_n –U_n, where the A_n's are drawn independently at random at each time n from a known distribution with unbounded support, and where the controller receives at most R bits about the system state at each time from an encoder. We provide a time-varying achievable strategy to stabilize the system in a second-moment sense with fixed, finite R. While our previous result provided a strategy to stabilize this system using a variable-rate code, this work provides an achievable strategy using a fixed-rate code. The strategy we employ to achieve this is time-varying and takes different actions depending on the value of the state. It proceeds in two modes: a normal mode (or zoom-in), where the realization of A_n is typical, and an emergency mode (or zoom-out), where the realization of A_n is exceptionally large.

ID: CaltechAUTHORS:20181126-144509160

]]>

Abstract: The Gauss-Markov source produces U_i=aU_(i-1)+ Z_i for i ≥ 1, where U_0 = 0, |a| < 1 and Z_i ~ N(0, σ^2) are i.i.d. Gaussian random variables. We consider lossy compression of a block of n samples of the Gauss-Markov source under squared error distortion. We obtain the Gaussian approximation for the Gauss-Markov source with excess-distortion criterion for any distortion d > 0, and we show that the dispersion has a reverse waterfilling representation. This is the first finite blocklength result for lossy compression of sources with memory. We prove that the finite blocklength rate-distortion function R(n, d, ε) approaches the rate-distortion function R(d) as R(n, d, ε) = R(d)+√{[V(d)/n]}Q^(-1)(ε)+o([1/(√n)]), where V(d) is the dispersion, ε ∈ (0,1) is the excess-distortion probability, and Q^(-1) is the inverse of the Q-function. We give a reverse waterfilling integral representation for the dispersion V (d), which parallels that of the rate-distortion functions for Gaussian processes. Remarkably, for all 0 <; d ≤ σ2/(1+|a|)^2 ,R(n, d, c) of the Gauss-Markov source coincides with that of Zi, the i.i.d. Gaussian noise driving the process, up to the second-order term. Among novel technical tools developed in this paper is a sharp approximation of the eigenvalues of the covariance matrix of n samples of the Gauss-Markov source, and a construction of a typical set using the maximum likelihood estimate of the parameter a based on n observations.

ID: CaltechAUTHORS:20181126-141849980

]]>

Abstract: We study the stabilization of an unpredictable linear control system where the controller must act based on a rate-limited observation of the state. More precisely, we consider the system X_(n+1) = A_nX_n+W_n−U_n, where the A_n's are drawn independently at random at each time n from a known distribution with unbounded support, and where the controller receives at most R bits about the system state at each time from an encoder. We provide a time-varying achievable strategy to stabilize the system in a second-moment sense with fixed, finite R. While our previous result provided a strategy to stabilize this system using a variable-rate code, this work provides an achievable strategy using a fixed-rate code. The strategy we employ to achieve this is time-varying and takes different actions depending on the value of the state. It proceeds in two modes: a normal mode (or zoom-in), where the realization of A_n is typical, and an emergency mode (or zoom-out), where the realization of A_n is exceptionally large.

Publication: arXiv
ID: CaltechAUTHORS:20191004-141927403

]]>

Abstract: We derive a lower bound on the differential entropy of a log-concave random variable X in terms of the p-th absolute moment of X. The new bound leads to a reverse entropy power inequality with an explicit constant, and to new bounds on the rate-distortion function and the channel capacity. Specifically, we study the rate-distortion function for log-concave sources and distortion measure d(x,x^)=|x−x^|r , with r ≥ 1 , and we establish that the difference between the rate-distortion function and the Shannon lower bound is at most log(√(πe)) ≈ 1.5 bits, independently of r and the target distortion d. For mean-square error distortion, the difference is at most log(√((πe)/2)) ≈ 1 bit, regardless of d. We also provide bounds on the capacity of memoryless additive noise channels when the noise is log-concave. We show that the difference between the capacity of such channels and the capacity of the Gaussian channel with the same noise power is at most log(√((πe)/2)) ≈ 1 bit. Our results generalize to the case of a random vector X with possibly dependent coordinates. Our proof technique leverages tools from convex geometry.

Publication: Entropy Vol.: 20 No.: 3 ISSN: 1099-4300

ID: CaltechAUTHORS:20180430-101112645

]]>

Abstract: We consider the problem of sequential transmission of Gauss-Markov sources. We show that in the limit of large spatial block lengths, greedy compression with respect to the squared error distortion is optimal; that is, there is no tension between optimizing the distortion of the source in the current time instant and that of future times. We then extend this result to the case where at time t a random compression rate rt is allocated independently of the rate at other time instants. This, in turn, allows us to derive the optimal performance of sequential coding over packet-erasure channels with instantaneous feedback. For the case of packet erasures with delayed feedback, we connect the problem to that of compression with side information that is known at the encoder and may be known at the decoder — where the most recent packets serve as side information that may have been erased, and demonstrate that the loss due to a delay by one time unit is rather small.

ID: CaltechAUTHORS:20180209-081746219

]]>

Abstract: This paper considers lossy source coding of n-dimensional memoryless sources and shows an explicit approximation to the minimum source coding rate required to sustain the probability of exceeding distortion d no greater than ϵ, which is simpler than known dispersion-based approximations. Our approach takes inspiration in the celebrated classical result stating that the Shannon lower bound to rate-distortion function becomes tight in the limit d → 0. We formulate an abstract version of the Shannon lower bound that recovers both the classical Shannon lower bound and the rate-distortion function itself as special cases. Likewise, we show that a nonasymptotic version of the abstract Shannon lower bound recovers all previously known nonasymptotic converses. A necessary and sufficient condition for the Shannon lower bound to be attained exactly is presented. It is demonstrated that whenever that condition is met, the rate-dispersion function is given simply by the varentropy of the source. Remarkably, all finite alphabet sources with balanced distortion measures satisfy that condition in the range of low distortions. Most continuous sources violate that condition. Still, we show that lattice quantizers closely approach the nonasymptotic Shannon lower bound, provided that the source density is smooth enough and the distortion is low. This implies that fine multidimensional lattice coverings are nearly optimal in the rate-distortion sense even at finite . The achievability proof technique is based on a new bound on the output entropy of lattice quantizers in terms of the differential entropy of the source, the lattice cell size, and a smoothness parameter of the source density. The technique avoids both the usual random coding argument and the simplifying assumption of the presence of a dither signal.

Publication: IEEE Transactions on Information Theory Vol.: 63 No.: 7 ISSN: 0018-9448

ID: CaltechAUTHORS:20170308-161418854

]]>

Abstract: We derive a lower bound on the differential entropy for symmetric log-concave random variable X in terms of the p-th absolute moment of X, which shows that entropy and p-th absolute moment of a symmetric log-concave random variable are comparable. We apply our bound to study the rate distortion function under distortion measure |x − x|^r for sources that follow a log-concave probability distribution. In particular, we establish that the difference between the rate distortion function and the Shannon lower bound is at most log(√2e) ≈ 1.9 bits, independently of r and the target distortion d. For mean-square error distortion, the difference is at most log √πe ≈ 1.55 bits, regardless of d. Our results generalize to the case of vector X. Our proof technique leverages tools from convex geometry.

ID: CaltechAUTHORS:20170816-162727008

]]>

Abstract: This paper quantifies the fundamental limits of variable-length transmission of a general (possibly analog) source over a memoryless channel with noiseless feedback, under a distortion constraint. We consider excess distortion, average distortion and guaranteed distortion (d-semifaithful codes). In contrast to the asymptotic fundamental limit, a general conclusion is that allowing variable-length codes and feedback leads to a sizable improvement in the fundamental delay-distortion tradeoff. In addition, we investigate the minimum energy required to reproduce k source samples with a given fidelity after transmission over a memoryless Gaussian channel, and we show that the required minimum energy is reduced with feedback and an average (rather than maximal) power constraint.

Publication: IEEE Transactions on Information Theory Vol.: 63 No.: 6 ISSN: 0018-9448

ID: CaltechAUTHORS:20170524-171749779

]]>

Abstract: As an attempt to bridge the gap between classical information theory and the combinatorial world of zero-error information theory, this paper studies the performance of randomly generated codebooks over discrete memoryless channels under a zero-error constraint. This study allows the application of tools from one area to the other. Furthermore, it leads to an information-theoretic formulation of the birthday problem, which is concerned with the probability that in a given population, a fixed number of people have the same birthday. Due to the lack of a closed-form expression for this probability when the distribution of birthdays is not uniform, the resulting computation is not feasible in some applications; the information-theoretic formulation, however, can be analyzed for all distributions.

ID: CaltechAUTHORS:20170816-163747105

]]>

Abstract: In successive refinement of information, the decoder refines its representation of the source progressively as it receives more encoded bits. The rate-distortion region of successive refinement describes the minimum rates required to attain the target distortions at each decoding stage. In this paper, we derive a parametric characterization of the rate-distortion region for successive refinement of abstract sources. Our characterization extends Csiszar's result [1] to successive refinement, and generalizes a result by Tuncel and Rose [2], applicable for finite alphabet sources, to abstract sources. The new characterization leads to a family of outer bounds to the rate-distortion region. It also enables new nonasymptotic converse bounds.

ID: CaltechAUTHORS:20170816-161319771

]]>

Abstract: We consider the problem of controlling an unstable plant over an additive white Gaussian noise (AWGN) channel with a transmit power constraint, where the signaling rate of communication is larger than the sampling rate (for generating observations and applying control inputs) of the underlying plant. Such a situation is quite common since sampling is done at a rate that captures the dynamics of the plant and which is often much lower than the rate that can be communicated. This setting offers the opportunity of improving the system performance by employing multiple channel uses to convey a single message (output plant observation or control input). Common ways of doing so are through either repeating the message, or by quantizing it to a number of bits and then transmitting a channel coded version of the bits whose length is commensurate with the number of channel uses per sampled message. We argue that such “separated source and channel coding” can be suboptimal and propose to perform joint source-channel coding. Since the block length is short we obviate the need to go to the digital domain altogether and instead consider analog joint source-channel coding. For the case where the communication signaling rate is twice the sampling rate, we employ the Archimedean bi-spiral-based Shannon-Kotel'nikov analog maps to show significant improvement in stability margins and linear-quadratic Gaussian (LQG) costs over simple schemes that employ repetition.

ID: CaltechAUTHORS:20170106-132902522

]]>

Abstract: This paper shows new general nonasymptotic achievability and converse bounds and performs their dispersion analysis for the lossy compression problem in which the compressor observes the source through a noisy channel. While this problem is asymptotically equivalent to a noiseless lossy source coding problem with a modified distortion function, nonasymptotically there is a noticeable gap in how fast their minimum achievable coding rates approach the common rate-distortion function, as evidenced both by the refined asymptotic analysis (dispersion) and the numerical results. The size of the gap between the dispersions of the noisy problem and the asymptotically equivalent noiseless problem depends on the stochastic variability of the channel through which the compressor observes the source.

Publication: IEEE Transactions on Information Theory Vol.: 62 No.: 11 ISSN: 0018-9448

ID: CaltechAUTHORS:20160726-080006578

]]>

Abstract: Consider a distributed control problem with a communication channel connecting the observer of a linear stochastic system to the controller. The goal of the controller is minimize a quadratic cost function. The most basic special case of that cost function is the mean-square deviation of the system state from the desired state. We study the fundamental tradeoff between the communication rate r bits/sec and the limsup of the expected cost b, and show a lower bound on the rate necessary to attain b. The bound applies as long as the system noise has a probability density function. If target cost b is not too large, that bound can be closely approached by a simple lattice quantization scheme that only quantizes the innovation, that is, the difference between the controller's belief about the current state and the true state.

ID: CaltechAUTHORS:20170221-070702279

]]>

Abstract: Controlling and stabilizing systems involves countering the impact of explicit communication constraints in addition to inherent system model parameter uncertainty and random noise. Here we use an information-theoretic approach to jointly tackle all three issues and understand their interactions. Our main result bounds the minimum communication rate required for the mean-square stability of a system with uncertain system gain. Moreover, our techniques extend to provide a finer characterization of the required rate when specific finite bounds on the second moment of the state are desired.

ID: CaltechAUTHORS:20170221-064842476

]]>

Abstract: This paper formulates an abstract version of Shannon's lower bound that applies to abstract sources and arbitrary distortion measures and that recovers the classical Shannon lower bound as a special case. A necessary and sufficient condition for it to be attained exactly is presented. It is demonstrated that whenever that condition is met, the d-tilted information of the source adopts a simple, explicit representation that parallels Shannon's lower bound. That convenient representation simplifies the non-asymptotic analysis of achievable rate-distortion tradeoffs. In particular, if a memoryless source meets Shannon's lower bound with equality, then its rate-dispersion function is given simply by the varentropy of the source.

ID: CaltechAUTHORS:20170221-071301966

]]>

Abstract: This paper considers lossy source coding of n-dimensional continuous memoryless sources with low mean-square error distortion and shows a simple, explicit approximation to the minimum source coding rate. More precisely, a nonasymptotic version of Shannon's lower bound is presented. Lattice quantizers are shown to approach that lower bound, provided that the source density is smooth enough and the distortion is low, which implies that fine multidimensional lattice coverings are nearly optimal in the rate-distortion sense even at finite n. The achievability proof technique avoids both the usual random coding argument and the simplifying assumption of the presence of a dither signal.

ID: CaltechAUTHORS:20160412-084329467

]]>

Abstract: This paper studies the fundamental limits of the minimum average length of lossless and lossy variable-length compression, allowing a nonzero error probability ε, for lossless compression. We give nonasymptotic bounds on the minimum average length in terms of Erokhin’s rate-distortion function and we use those bounds to obtain a Gaussian approximation on the speed of approach to the limit, which is quite accurate for all but small blocklengths: (1 − ε)kH(S) − ((kV(S)/2π))^1/2 exp[−((Q−1 (ε))^(2)/2)], where Q^−1 (·) is the functional inverse of the standard Gaussian complementary cumulative distribution function, and V(S) is the source dispersion. A nonzero error probability thus not only reduces the asymptotically achievable rate by a factor of 1 − ε, but this asymptotic limit is approached from below, i.e., larger source dispersions and shorter blocklengths are beneficial. Variablelength lossy compression under an excess distortion constraint is shown to exhibit similar properties.

Publication: IEEE Transactions on Information Theory Vol.: 61 No.: 8 ISSN: 0018-9448

ID: CaltechAUTHORS:20150729-151803634

]]>

Abstract: This paper quantifies the fundamental limits of variable-length transmission of a general (possibly analog) source over a memoryless channel with noiseless feedback, under a distortion constraint. We consider excess distortion, average distortion and guaranteed distortion (d-semifaithful codes). In contrast to the asymptotic fundamental limit, a general conclusion is that allowing variable-length codes and feedback leads to a sizable improvement in the fundamental delay-distortion tradeoff.

ID: CaltechAUTHORS:20151006-111737876

]]>

Abstract: This paper shows the strong converse and the dispersion of memoryless channels with cost constraints and performs a refined analysis of the third-order term in the asymptotic expansion of the maximum achievable channel coding rate, showing that it is equal to (1/2)((log n)/n) in most cases of interest. The analysis is based on a nonasymptotic converse bound expressed in terms of the distribution of a random variable termed the mathsf b -tilted information density, which plays a role similar to that of the mathsf d -tilted information in lossy source coding. We also analyze the fundamental limits of lossy joint-source-channel coding over channels with cost constraints.

Publication: IEEE Transactions on Information Theory Vol.: 61 No.: 5 ISSN: 0018-9448

ID: CaltechAUTHORS:20150428-084727561

]]>

Abstract: We investigate the minimum transmitted energy required to reproduce k source samples with a given fidelity after transmission over a memoryless Gaussian channel. In particular, we analyze the reduction in transmitted energy that accrues thanks to the availability of noiseless feedback. Allowing a nonvanishing excess distortion probability ∈ boosts the asymptotic fundamental limit by a factor of 1−∈, with or without feedback. If feedback is available, achieving guaranteed distortion with finite average energy is possible.

ID: CaltechAUTHORS:20150630-073130634

]]>

Abstract: This paper provides a necessary condition good rate-distortion codes must satisfy. Specifically, it is shown that as the blocklength increases, the distribution of the input given the output of a good lossy code converges to the distribution of the input given the output of the joint distribution achieving the rate-distortion function, in terms of the normalized conditional relative entropy. The result holds for stationary ergodic sources with subadditive distortion measures, both for fixed-length and variable-length compression. A similar necessary condition is given for lossy joint source-channel coding.

ID: CaltechAUTHORS:20161115-082050134

]]>

Abstract: This paper studies the fundamental limits of the minimum average length of variable-length compression when a nonzero error probability ε is tolerated. We give non-asymptotic bounds on the minimum average length in terms of Erokhin's rate-distortion function and we use those bounds to obtain a Gaussian approximation on the speed of approach to the limit which is quite accurate for all but small blocklengths: (1-ε)kH(S) - √kV(S)/2π e – (Q^(-1)(ε))^2/2 where Q-1 (·) is the functional inverse of the Q-function and V (S) is the source dispersion. A nonzero error probability thus not only reduces the asymptotically achievable rate by a factor of 1-ε, but also this asymptotic limit is approached from below, i.e. a larger source dispersion and shorter blocklengths are beneficial. Further, we show that variable-length lossy compression under excess distortion constraint also exhibits similar properties.

Publication: IEEE Transactions on Information TheoryISSN: 0018-9448

ID: CaltechAUTHORS:20150730-142923980

]]>

Abstract: Convexity properties of error rates of a class of decoders, including the maximum-likelihood/min-distance one as a special case, are studied for arbitrary constellations, bit mapping, and coding. Earlier results obtained for the additive white Gaussian noise channel are extended to a wide class of noise densities, including unimodal and spherically invariant noise. Under these broad conditions, symbol and bit error rates are shown to be convex functions of the signal-to-noise ratio (SNR) in the high-SNR regime with an explicitly determined threshold, which depends only on the constellation dimensionality and minimum distance, thus enabling an application of the powerful tools of convex optimization to such digital communication systems in a rigorous way. It is the decreasing nature of the noise power density around the decision region boundaries that ensures the convexity of symbol error rates in the general case. The known high/low-SNR bounds of the convexity/concavity regions are tightened and no further improvement is shown to be possible in general. The high-SNR bound fits closely into the channel coding theorem: all codes, including capacity-achieving ones, whose decision regions include the hardened noise spheres (from the noise sphere hardening argument in the channel coding theorem), satisfy this high-SNR requirement and thus has convex error rates in both SNR and noise power. We conjecture that all capacity-achieving codes have convex error rates. Convexity properties in signal amplitude and noise power are also investigated. Some applications of the results are discussed. In particular, it is shown that fading is convexity-preserving and is never good in low dimensions under spherically invariant noise, which may also include any linear diversity combining.

Publication: IEEE Transactions on Information Theory Vol.: 59 No.: 10 ISSN: 0018-9448

ID: CaltechAUTHORS:20190213-084137169

]]>

Abstract: Convexity properties of error rates of a class of decoders, including the ML/min-distance one as a special case, are studied for arbitrary constellations. Earlier results obtained for the AWGN channel are extended to a wide class of (non-Gaussian) noise densities, including unimodal and spherically-invariant noise. Under these broad conditions, symbol error rates are shown to be convex functions of the SNR in the high-SNR regime with an explicitly-determined threshold, which depends only on the constellation dimensionality and minimum distance, thus enabling an application of the powerful tools of convex optimization to such digital communication systems in a rigorous way. It is the decreasing nature of the noise power density around the decision region boundaries that insures the convexity of symbol error rates in the general case. The known high/low SNR bounds of the convexity/concavity regions are tightened and no further improvement is shown to be possible in general.

ID: CaltechAUTHORS:20190213-075441906

]]>

Abstract: This paper finds new tight finite-blocklength bounds for the best achievable lossy joint source-channel code rate, and demonstrates that joint source-channel code design brings considerable performance advantage over a separate one in the nonasymptotic regime. A joint source-channel code maps a block of k source symbols onto a length- n channel codeword, and the fidelity of reproduction at the receiver end is measured by the probability ε that the distortion exceeds a given threshold d . For memoryless sources and channels, it is demonstrated that the parameters of the best joint source-channel code must satisfy nC - kR( d ) ≈ √( nV + k V ( d )) Q^(-1) (ε), where C and V are the channel capacity and channel dispersion, respectively; R ( d ) and V ( d ) are the source rate-distortion and rate-dispersion functions; and Q is the standard Gaussian complementary cumulative distribution function. Symbol-by-symbol (uncoded) transmission is known to achieve the Shannon limit when the source and channel satisfy a certain probabilistic matching condition. In this paper, we show that even when this condition is not satisfied, symbol-by-symbol transmission is, in some cases, the best known strategy in the nonasymptotic regime.

Publication: IEEE Transactions on Information Theory Vol.: 59 No.: 5 ISSN: 0018-9448

ID: CaltechAUTHORS:20190213-090748243

]]>

Abstract: This paper shows new tight finite-blocklength bounds for the best achievable lossy joint source-channel code rate, and demonstrates that joint source-channel code design brings considerable performance advantage over a separate one in the non-asymptotic regime. A joint source-channel code maps a block of k source symbols onto a length - n channel codeword, and the fidelity of reproduction at the receiver end is measured by the probability ϵ that the distortion exceeds a given threshold d. For memoryless sources and channels, it is demonstrated that the parameters of the best joint source-channel code must satisfy nC - kR(d) ≈ √(nV + kV(d)) Q^(-1) (ϵ), where C and V are the channel capacity and dispersion, respectively; R(d) and V(d) are the source rate-distortion and rate-dispersion functions; and Q is the standard Gaussian complementary cdf.

ID: CaltechAUTHORS:20140910-101108253

]]>

Abstract: This paper studies the minimum achievable source coding rate as a function of blocklength n and probability ϵ that the distortion exceeds a given level d. Tight general achievability and converse bounds are derived that hold at arbitrary fixed blocklength. For stationary memoryless sources with separable distortion, the minimum rate achievable is shown to be closely approximated by R(d) + √V(d)/(n) Q^(-1)(ϵ), where R(d) is the rate-distortion function, V(d) is the rate dispersion, a characteristic of the source which measures its stochastic variability, and Q-1(·) is the inverse of the standard Gaussian complementary cumulative distribution function.

Publication: IEEE Transactions on Information Theory Vol.: 58 No.: 6 ISSN: 0018-9448

ID: CaltechAUTHORS:20140910-113526494

]]>

Abstract: This paper shows new finite-blocklength converse bounds applicable to lossy source coding as well as joint source-channel coding, which are tight enough not only to prove the strong converse, but to find the rate-dispersion functions in both setups. In order to state the converses, we introduce the d-tilted information, a random variable whose expectation and variance (with respect to the source) are equal to the rate-distortion and rate-dispersion functions, respectively.

ID: CaltechAUTHORS:20140910-102046386

]]>

Abstract: For an i.i.d. Gaussian source with variance σ^2, we show that it is necessary to spend ½ ln σ^2/d + 1/√(2n) Q^(-1)(ε) + O (ln n/n) nats per sample in order to reproduce n source samples within mean-square error d with probability at least 1 - ε, where Q^(-1) (·) is the inverse of the standard Gaussian complementary cdf. The first-order term is the rate-distortion function of the Gaussian source, while the second-order term measures its stochastic variability. We derive new achievability and converse bounds that are valid at any blocklength and show that the second-order approximation is tightly wedged between them, thus providing a concise and accurate approximation of the minimum achievable source coding rate at a given fixed blocklength (unless the blocklength is very small).

ID: CaltechAUTHORS:20140910-112002771

]]>

Abstract: Several instantaneous optimization strategies for rate and/or power allocation in the coded V-BLAST are studied analytically. Outage probabilities and system capacities of these strategies in a spatial multiplexing system are compared under generic settings. The conventional waterfilling algorithm is shown to be suboptimal for the coded V-BLAST and a new algorithm ("fractional water-filling") is proposed, which simultaneously maximizes the system capacity and minimizes the outage probability. Closed-form performance analysis of the considered algorithms is given, and the fractional water-filling algorithm is shown to attain the full MIMO channel diversity, significantly outperforming other strategies. Many of the results also apply to generic multi-stream transmission systems (e.g. spatial multiplexing on the channel eigenmodes, OFDM) or the systems relying on successive interference cancelation (multi-user detection, channel equalization).

Publication: IEEE Transactions on Communications Vol.: 59 No.: 10 ISSN: 0090-6778

ID: CaltechAUTHORS:20140910-105022320

]]>

Abstract: This paper studies the minimum achievable source coding rate as a function of blocklength n and tolerable distortion level d. Tight general achievability and converse bounds are derived that hold at arbitrary fixed blocklength. For stationary memoryless sources with separable distortion, the minimum rate achievable is shown to be q closely approximated by R(d) + √v(d)/nQ^(-1)(ϵ), where R(d) is the rate-distortion function, V (d) is the rate dispersion, a characteristic of the source which measures its stochastic variability, Q-1 (·) is the inverse of the standard Gaussian complementary cdf, and ϵ is the probability that the distortion exceeds d. The new bounds and the second-order approximation of the minimum achievable rate are evaluated for the discrete memoryless source with symbol error rate distortion. In this case, the second-order approximation reduces to R(d) + 1/2 log n/n if the source is non-redundant.

ID: CaltechAUTHORS:20140910-110253022

]]>

Abstract: Several optimization strategies for instantaneous rate and/or power allocation in the coded V-BLAST are studied analytically. Outage probabilities and system capacities of these strategies in a spatial multiplexing system are compared under generic settings. Since the conventional waterfilling algorithm is suboptimal for the coded V-BLAST, a recently-proposed “fractional waterfilling” algorithm is studied, which simultaneously maximizes the system capacity and minimizes the outage probability. A comparative, closed-form performance analysis of this and other algorithms is presented, including bounds on the outage probability and its low-outage approximations. The fractional waterfilling algorithm attains the full MIMO channel diversity and outperforms the other algorithms by a wide margin.

ID: CaltechAUTHORS:20140909-140645304

]]>

Abstract: The aim of compressed sensing is to recover attributes of sparse signals using very few measurements. Given an overall bit budget for quantization, this paper demonstrates that there is value to redundant measurement. The measurement matrices considered here are required to have the property that signal recovery is still possible even after dropping certain subsets of D measurements. It introduces the concept of a measurement matrix that is weakly democratic in the sense that the amount of information about the signal carried by each of the designated D-subsets is the same. Examples of deterministic measurement matrices that are weakly democratic are constructed by exponentiating codewords from the binary second order Reed Muller code. The value in rejecting D measurements that are on average larger, is to be able to provide a finer grid for vector quantization of the remaining measurements, even after discounting the original budget by the bits used to identify the reject set. Simulation results demonstrate that redundancy improves recovery SNR, sometimes by a wide margin. Optimum performance occurs when a significant fraction of measurements are rejected.

ID: CaltechAUTHORS:20140910-103127300

]]>

Abstract: An analytical framework for performance analysis and optimization of coded V-BLAST is developed. Average power and/or rate allocations to minimize the outage probability as well as their robustness and dual problems are investigated. Compact, closed-form expressions for the optimum allocations and corresponding system performance are given. The uniform power allocation is shown to be near optimum in the low outage regime in combination with the optimum rate allocation. The average rate allocation provides the largest performance improvement (extra diversity gain), and the average power allocation offers a modest SNR gain limited by the number of transmit antennas but does not increase the diversity gain. The dual problems are shown to have the same solutions as the primal ones. All these allocation strategies are shown to be robust. The reported results also apply to coded multiuser detection and channel equalization systems relying on successive interference cancellation.

Publication: IEEE Transactions on Communications Vol.: 59 No.: 3 ISSN: 0090-6778

ID: CaltechAUTHORS:20140910-074750052

]]>

Abstract: Motivated by a wide-spread use of convex optimization techniques, convexity properties of bit error rate of the maximum likelihood detector operating in the AWGN channel are studied for arbitrary constellations and bit mappings, which also includes coding under maximum-likelihood decoding. Under this generic setting, the pairwise probability of error and bit error rate are shown to be convex functions of the SNR and noise power in the high SNR/low noise regime with explicitly-determined boundary. Any code, including capacity-achieving ones, whose decision regions include the hardened noise spheres (from the noise sphere hardening argument in the channel coding theorem) satisfies this high SNR requirement and thus has convex error rates in both SNR and noise power. We conjecture that all capacity-achieving codes have convex error rates.

ID: CaltechAUTHORS:20140910-094918979

]]>

Abstract: Motivated by a recent surge of interest in convex optimization techniques, convexity/concavity properties of error rates of the maximum likelihood detector operating in the AWGN channel are studied and extended to frequency-flat slow-fading channels. Generic conditions are identified under which the symbol error rate (SER) is convex/concave for arbitrary multidimensional constellations. In particular, the SER is convex in SNR for any one- and two-dimensional constellation, and also in higher dimensions at high SNR. Pairwise error probability and bit error rate are shown to be convex at high SNR, for arbitrary constellations and bit mapping. Universal bounds for the SER first and second derivatives are obtained, which hold for arbitrary constellations and are tight for some of them. Applications of the results are discussed, which include optimum power allocation in spatial multiplexing systems, optimum power/time sharing to decrease or increase (jamming problem) error rate, an implication for fading channels (Â¿fading is never good in low dimensionsÂ¿) and optimization of a unitary-precoded OFDM system. For example, the error rate bounds of a unitary-precoded OFDM system with QPSK modulation, which reveal the best and worst precoding, are extended to arbitrary constellations, which may also include coding. The reported results also apply to the interference channel under Gaussian approximation, to the bit error rate when it can be expressed or approximated as a nonnegative linear combination of individual symbol error rates, and to coded systems.

Publication: IEEE Transactions on Information Theory Vol.: 56 No.: 4 ISSN: 0018-9448

ID: CaltechAUTHORS:20140910-093556659

]]>

Abstract: Motivated by a wide-spread use of convex optimization techniques, convexity properties of bit error rate of the maximum likelihood detector operating in the AWGN channel are studied for arbitrary constellations and bit mappings, which may also include coding under maximum-likelihood decoding. Under this generic setting, the pairwise probability of error and bit error rate are shown to be convex functions of the SNR in the high SNR regime with explicitly-determined boundary. The bit error rate is also shown to be a convex function of the noise power in the low noise/high SNR regime.

Publication: arXiv
ID: CaltechAUTHORS:20190403-084738860

]]>

Abstract: An analytical framework for minimizing the outage probability of a coded spatial multiplexing system while keeping the rate close to the capacity is developed. Based on this framework, specific strategies of optimum power and rate allocation for the coded V-BLAST architecture are obtained and its performance is analyzed. A fractional waterfilling algorithm, which is shown to optimize both the capacity and the outage probability of the coded V-BLAST, is proposed. Compact, closed-form expressions for the optimum allocation of the average power are given. The uniform allocation of average power is shown to be near optimum at moderate to high SNR for the coded V-BLAST with the average rate allocation (when per-stream rates are set to match the per-stream capacity). The results reported also apply to multiuser detection and channel equalization relying on successive interference cancelation.

ID: CaltechAUTHORS:20140910-091940642

]]>

Abstract: A unified analytical framework for optimum power allocation in the unordered V-BLAST algorithm and its comparative performance analysis are presented. Compact closed-form approximations for the optimum power allocation are derived, based on average total and block error rates. The choice of the criterion has little impact on the power allocation and, overall, the optimum strategy is to allocate more power to lower step transmitters and less to higher ones. High-SNR approximations for optimized average block and total error rates are given. The SNR gain of optimization is rigorously defined and studied using analytical tools, including lower and upper bounds, high and low SNR approximations. The gain is upper bounded by the number of transmit antennas, for any modulation format and type of fading channel. While the average optimization is less complex than the instantaneous one, its performance is almost as good at high SNR. A measure of robustness of the optimized algorithm is introduced and evaluated. The optimized algorithm is shown to be robust to perturbations in individual and total transmit powers. Based on the algorithm robustness, a pre-set power allocation is suggested as a low-complexity alternative to the other optimization strategies, which exhibits only a minor loss in performance over the practical SNR range.

Publication: IEEE Transactions on Communications Vol.: 56 No.: 6 ISSN: 0090-6778

ID: CaltechAUTHORS:20140909-152126131

]]>

Abstract: Comprehensive performance analysis of the unordered V-BLAST algorithm with various power allocation strategies is presented, which makes use of analytical tools and resorts to Monte-Carlo simulations for validation purposes only. High-SNR approximations for the optimized average block and total error rates are given. The SNR gain of optimization is rigorously defined and studied using analytical tools, including lower and upper bounds, high and low SNR approximations. The gain is upper bounded by the number of transmitters, for any modulation format and any type of fading This upper bound is achieved at high SNR by the considered optimization strategies. While the average optimization is less complex than the instantaneous one, its performance is almost as good at high SNR. A measure of robustness of the optimized algorithm is introduced and evaluated, including compact closed-form approximations. The optimized algorithm is shown to be robust to perturbations in individual and total transmit powers. Based on the algorithm robustness, a pre-set power allocation is suggested as a low-complexity alternative to the other optimization strategies, which exhibits only a minor loss in performance over the practical SNR range.

ID: CaltechAUTHORS:20140910-070612419

]]>

Abstract: Convexity/concavity properties of symbol error rates (SER) of the maximum likelihood detector operating in the AWGN channel (non-fading and fading) are studied. Generic conditions are identified under which the SER is a convex/concave function of the SNR. Universal bounds for the SER 1st and 2nd derivatives are obtained, which hold for arbitrary constellations and are tight for some of them. Applications of the results are discussed, which include optimum power allocation in spatial multiplexing systems, optimum power/time sharing to decrease or increase (jamming problem) error rate, and implication for fading channels.

ID: CaltechAUTHORS:20140909-150741253

]]>

Abstract: Optimum power allocation for the V-BLAST algorithm, which is based on various criteria (average and instantaneous block and total error rates (BLER and TBER)), is considered. Closed-form solutions are derived for high-SNR case in a Rayleigh fading channel. They are shown to be robust to small variations in allocated power and average SNR. It is demonstrated that the optimization "on average" is only slightly worse than the instantaneous one, albeit the latter requires an instantaneous feedback and hence is of higher complexity. The generic upper-bound for the SNR gain of any power allocation technique is derived. The BLER and TBER optimization criteria result in the same performance. Power optimization (of unordered BLAST) and optimal ordering result in almost the same performance improvement at high SNR.

ID: CaltechAUTHORS:20140909-153041130

]]>

Abstract: Optimum power allocation for the V-BLAST algorithm, which is based on various criteria (average and instantaneous block and total error rates (BLER and TBER)), is considered. Closed-form expressions are derived for high-SNR case in a Rayleigh fading channel. It is demonstrated that, in that case, the optimization "on average" is almost identical to the instantaneous one (while the former requires only the feedback "on average", the latter requires instantaneous feedback and hence is of higher complexity). The BLER and TBER optimization criteria result in the same performance. Power optimization (of un-ordered BLAST) and optimal ordering result in the same performance improvement at high SNR.

ID: CaltechAUTHORS:20140909-154358392

]]>