Submitted - 1305.2714v5.pdf

", "abstract": "Denoising has to do with estimating a signal x_0 from its noisy observations y = x_0 + z. In this paper, we focus on the \"structured denoising problem,\" where the signal x_0 possesses a certain structure and z has independent normally distributed entries with mean zero and variance \u03c3^2. We employ a structure-inducing convex function f(\u22c5) and solve min_x {1/2 \u2225y\u2212x\u2225^2_2 +\u03c3\u03bbf(x)}to estimate x_0, for some \u03bb>0. Common choices for f(\u22c5) include the \u2113_1 norm for sparse vectors, the \u2113_1 \u2212\u2113_2 norm for block-sparse signals and the nuclear norm for low-rank matrices. The metric we use to evaluate the performance of an estimate x\u2217 is the normalized mean-squared error NMSE(\u03c3)=E\u2225x\u2217 \u2212 x_0\u2225^2_2/\u03c3^2. We show that NMSE is maximized as \u03c3\u21920 and we find the exact worst-case NMSE, which has a simple geometric interpretation: the mean-squared distance of a standard normal vector to the \u03bb-scaled subdifferential \u03bb\u2202f(x_0). When \u03bb is optimally tuned to minimize the worst-case NMSE, our results can be related to the constrained denoising problem min_(f(x)\u2264f(x_0)){\u2225y\u2212x\u22252}. The paper also connects these results to the generalized LASSO problem, in which one solves min_(f(x)\u2264f(x_0)){\u2225y\u2212Ax\u22252} to estimate x_0 from noisy linear observations y=Ax_0 + z. We show that certain properties of the LASSO problem are closely related to the denoising problem. In particular, we characterize the normalized LASSO cost and show that it exhibits a \"phase transition\" as a function of number of observations. We also provide an order-optimal bound for the LASSO error in terms of the mean-squared distance. Our results are significant in two ways. First, we find a simple formula for the performance of a general convex estimator. Secondly, we establish a connection between the denoising and linear inverse problems.", "date": "2016-08", "date_type": "published", "publication": "Foundations of Computational Mathematics", "volume": "16", "number": "4", "publisher": "Springer", "pagerange": "965-1029", "id_number": "CaltechAUTHORS:20160825-102038882", "issn": "1615-3375", "official_url": "https://resolver.caltech.edu/CaltechAUTHORS:20160825-102038882", "rights": "No commercial reproduction, distribution, display or performance rights in this work are provided.", "funders": { "items": [ { "agency": "NSF", "grant_number": "CCF-0729203" }, { "agency": "NSF", "grant_number": "CNS-0932428" }, { "agency": "NSF", "grant_number": "CIF-1018927" }, { "agency": "Office of Naval Research (ONR)", "grant_number": "N00014-08-1-0747" }, { "agency": "Qualcomm Inc." } ] }, "doi": "10.1007/s10208-015-9278-4", "primary_object": { "basename": "1305.2714v5.pdf", "url": "https://authors.library.caltech.edu/records/fbe5r-xn288/files/1305.2714v5.pdf" }, "resource_type": "article", "pub_year": "2016", "author_list": "Oymak, Samet and Hassibi, Babak" }, { "id": "https://authors.library.caltech.eduhttps://authors.library.caltech.edu/records/cqrjk-bxj69", "eprint_id": 70119, "eprint_status": "archive", "datestamp": "2023-08-20 04:16:03", "lastmod": "2024-01-13 16:53:25", "type": "book_section", "metadata_visibility": "show", "creators": { "items": [ { "id": "Thrampoulidis-Christos", "name": { "family": "Thrampoulidis", "given": "Christos" }, "orcid": "0000-0001-9053-9365" }, { "id": "Oymak-Samet", "name": { "family": "Oymak", "given": "Samet" } }, { "id": "Hassibi-B", "name": { "family": "Hassibi", "given": "Babak" } } ] }, "title": "Recovering Structured Signals in Noise: Least-Squares Meets Compressed Sensing", "ispublished": "unpub", "full_text_status": "restricted", "note": "\u00a9 2015 Springer International Publishing Switzerland. \n\nThe authors gratefully acknowledge the anonymous reviewers for their attention and their helpful comments.", "abstract": "The typical scenario that arises in most \"big data\" problems is one where the ambient dimension of the signal is very large (e.g., high resolution images, gene expression data from a DNA microarray, social network data, etc.), yet is such that its desired properties lie in some low dimensional structure (sparsity, low-rankness, clusters, etc.). In the modern viewpoint, the goal is to come up with efficient algorithms to reveal these structures and for which, under suitable conditions, one can give theoretical guarantees. We specifically consider the problem of recovering such a structured signal (sparse, low-rank, block-sparse, etc.) from noisy compressed measurements. A general algorithm for such problems, commonly referred to as generalized LASSO, attempts to solve this problem by minimizing a least-squares cost with an added \"structure-inducing\" regularization term (\u2113_ 1 norm, nuclear norm, mixed \u2113 _2/\u2113_ 1 norm, etc.). While the LASSO algorithm has been around for 20 years and has enjoyed great success in practice, there has been relatively little analysis of its performance. In this chapter, we will provide a full performance analysis and compute, in closed form, the mean-square-error of the reconstructed signal. We will highlight some of the mathematical vignettes necessary for the analysis, make connections to noiseless compressed sensing and proximal denoising, and will emphasize the central role of the \"statistical dimension\" of a structured signal.", "date": "2015", "date_type": "published", "publisher": "Springer", "place_of_pub": "Cham", "pagerange": "97-141", "id_number": "CaltechAUTHORS:20160901-121710649", "isbn": "978-3-319-16041-2", "book_title": "Compressed Sensing and its Applications", "official_url": "https://resolver.caltech.edu/CaltechAUTHORS:20160901-121710649", "rights": "No commercial reproduction, distribution, display or performance rights in this work are provided.", "contributors": { "items": [ { "id": "Boche-Holger", "name": { "family": "Boche", "given": "Holger" } }, { "id": "Calderbank-Robert", "name": { "family": "Calderbank", "given": "Robert" } }, { "id": "Kutyniok-Gitta", "name": { "family": "Kutyniok", "given": "Gitta" } }, { "id": "Vyb\u00edral-Jan", "name": { "family": "Vyb\u00edral", "given": "Jan" } } ] }, "doi": "10.1007/978-3-319-16042-9_4", "resource_type": "book_section", "pub_year": "2015", "author_list": "Thrampoulidis, Christos; Oymak, Samet; et el." } ]