On the Relationship Between Conjugate Gradient and Optimal First-Order Methods for Convex Optimization

2013

Author : Sahar Karimi
Release : 2013
Genre :
Kind : eBook
Book Rating : /5 ( reviews)

Book Synopsis On the Relationship Between Conjugate Gradient and Optimal First-Order Methods for Convex Optimization by : Sahar Karimi

Download or read book On the Relationship Between Conjugate Gradient and Optimal First-Order Methods for Convex Optimization written by Sahar Karimi. This book was released on 2013. Available in PDF, EPUB and Kindle. Book excerpt: In a series of work initiated by Nemirovsky and Yudin, and later extended by Nesterov, first-order algorithms for unconstrained minimization with optimal theoretical complexity bound have been proposed. On the other hand, conjugate gradient algorithms as one of the widely used first-order techniques suffer from the lack of a finite complexity bound. In fact their performance can possibly be quite poor. This dissertation is partially on tightening the gap between these two classes of algorithms, namely the traditional conjugate gradient methods and optimal first-order techniques. We derive conditions under which conjugate gradient methods attain the same complexity bound as in Nemirovsky-Yudin's and Nesterov's methods. Moreover, we propose a conjugate gradient-type algorithm named CGSO, for Conjugate Gradient with Subspace Optimization, achieving the optimal complexity bound with the payoff of a little extra computational cost. We extend the theory of CGSO to convex problems with linear constraints. In particular we focus on solving $l_1$-regularized least square problem, often referred to as Basis Pursuit Denoising (BPDN) problem in the optimization community. BPDN arises in many practical fields including sparse signal recovery, machine learning, and statistics. Solving BPDN is fairly challenging because the size of the involved signals can be quite large; therefore first order methods are of particular interest for these problems. We propose a quasi-Newton proximal method for solving BPDN. Our numerical results suggest that our technique is computationally effective, and can compete favourably with the other state-of-the-art solvers.

First-Order Methods in Optimization

2017-10-02 Mathematics

Author : Amir Beck
Release : 2017-10-02
Genre : Mathematics
Kind : eBook
Book Rating : 992/5 ( reviews)

GET EBOOK

Book Synopsis First-Order Methods in Optimization by : Amir Beck

Download or read book First-Order Methods in Optimization written by Amir Beck. This book was released on 2017-10-02. Available in PDF, EPUB and Kindle. Book excerpt: The primary goal of this book is to provide a self-contained, comprehensive study of the main ?rst-order methods that are frequently used in solving large-scale problems. First-order methods exploit information on values and gradients/subgradients (but not Hessians) of the functions composing the model under consideration. With the increase in the number of applications that can be modeled as large or even huge-scale optimization problems, there has been a revived interest in using simple methods that require low iteration cost as well as low memory storage. The author has gathered, reorganized, and synthesized (in a unified manner) many results that are currently scattered throughout the literature, many of which cannot be typically found in optimization books. First-Order Methods in Optimization offers comprehensive study of first-order methods with the theoretical foundations; provides plentiful examples and illustrations; emphasizes rates of convergence and complexity analysis of the main first-order methods used to solve large-scale problems; and covers both variables and functional decomposition methods.

Convex Optimization

2015-11-12 Convex domains

Author : Sébastien Bubeck
Release : 2015-11-12
Genre : Convex domains
Kind : eBook
Book Rating : 607/5 ( reviews)

GET EBOOK

Book Synopsis Convex Optimization by : Sébastien Bubeck

Download or read book Convex Optimization written by Sébastien Bubeck. This book was released on 2015-11-12. Available in PDF, EPUB and Kindle. Book excerpt: This monograph presents the main complexity theorems in convex optimization and their corresponding algorithms. It begins with the fundamental theory of black-box optimization and proceeds to guide the reader through recent advances in structural optimization and stochastic optimization. The presentation of black-box optimization, strongly influenced by the seminal book by Nesterov, includes the analysis of cutting plane methods, as well as (accelerated) gradient descent schemes. Special attention is also given to non-Euclidean settings (relevant algorithms include Frank-Wolfe, mirror descent, and dual averaging), and discussing their relevance in machine learning. The text provides a gentle introduction to structural optimization with FISTA (to optimize a sum of a smooth and a simple non-smooth term), saddle-point mirror prox (Nemirovski's alternative to Nesterov's smoothing), and a concise description of interior point methods. In stochastic optimization it discusses stochastic gradient descent, mini-batches, random coordinate descent, and sublinear algorithms. It also briefly touches upon convex relaxation of combinatorial problems and the use of randomness to round solutions, as well as random walks based methods.

Lectures on Convex Optimization

2018-11-19 Mathematics

Author : Yurii Nesterov
Release : 2018-11-19
Genre : Mathematics
Kind : eBook
Book Rating : 789/5 ( reviews)

GET EBOOK

Book Synopsis Lectures on Convex Optimization by : Yurii Nesterov

Download or read book Lectures on Convex Optimization written by Yurii Nesterov. This book was released on 2018-11-19. Available in PDF, EPUB and Kindle. Book excerpt: This book provides a comprehensive, modern introduction to convex optimization, a field that is becoming increasingly important in applied mathematics, economics and finance, engineering, and computer science, notably in data science and machine learning. Written by a leading expert in the field, this book includes recent advances in the algorithmic theory of convex optimization, naturally complementing the existing literature. It contains a unified and rigorous presentation of the acceleration techniques for minimization schemes of first- and second-order. It provides readers with a full treatment of the smoothing technique, which has tremendously extended the abilities of gradient-type methods. Several powerful approaches in structural optimization, including optimization in relative scale and polynomial-time interior-point methods, are also discussed in detail. Researchers in theoretical optimization as well as professionals working on optimization problems will find this book very useful. It presents many successful examples of how to develop very fast specialized minimization algorithms. Based on the author’s lectures, it can naturally serve as the basis for introductory and advanced courses in convex optimization for students in engineering, economics, computer science and mathematics.

Proximal Algorithms

2013-11 Mathematics

Author : Neal Parikh
Release : 2013-11
Genre : Mathematics
Kind : eBook
Book Rating : 167/5 ( reviews)

GET EBOOK

Book Synopsis Proximal Algorithms by : Neal Parikh

Download or read book Proximal Algorithms written by Neal Parikh. This book was released on 2013-11. Available in PDF, EPUB and Kindle. Book excerpt: Proximal Algorithms discusses proximal operators and proximal algorithms, and illustrates their applicability to standard and distributed convex optimization in general and many applications of recent interest in particular. Much like Newton's method is a standard tool for solving unconstrained smooth optimization problems of modest size, proximal algorithms can be viewed as an analogous tool for nonsmooth, constrained, large-scale, or distributed versions of these problems. They are very generally applicable, but are especially well-suited to problems of substantial recent interest involving large or high-dimensional datasets. Proximal methods sit at a higher level of abstraction than classical algorithms like Newton's method: the base operation is evaluating the proximal operator of a function, which itself involves solving a small convex optimization problem. These subproblems, which generalize the problem of projecting a point onto a convex set, often admit closed-form solutions or can be solved very quickly with standard or simple specialized methods. Proximal Algorithms discusses different interpretations of proximal operators and algorithms, looks at their connections to many other topics in optimization and applied mathematics, surveys some popular algorithms, and provides a large number of examples of proximal operators that commonly arise in practice.

Popular eBooks

On the Relationship Between Conjugate Gradient and Optimal First-Order Methods for Convex Optimization

First-Order Methods in Optimization

Convex Optimization

Lectures on Convex Optimization

Proximal Algorithms

You may also like...