凸优化:ADMM(Alternating Direction Method of Multipliers)交替方向乘子算法系列之六: L1-Norm Problems
2015-07-08 19:58
363 查看
最近开始对凸优化(convex optimization)中的ADMM(Alternating Direction Method of Multipliers)交替方向乘子算法开始感兴趣,接下来我会写一系列关于ADMM(Alternating Direction Method of Multipliers)交替方向乘子算法的内容。
凸优化:ADMM(Alternating Direction Method of Multipliers)交替方向乘子算法系列之六: L1-Norm Problems
本文地址:/article/1323063.html
[173] A. Y. Yang, A. Ganesh, Z. Zhou, S. S. Sastry, and Y. Ma, “A Review of Fast l1-Minimization Algorithms for Robust Face Recognition,” arXiv:1007.3753, 2010.
当数据包含较大的离群点时,Least Absolute Deviations(LAD) 通常比 least squares fitting(LSF) 提供一个更鲁棒的拟合 (Least absolute deviations provides a more robust fit than least squares when the data contains large outliers)。
写成 ADMM 形式,
![](http://img.blog.csdn.net/20150711165903791)
其中 f=0f = 0 和 g=||⋅||1g = ||· ||_{1}, 假设 ATAA^{T}A 可逆, 有
![](http://img.blog.csdn.net/20150711165912073)
![](http://img.blog.csdn.net/20150711165831204)
其中 Huber penalty function ghubg^{hub} :
![](http://img.blog.csdn.net/20150711165841056)
因此有,
![](http://img.blog.csdn.net/20150711165849275)
x-update 和 u-update 和 Least absolute deviations 的一样。
![](http://img.blog.csdn.net/20150711174441510)
[24] 是关于 Basis Pursuit 的一个综述。
写成 ADMM 形式
![](http://img.blog.csdn.net/20150711174451420)
其中 f 是
![](http://img.blog.csdn.net/20150711174701104)
的指示函数。
因此有
![](http://img.blog.csdn.net/20150711174510448)
其中 Π 是
![](http://img.blog.csdn.net/20150711174713192)
上的映射。
x-update 展开为
![](http://img.blog.csdn.net/20150711174518044)
最近提出的 Bregman iterative methods 解决类似 Basis Pursuit 的问题很有效。
For basis pursuit and related problems, Bregman iterative regularization [176] is equivalent to the method of multipliers, and the split Bregman method [88] is equivalent to ADMM [68].
[24] A. M. Bruckstein, D. L. Donoho, and M. Elad, “From sparse solutions of systems of equations to sparse modeling of signals and images,” SIAM Review, vol. 51, no. 1, pp. 34–81, 2009.
[176] W. Yin, S. Osher, D. Goldfarb, and J. Darbon, “Bregman iterative algorithms for l1-minimization with applications to compressed sensing,” SIAM Journal on Imaging Sciences, vol. 1, no. 1, pp. 143–168, 2008.
[88] T. Goldstein and S. Osher, “The split Bregman method for l1 regularized problems,” SIAM Journal on Imaging Sciences, vol. 2, no. 2, pp. 323–343, 2009.
[68] E. Esser, “Applications of Lagrangian-based alternating direction methods and connections to split Bregman,” CAM report, vol. 9, p. 31, 2009.
![](http://img.blog.csdn.net/20150711211417248)
其中 l 是任意的凸代价函数。
写成 ADMM 形式
![](http://img.blog.csdn.net/20150711211427497)
其中
![](http://img.blog.csdn.net/20150711211435441)
有
![](http://img.blog.csdn.net/20150711211443951)
In general, we can interpret ADMM for L1 regularized loss minimization as reducing it to solving a sequence of L2 (squared) regularized loss minimization problems.
![](http://img.blog.csdn.net/20150711212448784)
写成 ADMM 形式
![](http://img.blog.csdn.net/20150711212458976)
其中
![](http://img.blog.csdn.net/20150711212508806)
有
![](http://img.blog.csdn.net/20150711212518237)
The x-update is essentially a ridge regression (i.e., quadratically regularized least squares) computation, so ADMM can be interpreted as a method for solving the lasso problem by iteratively carrying out ridge regression.
6-4-1 广义Lasso(Generalized Lasso)
进一步一般化
![](http://img.blog.csdn.net/20150711212938544)
其中 F 是一个任意的线性变换。
一个特殊的例子是当 F∈R(n−1)×nF ∈ R^{(n−1)×n} 是差异矩阵
![](http://img.blog.csdn.net/20150711212947630)
和 A =I 时,
![](http://img.blog.csdn.net/20150711212956103)
写成 ADMM 形式
![](http://img.blog.csdn.net/20150711213005291)
有
![](http://img.blog.csdn.net/20150711213013440)
6-4-2 组 Lasso(Group Lasso)
![](http://img.blog.csdn.net/20150711213244801)
![](http://img.blog.csdn.net/20150711213253399)
![](http://img.blog.csdn.net/20150711213303596)
![](http://img.blog.csdn.net/20150711213312517)
[1] Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers
[2] 凸优化讲义
[3] A Note on the Alternating Direction Method of Multipliers
凸优化:ADMM(Alternating Direction Method of Multipliers)交替方向乘子算法系列之六: L1-Norm Problems
本文地址:/article/1323063.html
6- L1范式问题(L1-Norm Problems)
ADMM 自然地将飞光滑的 L1 项 与光滑的损失项分离开来,使得计算更有效。这一部分我们主要考虑非并行的 L1-Norm 问题。关于最近的 L1 算法的综述可以参考 [173][173] A. Y. Yang, A. Ganesh, Z. Zhou, S. S. Sastry, and Y. Ma, “A Review of Fast l1-Minimization Algorithms for Robust Face Recognition,” arXiv:1007.3753, 2010.
6-1 最小绝对偏差(Least Absolute Deviations )
最小化 ||Ax−b||1||Ax − b||_{1} 替代 ||Ax−b||22||Ax − b||^{2}_{2}当数据包含较大的离群点时,Least Absolute Deviations(LAD) 通常比 least squares fitting(LSF) 提供一个更鲁棒的拟合 (Least absolute deviations provides a more robust fit than least squares when the data contains large outliers)。
写成 ADMM 形式,
其中 f=0f = 0 和 g=||⋅||1g = ||· ||_{1}, 假设 ATAA^{T}A 可逆, 有
6-1-1 Huber 拟合(Huber Fitting)
Huber Fitting 位于 least squares 和 least absolute deviations 之间,其中 Huber penalty function ghubg^{hub} :
因此有,
x-update 和 u-update 和 Least absolute deviations 的一样。
6-2 基追踪(Basis Pursuit)
Basis Pursuit 是一个 等式约束的 L1 最小化问题[24] 是关于 Basis Pursuit 的一个综述。
写成 ADMM 形式
其中 f 是
的指示函数。
因此有
其中 Π 是
上的映射。
x-update 展开为
最近提出的 Bregman iterative methods 解决类似 Basis Pursuit 的问题很有效。
For basis pursuit and related problems, Bregman iterative regularization [176] is equivalent to the method of multipliers, and the split Bregman method [88] is equivalent to ADMM [68].
[24] A. M. Bruckstein, D. L. Donoho, and M. Elad, “From sparse solutions of systems of equations to sparse modeling of signals and images,” SIAM Review, vol. 51, no. 1, pp. 34–81, 2009.
[176] W. Yin, S. Osher, D. Goldfarb, and J. Darbon, “Bregman iterative algorithms for l1-minimization with applications to compressed sensing,” SIAM Journal on Imaging Sciences, vol. 1, no. 1, pp. 143–168, 2008.
[88] T. Goldstein and S. Osher, “The split Bregman method for l1 regularized problems,” SIAM Journal on Imaging Sciences, vol. 2, no. 2, pp. 323–343, 2009.
[68] E. Esser, “Applications of Lagrangian-based alternating direction methods and connections to split Bregman,” CAM report, vol. 9, p. 31, 2009.
6-3 广义 L1正则化损失最小化(General L1 Regularized Loss Minimization)
考虑广义的问题其中 l 是任意的凸代价函数。
写成 ADMM 形式
其中
有
In general, we can interpret ADMM for L1 regularized loss minimization as reducing it to solving a sequence of L2 (squared) regularized loss minimization problems.
6-4 Lasso(Lasso)
lasso [156] 是 (6.1)的一个特例,也叫 L1 regularized linear regression。写成 ADMM 形式
其中
有
The x-update is essentially a ridge regression (i.e., quadratically regularized least squares) computation, so ADMM can be interpreted as a method for solving the lasso problem by iteratively carrying out ridge regression.
6-4-1 广义Lasso(Generalized Lasso)
进一步一般化
其中 F 是一个任意的线性变换。
一个特殊的例子是当 F∈R(n−1)×nF ∈ R^{(n−1)×n} 是差异矩阵
和 A =I 时,
写成 ADMM 形式
有
6-4-2 组 Lasso(Group Lasso)
6-5 稀疏逆协方差选择(Sparse Inverse Covariance Selection)
参考或延伸材料:[1] Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers
[2] 凸优化讲义
[3] A Note on the Alternating Direction Method of Multipliers
相关文章推荐
- 网络请求回来json数据时去空格以及标示符
- android系统语言设置对app本身的影响
- 易經大意原稿(頁單位) 051-075
- Objective-C字符串处理
- vmware安装CentOS6.4详细步骤
- Python Web初学解惑之 WSGI、flup、fastcgi、web.py的关系
- 莫队算法
- HTTP协议中PUT和POST使用上的区别
- 随笔2
- Xcode设置项之Architectures和Valid Architectures
- CGI相关介绍
- I2C总线-I2C总线介绍
- HTML+CSS实现导航页
- mapreduce在hive中执行成功,在beeline中执行失败的问题解决
- 如何利用phpize在生产环境中为php添加新的扩展php-bcmath
- jdbc 连接mysql数据库
- CGI简介
- S3C2440 Linux下的I2C驱动以及I2C体系下对EEPROM进行读写操作。
- leetcode 012 —— Integer to Roman
- Visual Studio 快捷键