Fast l-1 MinimizationAlgorithms

Fast l-1 MinimizationAlgorithms: Homotopy and Augmented Lagrangian Method
-- Implementationfrom Fixed-Point MPUs to Many-Core CPUs/GPUs
 

Allen Y. Yang, ArvindGanesh, Zihan Zhou,

Andrew Wagner, VictorShia, Shankar Sastry, and Yi Ma   

 

© Copyright Notice:It is important that you read and understand the copyright of the followingsoftware packages as specified in the individual items. The copyright varieswith each package due to its author(s). The packages should
NOT be used for anycommercial purposes without direct consent of their author(s).

This project ispartially supported by NSF TRUST Centerat UC Berkeley, ARO MURI W911NF-06-1-0076, ARLMAST-CTA W911NF-08-2-0004.

Publications

1.Allen Yang, ArvindGanesh, Zihan Zhou, Shankar Sastry, and Yi Ma. A Review of Fast 11-MinimizationAlgorithms for Robust Face Recognition. (preprint)

2.Allen Yang, ArvindGanesh, Shankar Sastry, and Yi Ma.

Fast l1-Minimization Algorithms and AnApplication in Robust Face Recognition: A Review. ICIP 2010.

3.Victor Shia, AllenYang, Shankar Sastry, Andrew Wagner, and Yi Ma.

Fast l1-Minimization andParallelization for Face Recognition. Asilomar 2011.

MATLAB BenchmarkScripts

•L-1 Benchmark Package: http://www.eecs.berkeley.edu/~yang/software/l1benchmark/l1benchmark.zip  

The package containsa consolidated implementation of nine l-1 minimization algorithms in MATLAB.Each function uses a consistent set of parameters (e.g., stopping criterion andtolerance) to interface with our benchmark scripts.

1.Orthogonal MatchingPursuit: SolveOMP.m

2.Primal-DualInterior-Point Method: SolveBP.m

3.GradientProjection: SolveL1LS.m

4.Homotopy:SolveHomotopy.m

5.Polytope FacesPursuit: SolvePFP.m

6.IterativeThresholding: SolveSpaRSA.m

7.Proximal Gradient:SolveFISTA.m

8.Primal AugmentedLagrange Multiplier: SolvePALM.m

9.Dual AugmentedLagrange Multiplier: SolveDALM.m; SolveDALM_fast.m

 

The package alsocontains a script to generate the synthetic data shown in the paper [1].

Note:

1.To run thealternating direction method (YALL1), one needs to separately download thepackage from its authors (following the link at the end of the page).

2.Please properlyacknowledge the respective authors in your publications when you use thispackage.

 

Single-Core l-1Minimization Library in C

•Homotopy and ALMalgorithms implemented in C with MATLAB wrapper:http://www.eecs.berkeley.edu/~yang/software/l1benchmark/L1-Homotopy-ALM.zip

 

Fixed-Point l-1Minimization for Mobile Platforms

•Fixed-point Homotopyalgorithm implemented in Java:http://www.eecs.berkeley.edu/~yang/software/l1benchmark/fixed_point_homotopy_java.zip

     

Many-Core l-1Minimization Library in C/CUDA

•Coming soon ...

 

Benchmark Results

 

Simulations

 

•Noiseless Delta-RhoPlot at 95% Confidence

 

The delta-rho plotmeasures the percentage of successes to recover a sparse signal at pairs of(delta, rho) combinations, where delta=d/n is the sampling rate and rho=k/n isthe sparsity rate. Then a fixed success rate of 95% over
all delta's can beinterpolated as a curve in the plot, as shown on the left.In general, thehigher the success rates, the better an algorthm recovers dense signals in thel-1 problem.

 

Observations:

1.Without concernsabout speed and data noise, the success rate of the interior-point method PDIPAis the highest of all the algorithms in the figure, especially when the signalbecomes dense.

2.The success ratesof L1LS and Homotopy are similar, and they are very close to those of PDIPA.

3.The success ratesof FISTA and DALM are comparable over all sampling rates. The performance alsoshows significant improvement over the IST algorithm, namely, SpaRSA.

     

 

•Fixed Low SparsitySimulation (only speed is shown here)

 

The figure on theleft shows the average run time over various projection dimensions d, where theambient dimension is n=2000. A low sparsity is fixed at k=200.

 

Observations:

1.The computationalcomplexity of PDIPA grows much faster than the other algorithms. Moreimportantly, in contrast to its noise-free performance, the estimation erroralso grows exponentially, in which case the algorithm fails to
converge to anestimate that is close to the ground truth (please refer to the paper).

2.L1LS and Homotopytake much longer time to converge than SpaRSA, FISTA, and DALM.

3.The average runtime of DALM is the smallest over all projection dimensions.

     

 

•Fixed High SamplingRate Simulation (only speed is shown here)

 

The figure on theleft shows the average run time over various sparsity ratios rho, where theambient dimension is again n=2000. A high sampling rate is fixed at d=1500.

 

Observations:

1.Again, PDIPAsignificantly underperforms compared to the rest five algorithms in terms ofboth accuracy and speed.

2.The average runtime of Homotopy grows almost linearly with the sparsity ratio, while the otheralgorithms are relatively unaffected. Thus, Homotopy is more suitable forscenarios where the unknown signal is expected to have a
very small sparsityratio.

3.DALM again is thefastest algorithm compared to SpaRSA and FISTA.

     

 

Robust FaceRecognition

Under Construction...

The CMU Multi-PIEdatabase can be purchased fromhere:http://cmu.wellspringsoftware.net/invention/detail/2309/

 

•SolvingCross-and-Bouquet Model in Robust Face Recognition

This experimenentselects 249 subjects from Multi-PIE, chooses 7 extreme illumination conditionsas the training images. The the testing images are corrupted at random pixelcoordinates from 0% to 90%. We measure the average classification
rate and thespeed under different corruption percentages.

 

Observations:

1.In terms ofaccuracy, Homotopy achieves the best overall performance. The performance ofPDIPA is very close to Homotopy, achieving the second best overall accuracy. Onthe other hand, FISTA obtains the lowest recognition rates.

2.In terms of speed,Homotopy is also one of the fastest algorithm, especially when the pixelcorruption percentage is small.

 

 

Other Public l-1Minimization Libraries

•     SparseLab:http://sparselab.stanford.edu/

o    Orthogonal Matching Pursuit (OMP):SolveOMP

o    Primal-Dual Basis Pursuit (BP): SolveBP.m

o    Lasso : SolveLasso.m

o    Polytope Faces Pursuite (PFP): SolvePFP.m

•    l1magic: http://www.acm.caltech.edu/l1magic/
o    Primal-Dual Basis Pursuit (BP): l1eq_pd.m

•    L1LS: http://www.stanford.edu/boyd/l1_ls/
o    Truncated Newton Interior-Point Method:l1_ls.m

•    GPSR: http://www.lx.it.pt/~mtf/GPSR/
o    Gradient Projection Sparse Representation:GPSR_BB

•    l1-Homotopy:http://users.ece.gatech.edu/~sasif/homotopy/

o    Homotopy Method: BPDN_homotopy_function.m

•    SpaRSA: http://www.lx.it.pt/~mtf/SpaRSA/
o    Iterative Shrinkage-Thresholding Algorithm:SpaRSA.m

•    FISTA:http://www.eecs.berkeley.edu/~yang/software/l1benchmark/

o    Fast IST Algorithm: SolveFISTA.m

•    FISTA for wavelet-based denoising:

o    http://iew3.technion.ac.il/~becka/papers/wavelet_FISTA.zip
•    NESTA: http://www.acm.caltech.edu/~nesta/
o    Nesterov's Algorithm: NESTA.m

•    YALL1: http://www.caam.rice.edu/~optimization/L1/YALL1/
o    Alternating Direction Method: yall1.m

•    Bregman Iterative Regularization:http://www.caam.rice.edu/~optimization/L1/bregman/

o    Fixed-Point Continuation and Active Set:FPC_AS.m