Optimization Methods for Tensor Toolbox

Most MATLAB optimization methods have different interfaces or none at all. In Tensor Toolbox, we are adopting wrappers for moderate consistency, and these will be called from within other Tensor Toolbox functions. Here we outline the choices, their installation instructions, and then the user-tunable parameters.

A few general notes:

For more information on the details of these methods, see Developer Information for Optimization Methods in Tensor Toolbox.

Contents

lbfgsb: Limited-Memory Quasi-Newton with Bound Constraints

In most methods, setting 'opt' to 'lbfgsb' will enable this method, and then any of the settings in the table below can be modified by passing these as additional options to the method.

NameDescriptionDefault
lowerLower bounds, can be vector or scalar-Inf
upperUpper bounds, can be vector or scalar+Inf
maxitersMax outer iterations1000
printitnPrinting frequency by iteration (0=no output)1
mLimited-memory parameter5
subitersControls maximum calls to function-gradient evalations10
ftolStopping condition based on relative function change1e-10
gtolStopping condition based on gradient norm1e-5
mdesc (*)Method description printed out before run'L-BFGS-B Optimization'
xdesc (*)Variable descriptionauto-generated

Installation Instructions. Download and install L-BFGS-B by Stephen Becker. Please see that web page for full details on references, installation, etc. Here we provide cursory instructions for installation:

  1. Download the zip file https://github.com/stephenbeckr/L-BFGS-B-C/archive/master.zip
  2. Unzip and goto the Matlab/ subdirectoy with MATLAB
  3. Type compile_mex
  4. Add this directory to your saved path!

Detailed notes. The wrapper for this method is tt_lbfgsb in the Tensor Toolbox. Notes regarding mappings to the parameters of Becker's L-BFGS-B code:

lbfgs: Limited-Memory Quasi-Newton

In most methods, setting 'opt' to 'lbfgs' will enable this method, and then any of the settings in the table below can be modified by passing these as additional options to the method.

NameDescriptionDefault
maxitersMax outer iterations1000
printitnPrinting frequency by iteration (0=no output)1
mLimited-memory parameter5
subitersControls maximum calls to function-gradient evalations10
ftolStopping condition based on relative function change1e-10
gtolStopping condition based on gradient norm1e-5
mdesc (*)Method description printed out before run'Poblano L-BFGS Optimization'
xdesc (*)Variable descriptionauto-generated

Installation Instructions. Download and install Poblano Toolbox, v1.2. Please see that web page for full details on references, installation, etc. Here we provide cursory instructions for installation:

  1. Download the zip file https://github.com/sandialabs/poblano_toolbox/archive/v1.2.zip
  2. Unzip and goto the poblano_toolbox-1.2/ subdirectoy within MATLAB
  3. Type install_poblano to save this directory to your path

Detailed notes. The wrapper for this method is tt_lbfgs in the Tensor Toolbox. Notes regarding mappings to the parameters of Poblano's L-BFGS code:

fminunc: Optimizaton Toolbox Quasi-Newton Method

In most methods, setting 'opt' to 'fminunc' will enable this method, and then any of the settings in the table below can be modified by passing these as additional options to the method. This requires the MATLAB Optimization Toolbox.

NameDescriptionDefault
maxitersMax outer iterations1000
printitnDisplay (0=no output)1
subitersControls maximum calls to function-gradient evalations10
gtolStopping condition based on gradient norm1e-5
mdesc (*)Method description printed out before run'Quasi-Newton Optimization (via Optimization Toolbox)'
xdesc (*)Variable descriptionauto-generated

adam: Stochastic Gradient Descent with Momentum

In most methods, setting 'opt' to 'adam' will enable this method, and then any of the settings in the table below can be modified by passing these as additional options to the method.

This is our own implementation of Adam. A failed epoch is one where the function value does not decrease. After a failed epoch, the method either reduces the learning rate (by decay) or exits (once the number of failed epochs exceeds maxfails).

NameDescriptionDefault
lowerLower bounds, can be vector or scalar-Inf
subitersNumber of iterations per epoch100
maxitersMaximum number of epochs100
rateInitial learning rate1e-2
maxfailsMaximum number of failed epochs1
decayDecay of learning rate after failed epoch0.1
backupRevert to end of previous epoch after failuretrue
ftoluit if function value goes below this value-Inf
beta1Adam parameter0.9
beta2Adam parameter0.999
epsilonAdam parameter1e-8
printitnPrinting frequency by epoch (0=no output)1
state (*)State of random number generatorcurrent state
mdesc (*)Method description printed out before run'Adam Stochastic Optimization'
xdesc (*)Variable descriptionauto-generated
fdesc (*)Description of (approximate) function computationnone
gdesc (*)Description of stochastic gradient computationnone
fexact (*)Boolean if function is computed exactlytrue