Maxout Networks

Ian J. Goodfellow goodfeli@iro.umontreal.caDavid Warde-Farley wardefar@iro.umontreal.caMehdi Mirza mirzamom@iro.umontreal.caAaron Courville aaron.courville@umontreal.caYoshua Bengio yoshua.bengio@umontreal.caDepartement d’Informatique et de Recherche Operationelle, Universite de Montreal2920, chemin de la Tour, Montreal, Quebec, Canada, H3T 1J8

AbstractWe consider the problem of designing mod-els to leverage a recently introduced ap-proximate model averaging technique calleddropout. We define a simple new model calledmaxout (so named because its output is themax of a set of inputs, and because it is a nat-ural companion to dropout) designed to bothfacilitate optimization by dropout and im-prove the accuracy of dropout’s fast approxi-mate model averaging technique. We empir-ically verify that the model successfully ac-complishes both of these tasks. We use max-out and dropout to demonstrate state of theart classification performance on four bench-mark datasets: MNIST, CIFAR-10, CIFAR-100, and SVHN.

1. IntroductionDropout (Hinton et al., 2012) provides an inexpensiveand simple means of both training a large ensemble ofmodels that share parameters and approximately aver-aging together these models’ predictions. Dropout ap-plied to multilayer perceptrons and deep convolutionalnetworks has improved the state of the art on tasksranging from audio classification to very large scale ob-ject recognition (Hinton et al., 2012; Krizhevsky et al.,2012). While dropout is known to work well in prac-tice, it has not previously been demonstrated to actu-ally perform model averaging for deep architectures 1 .

Proceedings of the 30 th International Conference on Ma-chine Learning, Atlanta, Georgia, USA, 2013. JMLR:W&CP volume 28. Copyright 2013 by the author(s).

1Between submission and publication of this paper, wehave learned that Srivastava (2013) performed experimentson this subject similar to ours.

Dropout is generally viewed as an indiscriminately ap-plicable tool that reliably yields a modest improvementin performance when applied to almost any model.

We argue that rather than using dropout as a slightperformance enhancement applied to arbitrary mod-els, the best performance may be obtained by directlydesigning a model that enhances dropout’s abilities asa model averaging technique. Training using dropoutdiffers significantly from previous approaches such asordinary stochastic gradient descent. Dropout is mosteffective when taking relatively large steps in param-eter space. In this regime, each update can be seenas making a significant update to a different model ona different subset of the training set. The ideal oper-ating regime for dropout is when the overall trainingprocedure resembles training an ensemble with bag-ging under parameter sharing constraints. This differsradically from the ideal stochastic gradient operatingregime in which a single model makes steady progressvia small steps. Another consideration is that dropoutmodel averaging is only an approximation when ap-plied to deep models. Explicitly designing models tominimize this approximation error may thus enhancedropout’s performance as well.

We propose a simple model that we call maxout thathas beneficial characteristics both for optimization andmodel averaging with dropout. We use this model inconjunction with dropout to set the state of the art onfour benchmark datasets 2 .

2. Review of dropoutDropout is a technique that can be applied to deter-ministic feedforward architectures that predict an out-put y given input vector v. These architectures contain

2Code and hyperparameters available at http://www-etud.iro.umontreal.ca/˜goodfeli/maxout.html

arX

iv:1

302.

4389

v4 [

stat

.ML

] 2

0 Se

p 20

13

Maxout Networks

a series of hidden layers h = {h(1), . . . , h(L)}. Dropouttrains an ensemble of models consisting of the set ofall models that contain a subset of the variables inboth v and h. The same set of parameters θ is usedto parameterize a family of distributions p(y | v; θ, µ)where µ ∈M is a binary mask determining which vari-ables to include in the model. On each presentation ofa training example, we train a different sub-model byfollowing the gradient of log p(y | v; θ, µ) for a differentrandomly sampled µ. For many parameterizations of p(such as most multilayer perceptrons) the instantiationof different sub-models p(y | v; θ, µ) can be obtained byelementwise multiplication of v and h with the maskµ. Dropout training is similar to bagging (Breiman,1994), where many different models are trained on dif-ferent subsets of the data. Dropout training differsfrom bagging in that each model is trained for onlyone step and all of the models share parameters. Forthis training procedure to behave as if it is training anensemble rather than a single model, each update musthave a large effect, so that it makes the sub-model in-duced by that µ fit the current input v well.

The functional form becomes important when it comestime for the ensemble to make a prediction by aver-aging together all the sub-models’ predictions. Mostprior work on bagging averages with the arithmeticmean, but it is not obvious how to do so with theexponentially many models trained by dropout. For-tunately, some model families yield an inexpensive ge-ometric mean. When p(y | v; θ) = softmax(vTW + b),the predictive distribution defined by renormalizingthe geometric mean of p(y | v; θ, µ) over M is simplygiven by softmax(vTW/2+b). In other words, the aver-age prediction of exponentially many sub-models canbe computed simply by running the full model withthe weights divided by 2. This result holds exactlyin the case of a single layer softmax model. Previouswork on dropout applies the same scheme in deeper ar-chitectures, such as multilayer perceptrons, where theW/2 method is only an approximation to the geometricmean. The approximation has not been characterizedmathematically, but performs well in practice.

3. Description of maxoutThe maxout model is simply a feed-forward achitec-ture, such as a multilayer perceptron or deep convo-lutional neural network, that uses a new type of ac-tivation function: the maxout unit. Given an inputx ∈ Rd (x may be v, or may be a hidden layer’s state),a maxout hidden layer implements the function

hi(x) = maxj∈[1,k]

zij

where zij = xTW···ij + bij , and W ∈ Rd×m×k andb ∈ Rm×k are learned parameters. In a convolutionalnetwork, a maxout feature map can be constructedby taking the maximum across k affine feature maps(i.e., pool across channels, in addition spatial loca-tions). When training with dropout, we perform theelementwise multiplication with the dropout mask im-mediately prior to the multiplication by the weights inall cases–we do not drop inputs to the max operator.A single maxout unit can be interpreted as making apiecewise linear approximation to an arbitrary convexfunction. Maxout networks learn not just the rela-tionship between hidden units, but also the activationfunction of each hidden unit. See Fig. 1 for a graphicaldepiction of how this works.

x

hi(x

)

Rectifier

x

hi(x

)

Absolute value

x

hi(x

)

Quadratic

Figure 1. Graphical depiction of how the maxout activa-tion function can implement the rectified linear, absolutevalue rectifier, and approximate the quadratic activationfunction. This diagram is 2D and only shows how max-out behaves with a 1D input, but in multiple dimensions amaxout unit can approximate arbitrary convex functions.

Maxout abandons many of the mainstays of traditionalactivation function design. The representation it pro-duces is not sparse at all (see Fig. 2), though thegradient is highly sparse and dropout will artificiallysparsify the effective representation during training.While maxout may learn to saturate on one side orthe other this is a measure zero event (so it is almostnever bounded from above). While a significant pro-portion of parameter space corresponds to the functionbeing bounded from below, maxout is not constrainedto learn to be bounded at all. Maxout is locally lin-ear almost everywhere, while many popular activationfunctions have signficant curvature. Given all of thesedepartures from standard practice, it may seem sur-prising that maxout activation functions work at all,but we find that they are very robust and easy to trainwith dropout, and achieve excellent performance.

4. Maxout is a universal approximatorA standard MLP with enough hidden units is a uni-versal approximator. Similarly, maxout networks areuniversal approximators. Provided that each individ-ual maxout unit may have arbitrarily many affine com-ponents, we show that a maxout model with just twohidden units can approximate, arbitrarily well, any

Maxout Networks

-4 -2 0 2 4 6Activation

05

101520253035

#of

occu

rren

ces

Histogram of maxout responses

Figure 2. The activations of maxout units are not sparse.

h2h1

g

z2,·z1,·

v

W1 = 1 W2 = −1

Figure 3. An MLP containing two maxout units can arbi-trarily approximate any continuous function. The weightsin the final layer can set g to be the difference of h1 and h2.If z1 and z2 are allowed to have arbitrarily high cardinal-ity, h1 and h2 can approximate any convex function. g canthus approximate any continuous function due to being adifference of approximations of arbitrary convex functions.

continuous function of v ∈ Rn. A diagram illustratingthe basic idea of the proof is presented in Fig. 3.

Consider the continuous piecewise linear (PWL) func-tion g(v) consisting of k locally affine regions on Rn.

Proposition 4.1 (From Theorem 2.1 in (Wang,2004)) For any positive integers m and n, there existtwo groups of n + 1-dimensional real-valued parame-ter vectors [W1j , b1j ], j ∈ [1, k] and [W2j , b2j ], j ∈ [1, k]such that:

g(v) = h1(v)− h2(v) (1)That is, any continuous PWL function can be ex-pressed as a difference of two convex PWL functions.The proof is given in (Wang, 2004).

Proposition 4.2 From the Stone-Weierstrass ap-proximation theorem, let C be a compact domainC ⊂ Rn, f : C → R be a continuous function, andε > 0 be any positive real number. Then there exists acontinuous PWL function g, (depending upon ε), suchthat for all v ∈ C, |f(v)− g(v)| < ε.

Theorem 4.3 Universal approximator theorem. Anycontinuous function f can be approximated arbitrar-ily well on a compact domain C ⊂ Rn by a maxoutnetwork with two maxout hidden units.

Table 1. Test set misclassification rates for the best meth-ods on the permutation invariant MNIST dataset. Onlymethods that are regularized by modeling the input distri-bution outperform the maxout MLP.

Method Test error

Rectifier MLP + dropout (Sri-vastava, 2013)

1.05%

DBM (Salakhutdinov & Hin-ton, 2009)

0.95%

Maxout MLP + dropout 0.94%MP-DBM (Goodfellow et al.,2013)

0.91%

Deep Convex Network (Yu &Deng, 2011)

0.83%

Manifold Tangent Classifier(Rifai et al., 2011)

0.81%

DBM + dropout (Hintonet al., 2012)

0.79%

Sketch of Proof By Proposition 4.2, any continuousfunction can be approximated arbitrarily well (up toε), by a piecewise linear function. We now note thatthe representation of piecewise linear functions givenin Proposition 4.1 exactly matches a maxout networkwith two hidden units h1(v) and h2(v), with suffi-ciently large k to achieve the desired degree of approx-imation ε. Combining these, we conclude that a twohidden unit maxout network can approximate any con-tinuous function f(v) arbitrarily well on the compactdomain C. In general as ε→ 0, we have k →∞.

Figure 4. Example filters learned by a maxout MLPtrained with dropout on MNIST. Each row contains thefilters whose responses are pooled to form a maxout unit.

5. Benchmark resultsWe evaluated the maxout model on four benchmarkdatasets and set the state of the art on all of them.

Maxout Networks

Table 2. Test set misclassification rates for the best meth-ods on the general MNIST dataset, excluding methods thataugment the training data.

Method Test error

2-layer CNN+2-layer NN (Jar-rett et al., 2009)

0.53%

Stochastic pooling (Zeiler &Fergus, 2013)

0.47%

Conv. maxout + dropout 0.45%

5.1. MNIST

The MNIST (LeCun et al., 1998) dataset consists of 28× 28 pixel greyscale images of handwritten digits 0-9,with 60,000 training and 10,000 test examples. For thepermutation invariant version of the MNIST task, onlymethods unaware of the 2D structure of the data arepermitted. For this task, we trained a model consist-ing of two densely connected maxout layers followedby a softmax layer. We regularized the model withdropout and by imposing a constraint on the norm ofeach weight vector, as in (Srebro & Shraibman, 2005).Apart from the maxout units, this is the same archi-tecture used by Hinton et al. (2012). We selected thehyperparameters by minimizing the error on a valida-tion set consisting of the last 10,000 training examples.To make use of the full training set, we recorded thevalue of the log likelihood on the first 50,000 exam-ples at the point of minimal validation error. We thencontinued training on the full 60,000 example train-ing set until the validation set log likelihood matchedthis number. We obtained a test set error of 0.94%,which is the best result we are aware of that does notuse unsupervised pretraining. We summarize the bestpublished results on permutation invariant MNIST inTable 1.

We also considered the MNIST dataset without thepermutation invariance restriction. In this case, weused three convolutional maxout hidden layers (withspatial max pooling on top of the maxout layers) fol-lowed by a densely connected softmax layer. We wereable to rapidly explore hyperparameter space thanksto the extremely fast GPU convolution library devel-oped by Krizhevsky et al. (2012). We obtained a testset error rate of 0.45%, which sets a new state of theart in this category. (It is possible to get better resultson MNIST by augmenting the dataset with transfor-mations of the standard set of images (Ciresan et al.,2010) ) A summary of the best methods on the generalMNIST dataset is provided in Table 2.

Table 3. Test set misclassification rates for the best meth-ods on the CIFAR-10 dataset.

Method Test error

Stochastic pooling (Zeiler &Fergus, 2013)

15.13%

CNN + Spearmint (Snoeket al., 2012)

14.98%

Conv. maxout + dropout 11.68 %CNN + Spearmint + data aug-mentation (Snoek et al., 2012)

9.50 %

Conv. maxout + dropout +data augmentation

9.38 %

5.2. CIFAR-10

The CIFAR-10 dataset (Krizhevsky & Hinton, 2009)consists of 32 × 32 color images drawn from 10 classessplit into 50,000 train and 10,000 test images. We pre-process the data using global contrast normalizationand ZCA whitening.

We follow a similar procedure as with the MNISTdataset, with one change. On MNIST, we find thebest number of training epochs in terms of validationset error, then record the training set log likelihoodand continue training using the entire training set un-til the validation set log likelihood has reached thisvalue. On CIFAR-10, continuing training in this fash-ion is infeasible because the final value of the learn-ing rate is very small and the validation set error isvery high. Training until the validation set likelihoodmatches the cross-validated value of the training like-lihood would thus take prohibitively long. Instead, weretrain the model from scratch, and stop when the newlikelihood matches the old one.

Our best model consists of three convolutional maxoutlayers, a fully connected maxout layer, and a fully con-nected softmax layer. Using this approach we obtain atest set error of 11.68%, which improves upon the stateof the art by over two percentage points. (If we do nottrain on the validation set, we obtain a test set errorof 13.2%, which also improves over the previous stateof the art). If we additionally augment the data withtranslations and horizontal reflections, we obtain theabsolute state of the art on this task at 9.35% error.In this case, the likelihood during the retrain neverreaches the likelihood from the validation run, so weretrain for the same number of epochs as the valida-tion run. A summary of the best CIFAR-10 methodsis provided in Table 3.

Maxout Networks

0.0 0.2 0.4 0.6 0.8 1.0 1.2

# examples ×107

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Err

or

CIFAR-10 validation error with and without droput

Dropout/ValidationNo dropout/ValidationDropout/TrainNo dropout/Train

Figure 5. When training maxout, the improvement in val-idation set error that results from using dropout is dra-matic. Here we find a greater than 25% reduction in ourvalidation set error on CIFAR-10.Table 4. Test set misclassification rates for the best meth-ods on the CIFAR-100 dataset.

Method Test error

Learned pooling (Malinowski& Fritz, 2013)

43.71%

Stochastic pooling(Zeiler &Fergus, 2013)

42.51%

Conv. maxout + dropout 38.57%

5.3. CIFAR-100

The CIFAR-100 (Krizhevsky & Hinton, 2009) datasetis the same size and format as the CIFAR-10 dataset,but contains 100 classes, with only one tenth as manylabeled examples per class. Due to lack of time wedid not extensively cross-validate hyperparameters onCIFAR-100 but simply applied hyperparameters wefound to work well on CIFAR-10. We obtained a testset error of 38.57%, which is state of the art. If we donot retrain using the entire training set, we obtain atest set error of 41.48%, which also surpasses the cur-rent state of the art. A summary of the best methodson CIFAR-100 is provided in Table 4.

5.4. Street View House Numbers

The SVHN (Netzer et al., 2011) dataset consists ofcolor images of house numbers collected by GoogleStreet View. The dataset comes in two formats. Weconsider the second format, in which each image is ofsize 32 × 32 and the task is to classify the digit inthe center of the image. Additional digits may appearbeside it but must be ignored. There are 73,257 digitsin the training set, 26,032 digits in the test set and531,131 additional, somewhat less difficult examples,

Table 5. Test set misclassification rates for the best meth-ods on the SVHN dataset.

Method Test error

(Sermanet et al., 2012a) 4.90%Stochastic pooling (Zeiler &Fergus, 2013)

2.80 %

Rectifiers + dropout (Srivas-tava, 2013)

2.78 %

Rectifiers + dropout +synthetic translation (Srivas-tava, 2013)

2.68 %

Conv. maxout + dropout 2.47 %

to use as an extra training set. Following Sermanetet al. (2012b), to build a validation set, we select 400samples per class from the training set and 200 sam-ples per class from the extra set. The remaining digitsof the train and extra sets are used for training.

For SVHN, we did not train on the validation set atall. We used it only to find the best hyperparameters.We applied local contrast normalization preprocessingthe same way as Zeiler & Fergus (2013). Otherwise,we followed the same approach as on MNIST. Our bestmodel consists of three convolutional maxout hiddenlayers and a densely connected maxout layer followedby a densely connected softmax layer. We obtained atest set error rate of 2.47%, which sets the state of theart. A summary of comparable methods is providedin Table 5.

6. Comparison to rectifiersOne obvious question about our results is whether weobtained them by improved preprocessing or largermodels, rather than by the use of maxout. For MNISTwe used no preprocessing, and for SVHN, we use thesame preprocessing as Zeiler & Fergus (2013). How-ever on the CIFAR datasets we did use a new form ofpreprocessing. We therefore compare maxout to rec-tifiers run with the same processing and a variety ofmodel sizes on this dataset.

By running a large cross-validation experiment (seeFig. 6) we found that maxout offers a clear im-provement over rectifiers. We also found that ourpreprocessing and size of models improves rectifiersand dropout beyond the previous state of the art re-sult. Cross-channel pooling is a method for reducingthe size of state and number of parameters needed tohave a given number of filters in the model. Perfor-mance seems to correlate well with the number of fil-ters for maxout but with the number of output unitsfor rectifiers–i.e, rectifier units do not benefit much

Maxout Networks

from cross-channel pooling. Rectifier units do bestwithout cross-channel pooling but with the same num-ber of filters, meaning that the size of the state and thenumber of parameters must be about k times higherfor rectifiers to obtain generalization performance ap-proaching that of maxout.

0 100 200 300 400 500 600 700 800

training epochs

0.125

0.130

0.135

0.140

0.145

0.150

0.155

0.160

valid

atio

nse

terr

orfo

rbes

texp

erim

ent

Comparison of large rectifier networks to maxout

MaxoutRectifier, no channel poolingRectifier + channel poolingLarge rectifier, no channel pooling

Figure 6. We cross-validated the momentum and learn-ing rate for four architectures of model: 1) Medium-sizedmaxout network. 2) Rectifier network with cross-channelpooling, and exactly the same number of parameters andunits as the maxout network. 3) Rectifier network withoutcross-channel pooling, and the same number of units asthe maxout network (thus fewer parameters). 4) Rectifiernetwork without cross-channel pooling, but with k timesas many units as the maxout network. Because makinglayer i have k times more outputs increases the numberof inputs to layer i + 1, this network has roughly k timesmore parameters than the maxout network, and requiressignificantly more memory and runtime. We sampled 10learning rate and momentum schedules and random seedsfor dropout, then ran each configuration for all 4 architec-tures. Each curve terminates after failing to improve thevalidation error in the last 100 epochs.

7. Model averagingHaving demonstrated that maxout networks are effec-tive models, we now analyze the reasons for their suc-cess. We first identify reasons that maxout is highlycompatible with dropout’s approximate model averag-ing technique.

The intuitive justification for averaging sub-models bydividing the weights by 2 given by (Hinton et al., 2012)is that this does exact model averaging for a singlelayer model, softmax regression. To this characteriza-tion, we add the observation that the model averagingremains exact if the model is extended to multiple lin-ear layers. While this has the same representationalpower as a single layer, the expression of the weightsas a product of several matrices could have a differ-

100 101 102 103

# samples

0.008

0.010

0.012

0.014

0.016

0.018

0.020

0.022

0.024

0.026

Test

erro

r

Model averaging: MNIST classification

Sampling, maxoutSampling, tanhW/2, maxoutW/2, tanh

Figure 7. The error rate of the prediction obtained by sam-pling several sub-models and taking the geometric mean oftheir predictions approaches the error rate of the predic-tion made by dividing the weights by 2. However, thedivided weights still obtain the best test error, suggestingthat dropout is a good approximation to averaging over avery large number of models. Note that the correspondenceis more clear in the case of maxout.

ent inductive bias. More importantly, it indicates thatdropout does exact model averaging in deeper archi-tectures provided that they are locally linear amongthe space of inputs to each layer that are visited byapplying different dropout masks.

We argue that dropout training encourages maxoutunits to have large linear regions around inputs thatappear in the training data. Because each sub-modelmust make a good prediction of the output, each unitshould learn to have roughly the same activation re-gardless of which inputs are dropped. In a maxoutnetwork with arbitrarily selected parameters, varyingthe dropout mask will often move the effective in-puts far enough to escape the local region surround-ing the clean inputs in which the hidden units are lin-ear, i.e., changing the dropout mask could frequentlychange which piece of the piecewise function an inputis mapped to. Maxout trained with dropout may havethe identity of the maximal filter in each unit changerelatively rarely as the dropout mask changes. Net-works of linear operations and max(·) may learn toexploit dropout’s approximate model averaging tech-nique well.

Many popular activation functions have significantcurvature nearly everywhere. These observations sug-gest that the approximate model averaging of dropoutwill not be as accurate for networks incorporating suchactivation functions. To test this, we compared thebest maxout model trained on MNIST with dropoutto a hyperbolic tangent network trained on MNIST

Maxout Networks

100 101 102 103

# samples

0.0004

0.0006

0.0008

0.0010

0.0012

0.0014

0.0016

0.0018

KL

dive

rgen

ce

KL divergence between model averaging strategies

MaxoutTanh

Figure 8. The KL divergence between the distribution pre-dicted using the dropout technique of dividing the weightsby 2 and the distribution obtained by taking the geomet-ric mean of the predictions of several sampled models de-creases as the number of samples increases. This suggeststhat dropout does indeed do model averaging, even for deepnetworks. The approximation is more accurate for maxoutunits than for tanh units.with dropout. We sampled several subsets of eachmodel and compared the geometric mean of these sam-pled models’ predictions to the prediction made usingthe dropout technique of dividing the weights by 2.We found evidence that dropout is indeed performingmodel averaging, even in multilayer networks, and thatit is more accurate in the case of maxout. See Fig. 7and Fig. 8 for details.

8. OptimizationThe second key reason that maxout performs well isthat it improves the bagging style training phase ofdropout. Note that the arguments in section 7 moti-vating the use of maxout also apply equally to rectifiedlinear units (Salinas & Abbott, 1996; Hahnloser, 1998;Glorot et al., 2011). The only difference between max-out and max pooling over a set of rectified linear unitsis that maxout does not include a 0 in the max. Su-perficially, this seems to be a small difference, but wefind that including this constant 0 is very harmful tooptimization in the context of dropout. For instance,on MNIST our best validation set error with an MLPis 1.04%. If we include a 0 in the max, this rises toover 1.2%. We argue that, when trained with dropout,maxout is easier to optimize than rectified linear unitswith cross-channel pooling.

8.1. Optimization experiments

To verify that maxout yields better optimization per-formance than max pooled rectified linear units when

training with dropout, we carried out two experiments.First, we stressed the optimization capabilities of thetraining algorithm by training a small (two hiddenconvolutional layers with k = 2 and sixteen kernels)model on the large (600,000 example) SVHN dataset.When training with rectifier units the training errorgets stuck at 7.3%. If we train instead with max-out units, we obtain 5.1% training error. As anotheroptimization stress test, we tried training very deepand narrow models on MNIST, and found that max-out copes better with increasing depth than pooledrectifiers. See Fig. 9 for details.

1 2 3 4 5 6 7

# layers

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

Err

or

MNIST classification error versus network depth

Maxout test errorRectifier test errorMaxout train errorRectifier train error

Figure 9. We trained a series of models with increasingdepth on MNIST. Each layer contains only 80 units (k=5)to make fitting the training set difficult. Maxout optimiza-tion degrades gracefully with depth but pooled rectifierunits worsen noticeably at 6 layers and dramatically at 7.

8.2. Saturation

20 30 40 50 60 70 80 90 100

# epochs

0.00

0.05

0.10

0.15

0.20

0.25

Pro

port

ion

ofsi

gnth

atsw

itche

sth

isep

och

Training set h0 activation sign switches/epoch

Maxout: pos→negMaxout: neg→posPooled rect: pos→zeroPooled rect: zero→pos

Figure 10. During dropout training, rectifier units transi-tion from positive to 0 activation more frequently than theymake the opposite transition, resulting a preponderence of0 activations. Maxout units freely move between positiveand negative signs at roughly equal rates.

Maxout Networks

Optimization proceeds very differently when usingdropout than when using ordinary stochastic gradi-ent descent. SGD usually works best with a smalllearning rate that results in a smoothly decreasingobjective function, while dropout works best with alarge learning rate, resulting in a constantly fluctuat-ing objective function. Dropout rapidly explores manydifferent directions and rejects the ones that worsenperformance, while SGD moves slowly and steadilyin the most promising direction. We find empiricallythat these different operating regimes result in differ-ent outcomes for rectifier units. When training withSGD, we find that the rectifier units saturate at 0 lessthan 5% of the time. When training with dropout, weinitialize the units to sature rarely but training grad-ually increases their saturation rate to 60%. Becausethe 0 in the max(0, z) activation function is a con-stant, this blocks the gradient from flowing throughthe unit. In the absence of gradient through the unit,it is difficult for training to change this unit to be-come active again. Maxout does not suffer from thisproblem because gradient always flows through everymaxout unit–even when a maxout unit is 0, this 0 is afunction of the parameters and may be adjusted Unitsthat take on negative activations may be steered tobecome positive again later. Fig. 10 illustrates howactive rectifier units become inactive at a greater ratethan inactive units become active when training withdropout, but maxout units, which are always active,transition between positive and negative activations atabout equal rates in each direction. We hypothesizethat the high proportion of zeros and the difficulty ofescaping them impairs the optimization performanceof rectifiers relative to maxout.

To test this hypothesis, we trained two MLPs onMNIST, both with two hidden layers and 1200 fil-ters per layer pooled in groups of 5. When we in-clude a constant 0 in the max pooling, the resultingtrained model fails to make use of 17.6% of the filtersin the second layer and 39.2% of the filters in the sec-ond layer. A small minority of the filters usually tookon the maximal value in the pool, and the rest of thetime the maximal value was a constant 0. Maxout, onthe other hand, used all but 2 of the 2400 filters inthe network. Each filter in each maxout unit in thenetwork was maximal for some training example. Allfilters had been utilised and tuned.

8.3. Lower layer gradients and bagging

To behave differently from SGD, dropout requires thegradient to change noticeably as the choice of whichunits to drop changes. If the gradient is approximatelyconstant with respect to the dropout mask, then

dropout simplifies to SGD training. We tested thehypothesis that rectifier networks suffer from dimin-ished gradient flow to the lower layers of the networkby monitoring the variance with respect to dropoutmasks for fixed data during training of two differentMLPs on MNIST. The variance of the gradient on theoutput weights was 1.4 times larger for maxout on anaverage training step, while the variance on the gradi-ent of the first layer weights was 3.4 times larger formaxout than for rectifiers. Combined with our previ-ous result showing that maxout allows training deepernetworks, this greater variance suggests that maxoutbetter propagates varying information downward tothe lower layers and helps dropout training to betterresemble bagging for the lower-layer parameters. Rec-tifier networks, with more of their gradient lost to satu-ration, presumably cause dropout training to resembleregular SGD toward the bottom of the network.

9. ConclusionWe have proposed a new activation function calledmaxout that is particularly well suited for trainingwith dropout, and for which we have proven a univer-sal approximation theorem. We have shown empiricalevidence that dropout attains a good approximationto model averaging in deep models. We have shownthat maxout exploits this model averaging behaviorbecause the approximation is more accurate for max-out units than for tanh units. We have demonstratedthat optimization behaves very differently in the con-text of dropout than in the pure SGD case. By de-signing the maxout gradient to avoid pitfalls such asfailing to use many of a model’s filters, we are able totrain deeper networks than is possible using rectifierunits. We have also shown that maxout propagatesvariations in the gradient due to different choices ofdropout masks to the lowest layers of a network, en-suring that every parameter in the model can enjoythe full benefit of dropout and more faithfully emulatebagging training. The state of the art performance ofour approach on five different benchmark tasks moti-vates the design of further models that are explicitlyintended to perform well when combined with inex-pensive approximations to model averaging.

AcknowledgementsThe authors would like to thank the developers ofTheano (Bergstra et al., 2010; Bastien et al., 2012),in particular Frederic Bastien and Pascal Lamblin fortheir assistance with infrastructure development andperformance optimization. We would also like to thankYann Dauphin for helpful discussions.

Maxout Networks

ReferencesBastien, Frederic, Lamblin, Pascal, Pascanu, Razvan,

Bergstra, James, Goodfellow, Ian, Bergeron, Arnaud,Bouchard, Nicolas, and Bengio, Yoshua. Theano: newfeatures and speed improvements. Deep Learning andUnsupervised Feature Learning NIPS 2012 Workshop,2012.

Bergstra, James, Breuleux, Olivier, Bastien, Frederic,Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guil-laume, Turian, Joseph, Warde-Farley, David, and Ben-gio, Yoshua. Theano: a CPU and GPU math expressioncompiler. In Proceedings of the Python for ScientificComputing Conference (SciPy), June 2010. Oral Pre-sentation.

Breiman, Leo. Bagging predictors. Machine Learning, 24(2):123–140, 1994.

Ciresan, D. C., Meier, U., Gambardella, L. M., andSchmidhuber, J. Deep big simple neural nets for hand-written digit recognition. Neural Computation, 22:1–14,2010.

Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua.Deep sparse rectifier neural networks. In JMLR W&CP:Proceedings of the Fourteenth International Confer-ence on Artificial Intelligence and Statistics (AISTATS2011), April 2011.

Goodfellow, Ian J., Courville, Aaron, and Bengio, Yoshua.Joint training of deep Boltzmann machines for classifi-cation. In International Conference on Learning Repre-sentations: Workshops Track, 2013.

Hahnloser, Richard H. R. On the piecewise analysis ofnetworks of linear threshold neurons. Neural Networks,11(4):691–697, 1998.

Hinton, Geoffrey E., Srivastava, Nitish, Krizhevsky, Alex,Sutskever, Ilya, and Salakhutdinov, Ruslan. Improvingneural networks by preventing co-adaptation of featuredetectors. Technical report, arXiv:1207.0580, 2012.

Jarrett, Kevin, Kavukcuoglu, Koray, Ranzato,Marc’Aurelio, and LeCun, Yann. What is the bestmulti-stage architecture for object recognition? InProc. International Conference on Computer Vision(ICCV’09), pp. 2146–2153. IEEE, 2009.

Krizhevsky, Alex and Hinton, Geoffrey. Learning multiplelayers of features from tiny images. Technical report,University of Toronto, 2009.

Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey.ImageNet classification with deep convolutional neuralnetworks. In Advances in Neural Information ProcessingSystems 25 (NIPS’2012). 2012.

LeCun, Yann, Bottou, Leon, Bengio, Yoshua, and Haffner,Patrick. Gradient-based learning applied to documentrecognition. Proceedings of the IEEE, 86(11):2278–2324,November 1998.

Malinowski, Mateusz and Fritz, Mario. Learnable poolingregions for image classification. In International Con-ference on Learning Representations: Workshop track,2013.

Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B.,and Ng, A. Y. Reading digits in natural images withunsupervised feature learning. Deep Learning and Un-supervised Feature Learning Workshop, NIPS, 2011.

Rifai, Salah, Dauphin, Yann, Vincent, Pascal, Bengio,Yoshua, and Muller, Xavier. The manifold tangent clas-sifier. In NIPS’2011, 2011. Student paper award.

Salakhutdinov, R. and Hinton, G.E. Deep Boltzmannmachines. In Proceedings of the Twelfth InternationalConference on Artificial Intelligence and Statistics (AIS-TATS 2009), volume 8, 2009.

Salinas, E. and Abbott, L. F. A model of multiplicativeneural responses in parietal cortex. Proc Natl Acad SciU S A, 93(21):11956–11961, October 1996.

Sermanet, Pierre, Chintala, Soumith, and LeCun, Yann.Convolutional neural networks applied to house numbersdigit classification. CoRR, abs/1204.3968, 2012a.

Sermanet, Pierre, Chintala, Soumith, and LeCun, Yann.Convolutional neural networks applied to house num-bers digit classification. In International Conference onPattern Recognition (ICPR 2012), 2012b.

Snoek, Jasper, Larochelle, Hugo, and Adams,Ryan Prescott. Practical bayesian optimization ofmachine learning algorithms. In Neural InformationProcessing Systems, 2012.

Srebro, Nathan and Shraibman, Adi. Rank, trace-normand max-norm. In Proceedings of the 18th AnnualConference on Learning Theory, pp. 545–560. Springer-Verlag, 2005.

Srivastava, Nitish. Improving neural networks withdropout. Master’s thesis, U. Toronto, 2013.

Wang, Shuning. General constructive representations forcontinuous piecewise-linear functions. IEEE Trans. Cir-cuits Systems, 51(9):1889–1896, 2004.

Yu, Dong and Deng, Li. Deep convex net: A scalable ar-chitecture for speech pattern classification. In INTER-SPEECH, pp. 2285–2288, 2011.

Zeiler, Matthew D. and Fergus, Rob. Stochastic pooling forregularization of deep convolutional neural networks. InInternational Conference on Learning Representations,2013.