Hey, nice to make your acquaintance, and thanks for offering your contact info. Mine can be backed out my HN profile as well. I actually work in Menlo Park; the AI group is split across New York and Menlo Park, with a very small European contingent for now.
WRT the Glorot and Bengio paper, itβs true that there was a lot of excitement surrounding unsupervised pre-training of DNNs, but this mostly preceded the current wave of successes. The big differences between the architectures that are working on image processing today and that paper are:
1. Moar data. The datasets this paper was looking at were on the order 10^5 or so images. 10^6 is a different ballgame.
2. Convolution. Sharing weights really is special. This means there are far fewer parameters to learn in the early parts of the network, and so pre-training seems less necessary.
3. RelU activations. The survey of activation functions uses only smoothly differentiable ones whose gradients get tiny as you are far from zero. RelU has fewer problems with the gradient getting tiny or huge at idiosyncratic points, and also has the virtue of sparsifying the gradients as you backprop (since anything that landed in the negative tail has zero gradient).
So yeah, we really do do entirely unpretrained learning of low-level features, straight from RGB values between 0 and 256, and it works! Isn't that cool??
That was a big deal, and I think pretty much eliminated greedy layerwise pretraining in the "we have plenty of data, but can't generalize well" case. Good initialization rules help too, but are mostly heuristic and problem dependent to my knowledge.
The last few slides have a kind of "survey list" to get up to speed with modern deep learning approaches for images. I also put the slides on github at http://github.com/kastnerkyle/EuroScipy2014 , which hopefully preserves the hyperlinks where speakerdeck does not.
WRT the Glorot and Bengio paper, itβs true that there was a lot of excitement surrounding unsupervised pre-training of DNNs, but this mostly preceded the current wave of successes. The big differences between the architectures that are working on image processing today and that paper are:
1. Moar data. The datasets this paper was looking at were on the order 10^5 or so images. 10^6 is a different ballgame.
2. Convolution. Sharing weights really is special. This means there are far fewer parameters to learn in the early parts of the network, and so pre-training seems less necessary.
3. RelU activations. The survey of activation functions uses only smoothly differentiable ones whose gradients get tiny as you are far from zero. RelU has fewer problems with the gradient getting tiny or huge at idiosyncratic points, and also has the virtue of sparsifying the gradients as you backprop (since anything that landed in the negative tail has zero gradient).
So yeah, we really do do entirely unpretrained learning of low-level features, straight from RGB values between 0 and 256, and it works! Isn't that cool??