Deep Learning Guest Lecture

A motivational talk about deep artificial neural networks, given to the students in FFR135 (Artificial neural networks). I gave motivations for using deep architechtures, and to learn hierarchical representations for data.

Reading (web):

• My blog post about neural machine translation
• Deep Learning Book, Goodfellow, Bengio, Courville
• playground.tensorflow.org
• cs231n.stanford.edu - Stanford's course on convnets, with nice browser-based demo!

Reading (publications):

• A Fast Learning Algorithm for Deep Belief Nets; Hinton, Osindero, Tehi; Neural Computation; 2006 PDF, cs.toronto.edu
• Exploring Strategies for Training Deep Neural Networks; Larochelle, Bengio, Louradour, Lamblin; JMLR 2009 PDF, jmlr.org
• Imagenet classification with deep convolutional neural networks (“AlexNet”); Krizhevsky, Sutskever, Hinton, NIPS 2012; PDF, cs.toronto.edu
• Very deep convolutional networks for large-scale image recognition (“VGGNet”); Simonyan, Zisserman; 2014; arXiv 1409.1556; PDF, arXiv
• Going Deeper with Convolutions (“GoogLeNet”); Szegedy, Liu, Jia, Sermanet, Reed, Anguelov, Erhan, Vanhoucke, Rabinovich; 2014; arXiv 1409.4842; PDF, arXiv
• Improving neural networks by preventing co-adaptation of feature detectors; Hinton, Srivastava, Krizhevsky, Sutskever, Salakhutdinov; 2012; arXiv:1207.0580 PDF, arXiv
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift; Ioffe, Szegedy; arXiv:1502.03167 PDF, arXiv
• Deep Residual Learning for Image Recognition (“ResNet”); He, Zhang, Ren, Sun; arXiv:1512.03385 PDF, arXiv
• The loss surfaces of multilayer networks; Choromanska, Henaff, Mathieu, Arous, LeCun; AISTATS 2015 PDF, arXiv
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization; Dauphin, Pascanu, Gulcehre, Cho, Ganguli, Bengio; NIPS 2014 PDF, papers.nips.cc
• Sequence to Sequence Learning with Neural Networks, Ilya Sutskever, Oriol Vinyals, Quoc V. Le. NIPS 2014 PDF, arXiv
• Neural Machine Translation of Rare Words with Subword Units, Rico Sennrich and Barry Haddow and Alexandra Birch, ACL 2016: PDF, aclweb.org
  
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, Yonghui Wu et.al. (Google): PDF, arXiv
  

Slides (PDF)

FFR135, Artificial Neural Networks, 2016-10-06
Olof Mogren