Profile cover photo
Profile photo
Juergen Schmidhuber
2,973 followers -
Towards True AI Since 1987
Towards True AI Since 1987

2,973 followers
About
Juergen's posts

Post has attachment
According to the stock market rankings of the first quarter of 2017, the four most valuable public companies in the world are massively using our deep learning methods developed since the early 1990s, in particular, the Long Short-Term Memory (LSTM): Apple (#1 as of 31 March 2017 with a market capitalization of USD 753bn), Google (Alphabet, #2, 573bn), Microsoft (#3, 508bn), and Amazon (#4, 423bn). LSTM-based systems learn to translate languages, control robots, analyse images, summarise documents, recognise speech and videos and handwriting, run chat bots, predict diseases and click rates and stock markets, compose music, and much more. Here the overview page with numerous references: http://people.idsia.ch/~juergen/impact-on-most-valuable-companies.html

#computervision
#deeplearning
#machinelearning
#artificialintelligence
Photo

Post has attachment
History of computer vision contests won by deep CNNs on GPU. Modern computer vision since 2011 relies on deep convolutional neural networks (CNNs) efficiently implemented on massively parallel graphics processing units (GPUs). Here is a list of important computer vision competitions won by deep GPU-CNNs, ordered by date, with a focus on those contests that brought "Deep Learning Firsts" and/or major improvements over previous best or second best (with background and references): http://people.idsia.ch/~juergen/computer-vision-contests-won-by-gpu-cnns.html

#computervision
#deeplearning
#machinelearning
#artificialintelligence


Photo

Post has attachment
Videos of the NIPS 2016 Symposium: Recurrent Neural Networks and Other Machines that Learn Algorithms (Barcelona, Dec 2016)

Thousands of people participated in this event, but many others were blocked from entering the lecture hall for safety reasons. Videos and slides of the speakers are now available for all under http://people.idsia.ch/~rupesh/rnnsymposium2016/program.html

Blurb: Soon after the birth of modern computer science in the 1930s, two fundamental questions arose: 1. How can computers learn useful programs from experience, as opposed to being programmed by human programmers? 2. How to program parallel multiprocessor machines, as opposed to traditional serial architectures? Both questions found natural answers in the field of Recurrent Neural Networks (RNNs), which are brain-inspired general purpose computers that can learn parallel-sequential programs or algorithms encoded as weight matrices.

The first RNNaissance NIPS workshop dates back to 2003: http://people.idsia.ch/~juergen/rnnaissance.html . Since then, a lot has happened. Some of the most successful applications in machine learning (including deep learning) are now driven by RNNs such as Long Short-Term Memory, e.g., speech recognition, video recognition, natural language processing, image captioning, time series prediction, etc. Through the world's most valuable public companies, billions of people can now access this technology through their smartphones and other devices, e.g., in the form of Google Voice or on Apple's iOS. Reinforcement-learning and evolutionary RNNs are solving complex control tasks from raw video input. Many RNN-based methods learn sequential attention strategies.

At this symposium, we review the latest developments in all of these fields, and focus not only on RNNs, but also on learning machines in which RNNs interact with external memory such as neural Turing machines, memory networks, and related memory architectures such as fast weight networks and neural stack machines. In this context we also discuss asymptotically optimal program search methods and their practical relevance.

Our target audience has heard a bit about RNNs, the deepest of all neural networks, but is happy to hear again a summary of the basics and then delve into the latest advanced topics to see and understand what has recently become possible. All invited talks are followed by open discussions, with further discussions during a poster session. Finally, a panel discusses the bright future of RNNs, and their pros and cons.

Symposium website: http://people.idsia.ch/~rupesh/rnnsymposium2016/index.html

Organizers: Jürgen Schmidhuber & Sepp Hochreiter & Alex Graves & Rupesh Srivastava


#artificialintelligence
#deeplearning
#machinelearning
#computervision
Photo

Post has attachment
First Swiss AI Master Program (Fall 2017)
Univ. Lugano & The Swiss AI Lab, IDSIA

Get a Master's Degree in Artificial Intelligence, through the first AI Master program in Switzerland. Please follow instructions under http://people.idsia.ch/~juergen/aimaster.html

IDSIA is among the world's top-ranked AI institutes. Its deep learning artificial neural networks are now available to billions of users through the world’s most valuable public companies such as Google, Apple, Microsoft, Amazon, Baidu, etc.

On a per capita basis, Switzerland is leading the world in Nobel Prizes, patents, publications, citations, quality of life, competitiveness, happiness, many sports: http://people.idsia.ch/~juergen/switzerland.html

Jobs for PhD students and postdocs: http://people.idsia.ch/~juergen/jobs2017.html


Jürgen Schmidhuber
Scientific Director, Swiss AI Lab IDSIA
Professor of AI, USI & SUPSI, Switzerland
President, NNAISENSE
http://people.idsia.ch/~juergen/whatsnew.html

#artificialintelligence
#deeplearning
#machinelearning
#computervision


Photo

Post has attachment
Update of 23 May 2017: Thanks again for many excellent applications! We have not yet filled all positions announced under http://people.idsia.ch/~juergen/jobs2017.html But we are proceeding more slowly now, hoping to find people with the profile necessary to complement the already existing expertise. Soon we will re-open the online application forms (see instructions). In addition, we are about to announce numerous additional job openings soon, thanks to a big new grant.

13 February 2017: New Jobs for PostDocs and PhD students thanks to Google DeepMind, NVIDIA and SNF: Please follow instructions under http://people.idsia.ch/~juergen/jobs2017.html

11 November 2016: We have an open call for Tenure-Track Assistant Professor - deadline Dec 15: http://www.usi.ch/call-inf-assistant-professor-tenure-track-291728.pdf We encourage experts in Computer Vision / Machine Learning / Neural Networks / Deep Learning to apply!

Fall 2016 - jobs for postdocs and PhD students: Join the Deep Learning team (since 1991) that won more competitions than any other. We are seeking researchers for the project RNNAIssance based on this tech report on "learning to think:” http://arxiv.org/abs/1511.09249 . The project is about general purpose artificial intelligence for agents living in partially observable environments, controlled by reinforcement learning recurrent neural networks (RNNs), supported by unsupervised predictive RNN world models. Location: The Swiss AI Lab, IDSIA, in Switzerland, the world’s leading science nation, and most competitive country for the 7th year in a row. Competitive Swiss salary. Preferred start: As soon as possible.  More details and instructions can be found here: http://people.idsia.ch/~juergen/rnnai2016.html

#artificialintelligence
#deeplearning
#machinelearning
#computervision
Photo

Post has attachment

Post has attachment
Microsoft wins ImageNet 2015 through feedforward LSTM without gates

Microsoft Research dominated the ImageNet 2015 contest with a deep neural network of 150 layers [1]. Congrats to Kaiming He & Xiangyu Zhang & Shaoqing Ren & Jian Sun on the great results [2]!

Their CNN layers compute G(F(x)+x), which is essentially a feedforward Long Short-Term Memory (LSTM) [3] without gates!

Their net is similar to the very deep Highway Networks [4] (with hundreds of layers), which are feedforward LSTMs with forget gates (= gated recurrent units) [5].

The authors mention the vanishing gradient problem, but do not mention my very first student Sepp Hochreiter (now professor) who identified and analyzed this fundamental deep learning problem in 1991, years before anybody else did [6].

Apart from the above, I liked the paper [1] a lot. LSTM concepts keep invading CNN territory [e.g., 7a-e], also through GPU-friendly multi-dimensional LSTMs [8].

References

[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Deep Residual Learning for Image Recognition. http://arxiv.org/abs/1512.03385

[2] ImageNet Large Scale Visual Recognition Challenge 2015 (ILSVRC2015) - Results: http://www.image-net.org/challenges/LSVRC/2015/results

[3] S. Hochreiter, J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735-1780, 1997. Led to a lot of follow-up work http://people.idsia.ch/~juergen/rnn.html, and is now heavily used by leading IT companies around the world.

[4] R. K. Srivastava, K. Greff, J. Schmidhuber. Training Very Deep Networks. NIPS 2015; http://arxiv.org/abs/1505.00387

[5] F. A. Gers, J. Schmidhuber, F. Cummins. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10):2451-2471, 2000. ftp://ftp.idsia.ch/pub/juergen/FgGates-NC.pdf

[6] Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TU Munich. Advisor: J. Schmidhuber. Overview: http://people.idsia.ch/~juergen/fundamentaldeeplearningproblem.html

[7a] 2011: first superhuman CNNs http://people.idsia.ch/~juergen/superhumanpatternrecognition.html
[7b] 2011: First human-competitive CNNs for handwriting http://people.idsia.ch/~juergen/handwriting.html
[7b] 2012: first CNN to win segmentation contest http://people.idsia.ch/~juergen/deeplearningwinsbraincontest.html
[7c] 2012: first CNN to win object discovery contest http://people.idsia.ch/~juergen/deeplearningwinsMICCAIgrandchallenge.html
[7d] Scholarpedia: http://www.scholarpedia.org/article/Deep_Learning

[8] M. Stollenga, W. Byeon, M. Liwicki, J. Schmidhuber. Parallel Multi-Dimensional LSTM, with Application to Fast Biomedical Volumetric Image Segmentation. NIPS 2015; http://arxiv.org/abs/1506.07452

Link: http://people.idsia.ch/~juergen/microsoft-wins-imagenet-through-feedforward-LSTM-without-gates.html

#computervision
#deeplearning
#machinelearning
#artificialintelligence
Photo

Post has attachment
How to Learn an Algorithm (video). I review 3 decades of our research on both gradient-based and more general problem solvers that search the space of algorithms running on general purpose computers with internal memory. Architectures include traditional computers, Turing machines, recurrent neural networks, fast weight networks, stack machines, and others. Some of our algorithm searchers are based on algorithmic information theory and are optimal in asymptotic or other senses. Most can learn to direct internal and external spotlights of attention. Some of them are self-referential and can even learn the learning algorithm itself (recursive self-improvement). Without a teacher, some of them can reinforcement-learn to solve very deep algorithmic problems (involving billions of steps) infeasible for more recent memory-based deep learners. And algorithms learned by our Long Short-Term Memory recurrent networks defined the state-of-the-art in handwriting recognition, speech recognition, natural language processing, machine translation, image caption generation, etc. Google and other companies made them available to over a billion users.

The video was taped on Oct 7 2015 during MICCAI 2015 at the Deep Learning Meetup Munich:  http://www.meetup.com/en/deeplearning/events/225423302/  Link to video: https://www.youtube.com/watch?v=mF5-tr7qAF4

Similar talk at the Deep Learning London Meetup of Nov 4 2015: http://www.meetup.com/Deep-Learning-London/events/225841989/ (video not quite ready yet)

Most of the slides for these talks are here: http://people.idsia.ch/~juergen/deep2015white.pdf

These also includes slides for the AGI keynote in Berlin http://agi-conf.org/2015/keynotes/, the IEEE distinguished lecture in Seattle (Microsoft Research, Amazon), the INNS BigData plenary talk in San Francisco, the keynote for the Swiss eHealth summit, two MICCAI 2015 workshops, and a recent talk for CERN (some of the above were videotaped as well).

Parts of these talks (and some of the slides) are also relevant for upcoming talks in the NYC area (Dec 4-6 and 13-16) and at NIPS workshops in Montreal:

1. Reasoning, Attention, Memory (RAM) Workshop, NIPS 2015 https://research.facebook.com/pages/764602597000662/reasoning-attention-memory-ram-nips-workshop-2015/

2. Deep Reinforcement Learning Workshop, NIPS 2015 http://rll.berkeley.edu/deeprlworkshop/

3. Applying (machine) Learning to Experimental Physics (ALEPH) Workshop, NIPS 2015 http://yandexdataschool.github.io/aleph2015/pages/keynote-speakers.html

More videos: http://people.idsia.ch/~juergen/videos.html

Also available now: Scholarpedia article on Deep Learning: http://www.scholarpedia.org/article/Deep_Learning

Finally, a recent arXiv preprint: On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models. http://arxiv.org/abs/1511.09249

#machinelearning
#artificialintelligence
#computervision
#deeplearning


Post has attachment
Announcing Brainstorm Open Source Software for Neural Networks

We are open-sourcing a new neural networks library called Brainstorm, developed over the past year at the Swiss AI Lab IDSIA by PhD students Klaus Greff and Rupesh Srivastava: https://github.com/IDSIA/brainstorm

Brainstorm is designed to make neural networks fast, flexible and fun. Lessons learned from earlier open source projects led to new design elements compatible with multiple platforms and computing backends.

Brainstorm already has a robust base feature set, including support for recurrent neural networks (RNNs) such as LSTM, Clockwork, 2D Convolution/Pooling and Highway layers on CPU/GPU. All data is considered sequential, and RNNs are first-class citizens.

We hope the community will help us to further improve Brainstorm.

#machinelearning
#artificialintelligence
#computervision
#deeplearning

http://people.idsia.ch/~juergen/brainstorm.html
Photo

Post has attachment
The good news came on 8/5/15: I am recipient of the 2016 IEEE CIS Neural Networks Pioneer Award, “for pioneering contributions to deep learning and neural networks.” The list of all awardees since 1991 is here: http://cis.ieee.org/award-recipients.html

#machinelearning
#artificialintelligence
#computervision
#deeplearning
Photo
Wait while more posts are being loaded