Profile cover photo
Profile photo
Samuel Bosch
Geographer/software developer/phd student/cyclist/father (not necessarily in that order).
Geographer/software developer/phd student/cyclist/father (not necessarily in that order).

Samuel's posts

Post has attachment

Post has attachment
Upside down

Post has attachment

Post has attachment
"The P value was never meant to be used the way it's used today"

Post has attachment
F# (with and without units of measure), python and julia implementation of the azimuthal equidistant projection 

Post has attachment

Post has shared content

Post has shared content
Our Deep Learning Neural Networks (deep NN) also won the ISBI'12 Brain Segmentation Contest [2], through the efforts of Dan Claudiu Cireșan and Alessandro Giusti - see recent NIPS paper [1].

This is relevant for the recently approved huge brain projects in Europe and the US. Given electron microscopy images of stacks of thin slices of animal brains, the goal is to build a detailed 3D model of the brain's neurons and dendrites. But human experts need many hours and days and weeks to annotate the images: Which parts depict neuronal membranes? Which parts are irrelevant background? This needs to be automated. Our deep NN learned to solve this task through experience with many training images. They won the contest on all three evaluation metrics by a large margin, with superhuman performance in terms of pixel error. (Ranks 2-6: for researchers at ETHZ, MIT, CMU, Harvard.)

It is ironic that artificial NN (ANN) can help to better understand biological NN (BNN). And in more than one way - during my 2013 lecture tour [12] I kept trying to convince neuroscientists that feature detectors invented by our deep visual ANN are highly predictive of what they will find in BNN once they have figured out how to measure synapse strengths. While the visual cortex of BNN may use quite different learning algorithms, its objective function to be minimised must be similar to our ANN's.

As always in such competitions, we applied GPU-based pure supervised gradient descent (40-year-old backprop [3]) to our deep and wide GPU-based multi-column max-pooling convolutional networks (multi-column MPCNN) [4,5] with alternating convolutional layers (e.g., [6]) and max-pooling layers of winner-take-all units (e.g., [7,8,9,10]). Plus a few additional tricks [1,4,5,11]. Our architecture is biologically rather plausible, inspired by early neuroscience-related work on so-called simple cells and complex cells [7,8].

Already in 2011, our GPU-based multi-column MPCNN became the first artificial device to achieve human-competitive performance on major benchmarks, e.g.,  [5]. Most if not all leading IT companies and research labs are now using (or trying to use) this technique, too.

When we started Deep Learning research over two decades ago [11], slow computers forced us to focus on toy applications. Today, our deep NN can already learn to rival human pattern recognisers in important domains. This is just the beginning...


[1] D. Cireșan, A. Giusti, L. Gambardella, J. Schmidhuber. Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images. NIPS 26, Lake Tahoe,

[2] Segmentation of neuronal structures in EM stacks challenge - ISBI 2012

[3] Paul J. Werbos. Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard University, 1974

[4] D. C. Cireșan, U. Meier, J. Masci, L. M. Gambardella, J. Schmidhuber. Flexible, High Performance Convolutional Neural Networks for Image Classification. IJCAI-2011, Barcelona, 2011. Preprint

[5] D. C. Cireșan, U. Meier, J. Schmidhuber. Multi-column Deep Neural Networks for Image Classification.  CVPR 2012, p 3642-3649, 2012. , preprint

[6] Y. LeCun, Y. Bengio. Convolutional networks for images, speech, and time-series. In M. A. Arbib, editor, The Handbook of Brain Theory and Neural Networks. MIT Press, 1995.

[7] Hubel, D. H., T. N. Wiesel. Receptive Fields, Binocular Interaction And Functional Architecture In The Cat's Visual Cortex. Journal of Physiology, 1962.

[8] K. Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4): 193-202, 1980.

[9] M. Riesenhuber, T. Poggio. Hierarchical models of object recognition in cortex. Nature Neuroscience 11, p 1019-1025, 1999.

[10] S. Behnke. Hierarchical Neural Networks for Image Interpretation. Dissertation, FU Berlin, 2002. LNCS 2766, Springer 2003.

[11]  Deep Learning since 1991  


Credit for parts of the image: Daniel Berger, 2012
Wait while more posts are being loaded