Profile

Cover photo
Bradley Neuberg
Worked at Google Inc.
Attended Columbia University
Lives in San Francisco, California
3,062 followers|54,410 views
AboutPostsPhotosYouTube

Stream

Bradley Neuberg

Discussion  - 
 
I'm almost finished with the Hinton 2012 Coursera course. I know that two things have changed since then:

* Using statistical generative models such as RBMs, DBNs, and Sigmoid Belief Nets for pretraining is no longer done, as simply properly initializing your weights properly and using backprop is good enough, even for deep networks.
* The logistic activation function is not used as much, with ReLU's preferred as they are simpler and faster.

Does anyone know the scientific papers that established these two changes? I'd love to study the original papers.
Neural Networks for Machine Learning from University of Toronto. Take free online classes from 115+ top universities and educational organizations. We partner with schools like Stanford, Yale, Princeton, and others to offer courses in dozens of topics, from computer science to teaching and beyond. Whether you are pursuing a passion or looking to advance your career, Coursera provides open, free education for everyone.
54
20
Mike G's profile photoChristian Yonathan's profile photoDaniel Santiago's profile photoamina mollaysa's profile photo
5 comments
 
I suspect some of these properties are peculiar to Relu activations where most of the layers are convolutional rather than fully connected. 

I have encountered a great deal of difficulty with classification performance on multi-layer fully connected networks wherein a single hidden layer with a sigmoid activation coupled with a KL Divergence sparseness penalty beats everything deeper.

I have found that Relus in particular are downright temperamental in this case because of their unbounded outputs coupled with the much larger fan-in of fully connected layers.
Add a comment...

Bradley Neuberg

Discussion  - 
 
I'm training a siamese network (http://yann.lecun.com/exdb/publis/pdf/chopra-05.pdf) for binary classification of facial images (i.e. are these two faces the same or different)?

By default a siamese network simply outputs a 'distance' value where the same faces have lower values and different faces have higher values.

I'd like to turn this into a binary classifier so that 1 indicates the two faces are the same and 0 indicates they are not. It seems like it might be possible to take a siamese network and have its 'distance' value feed into another fully connected layer, with the output of this layer indicating 1 or 0 as the binary classifier. I could then train not only the distance layer but also the target binary classifier output layer in order to discover a threshold value that indicates that two distance values are the same face or not, using backpropagation. Having two backpropagation targets though seems like it might get confusing for the network. Perhaps this actually has to be two different networks?

Am I off here? Is there a better way to do this?

I'm using Caffe to drive all of this.
1
Kai Arulkumaran's profile photoBradley Neuberg's profile photo
5 comments
 
Ah right sigmoid saturates at 0 and 1. Thanks!
Add a comment...

Bradley Neuberg

Discussion  - 
 
Can Caffe be used for Recurrent Neural Networks (RNN)? If not, what are most people in the field using to model their RNNs?
1
1
Will Williams's profile photoSoumith Chintala's profile photoMin Ooch's profile photoMariano Phielipp's profile photo
5 comments
 
Check out the pull requests for caffe, there s one pending for integration that brings RNN and LSTM to cafe.
Add a comment...

Bradley Neuberg

Discussion  - 
 
Does anyone know if it's easy to generate charts from Caffe, such as seeing how the error rate changes over training epochs for the test and cross validation data sets? Seems like it's necessary to generate these charts as one tunes the hyperparameters of a neural network to be understand how they effect things.
3
Manuel Lopez Antequera's profile photoSancho McCann's profile photoBradley Neuberg's profile photo
8 comments
 
For future reference for others, here's an example summary of how you can capture Caffe's output, run it through the parse_log.sh script, and then make a simple plot of the results:

./examples/mnist/train_lenet.sh 2>&1 | tee "mnist.log"
./tools/extra/parse_log.sh ./mnist.log
gnuplot ./tools/extra/plot_log.gnuplot.example

You'll want to customize the plot_log.gnuplot.example file for your own uses. The 'tee' command will also print out the results as they run as well as save them to a file, so you can follow training as it happens.
Add a comment...

Bradley Neuberg

Discussion  - 
 
I'm working through Geoffrey Hinton's 2012 Coursera course on Neural Networks. In the course he mentions the exploding and vanishing gradient problem when working with Recurrent Neural Networks. He gives several solutions to this, including careful default initialization + momentum, using Hessian Free optimization, LSTMs, and Echo State Networks. What is the current state of the art when dealing with RNNs? Are these still the grab bag of techniques one would reach for or are there other, simpler options now? Are Echo State Networks still used, as their generality seems limited?
15
5
Mingwan Wang's profile photoAlbert Zeyer's profile photoNicolas Chapados's profile photoChristian Hudon's profile photo
9 comments
Jim Fan
 
+Junyoung Chung Oh sorry about that, I didn't really read through the paper in details, just heard about it from a friend. 
Add a comment...

Bradley Neuberg

Shared publicly  - 
 
Peter Diamandis always has unique, positive ideas for the future, and the elbow grease to back them up with action. Looks like this will be a great event.
 
Join Peter H. Diamandis to discuss Abundance, exponential technologies and anything else.
Hangout With Peter Diamandis
Thu, June 20, 2013, 12:00 PM EDT
Hangouts

1
Add a comment...
Have him in circles
3,062 people
Andrew Longstaff's profile photo
Brandon Donnelson's profile photo
rocelyn calog's profile photo
Александр Тархов's profile photo
Ray Ni's profile photo
Anders Fisher's profile photo
Robert Hamilton's profile photo
Emanuele Canavesi's profile photo
Dimitar Belchugov's profile photo

Communities

Bradley Neuberg

Discussion  - 
 
I've been learning deep learning through Coursera and other online courses. Are there any masters programs yet that go into deep learning, or is it restricted to mostly PhDs programs still? Do Berkeley/Cal and Stanford have masters programs in these subjects yet? How about online masters programs?
3
Bradley Neuberg's profile photoEthan Caballero's profile photoYagnesh Revar's profile photoSam Bowman's profile photo
5 comments
 
A fair number of students in the general Stanford CS masters program seem to be focusing on deep learning.
Add a comment...

Bradley Neuberg

Discussion  - 
 
I'm working on a neural network that can take two segmented facial images as input and return a binary "same/not same" answer on whether the two given images are the same person or not. Is anyone aware of any prior work in the literature that can help provide direction on a best approach to this?
4
1
Chris Russell's profile photoMilan Lajtoš's profile photoBradley Neuberg's profile photoPatrick Ehlen's profile photo
3 comments
 
Chris that paper looks incredible; thanks for pointing that out to me. Milan, Siamese networks look like a good fit; there's even a Caffe model checked into the Caffe repo recently I can use.
Add a comment...

Bradley Neuberg

Shared publicly  - 
 
 
A nice and largely accurate article in The Chronicle of Higher Education about the history of neural nets and deep learning, with quotes from +Geoffrey Hinton, +Terrence Sejnowski, +Yoshua Bengio, and yours truly.

http://chronicle.com/article/The-Believers/190147/
The hidden story behind the code that runs our lives.
View original post
1
Add a comment...

Bradley Neuberg

Shared publicly  - 
 
From Thailand, to Coworking, to HTML5, Now to Next Generation eBooks: Refocusing My Blog
1
Add a comment...

Bradley Neuberg

Shared publicly  - 
 
We just announced what I've been working on the last year! It's called Inkling Habitat (http://www.inkling.com/habitat/), and its the world's first browser-based scalable publishing environment for interactive content. It's nice to be able to talk about what I'm working on now :)
12
4
Michael[tm] Smith's profile photoDavid Dailey's profile photohtml5.org's profile photoLuther Huffman's profile photo
9 comments
 
HeyYa - Videos for "Social Learning Network" and "Unfair Advantage" (Gotta change that text; not real meaningful / slightly misleading) have me doing back-flips here! <inkling.com/features>
(DM to you in Twitter)

I just did Harvard's "Justice" course with Michael Sandel. Wonderful material. But OMG the edX system they use is like ... like something created in 1998. Painfully Web0.6b. <edx.org>

p.s. RFE: you should use avatar from Twitter or pic from Blog here! When I see that default pic I take it to mean "Account Inactive". cheers
Add a comment...
People
Have him in circles
3,062 people
Andrew Longstaff's profile photo
Brandon Donnelson's profile photo
rocelyn calog's profile photo
Александр Тархов's profile photo
Ray Ni's profile photo
Anders Fisher's profile photo
Robert Hamilton's profile photo
Emanuele Canavesi's profile photo
Dimitar Belchugov's profile photo
Communities
Places
Map of the places this user has livedMap of the places this user has livedMap of the places this user has lived
Currently
San Francisco, California
Previously
McAllen, Texas - New York, NY - Kamala, Phuket, Thailand - Esalen, California
Work
Occupation
Software Engineer
Employment
  • Google Inc.
    Software Engineer
  • Rojo
  • Bootstrap Foundation
  • Random Walk
  • Internet Archive
  • Sitepen
Education
  • Columbia University
  • The Science Academy
Basic Information
Gender
Male
Other names
Brad