I think a zero initialization might not be the best for softmax.
It can be interpret in some way as the prior log probability for the classes. For N classes, you could thus initialize them all with log(1/N). Or maybe with the prior log probabilities themselves.
Recurrent Models of Visual Attention, by , Nicolas Heess, , (Google DeepMind)
The title reminded me of Recurrent Processing during Object Recognition (http://psych.colorado.edu/~oreilly/papers/OReillyWyatteHerdEtAl13.pdf), by et al.
I thought this might be an appropriate forum to announce, that we recently made a 1000 hour corpus of read English speech available for download at http://www.openslr.org/12/. Example scripts are included in Kaldi.
I'd like to draw your attention to two papers that have been posted in the last few days from some of my colleagues at Google that I think are pretty interesting and exciting:
Learning to Execute: http://arxiv.org/abs/1410.4615
Neural Turing Machines: http://arxiv.org/abs/1410.5401
The first paper, "Learning to Execute", by +Wojciech Zaremba and attacks the problem of trying to train a neural network to take in a small Python program, one character at a time, and to predict its output. For example, as input, it might take:
print((c+8704) if 2641<8500 else 5308)"
During training, the model is given that the desired output for this program is "12185". During inference, though, the model is able to generalize to completely new programs and does a pretty good of learning a simple Python interpreter from examples.
The second paper, "Neural Turing Machines", by , Greg Wayne, and from Google's DeepMind group in London, couples an external memory ("the tape") with a neural network in a way that the whole system, including the memory access, is differentiable from end-to-end. This allows the system to be trained via gradient descent, and the system is able to learn a number of interesting algorithms, including copying, priority sorting, and associative recall.
Both of these are interesting steps along the way of having systems learn more complex behavior, such as learning entire algorithms, rather than being used for just learning functions.
(Edit: changed link to Learning to Execute paper to point to the top-level Arxiv HTML page, rather than to the PDF).
I explained my thoughts in detail here: http://math.stackexchange.com/questions/794322/mean-and-variance-normalization-of-vectors
I'm posting this here because you might be the better target group to answer this. The question has unfortunately not gained much attraction on Math.SE. Maybe I also just worded it badly.
Edit: I just found [this](http://metaoptimize.com/qa/questions/8307/regularizing-a-covariance-matrix). This might be related... I have to read further into the material.
- Command line dev tools like GCC were not available anymore. In Xcode, under Preferences, under Downloads, clicking on "Install" on Command Line Tools fixed this.
- Many Python tools stopped to work because most/all `easy_install`ed packages got lost. From what I tested so far, that was pycrypto and IPython. After installing the command line tools (GCC and co), just re`easy_install`ing the stuff worked.
- My self-developed screensaver stopped working. It always crashed with a segfault. It seems that some NSObject ARC related changes caused this crash. See here for my fixes: http://goo.gl/omV0b
Semantic overlap of "Christmas" with:
Jesus Christ: 10%
Santa Claus: 24%
.... Happy Holidays!
Recognize a cat having only seen dogs, but having read about both dogs and cats.
word2vec, DISSECT, Paragraph Vector. Phrase Representations using RNN Encoder–Decoder, Life-long Off-policy Learning, Neural Tensor Networks.
It's about a simple task to recognize whether there are 3 same shapes in a picture or not. And that all state-of-the-art ML algorithms fail to solve this without guidance.
I wonder a bit about that. I would have expected that a combination of some unsupervised deep convolutional NN with a supervised NN would be able to learn that. Maybe even removing the convolution and provide respectively more training data.
I searched a bit for any follow-up work about some methods which can solve this task. Maybe also with the use of evolutionary algorithms.
Does anyone have comments on that?
So I think this task is difficult not necessarily because learning about equality is difficult, but because to solve the task, you need to learn invariant object recognition first, and apparently if the supervision signal is only the equality signal then learning the object recognition implicitly is difficult (and it would be nice to analyse further why).
So if you ask how people learn this task, well, they probably learn invariant object recognition first, i.e. to recognize object categories across many instances and views, before they learn/reason about relationships between multiple categories...
As for learning invariant object recognition, of course there are many ideas how to do that including in relatively unsupervised ways. Coming back to that point once more, like I said, an unsupervised algorithm is not going to discover the categories unless there is structure in the data that reflects the categories (e.g. transformation sequences) and/or there is some relevant information built into the algorithm (e.g. in this case, knowledge about transformations).
Given a set of images, I can arbitrarily assign category boundaries. E.g. in this case, what if I now decide the categories aren't about the shape types, but about how many pixels there are, or whether a shape is symmetric or not, etc. An unsupervised algorithm isn't going to read my mind--all it has is the data and the assumptions built into the algorithm.
- RWTH AachenPhD Machine Learning, present
- RWTH AachenM.S. Mathematics, 2013
- RWTH AachenM.S. Computer Science, 2013
https://github.com/albertz, http://www.az2000.de/ and https://sourceforge.net/users/albertzeyer/.
I am also one of the developers of OpenLieroX:
- RWTH AachenPhD Student, presentMachine Learning, Speech Recognition, Deep Learning, Neural Networks, ...
- inmationSenior Software Engineer, present
Video-Clip "Die Story im Ersten: Der Fall Mollath" | ARD Mediathek | Das...
Reportage / Dokumentation - Seit mehr als sieben Jahren sitzt Gustl Mollath aus Nürnberg in der geschlossenen Psychiatrie. Der Film arbeitet
Project Zomboid: Giving Indie gaming a black eye? « Icrontic Gaming
Project Zomboid: Giving Indie gaming a black eye? David Kenkel (NiGHTS) A dramatic tale of an Indie gaming community, theft, intrigue, and a
Jailbreak iPhone 4S iOS 5 Untethered Status, Who Is Going To Do It?
Jailbreak iPhone 4S on iOS 5. iPhone 4S Jailbreak Untethered and Tethered. iOS 5 Untethered Jailbreak on iPhone 4S. Everything about iPhone