The code example is particularly instructive. After training on the Linux kernel source, it can generate mostly syntactically correct code based on whatever input you give it. That includes not just using keywords in the right places, but things such as matching parenthesis (and knowing what can appear within), and keeping correct code indentation. It can learn larger, context-dependent structure, not just the surface-level pattern sequences. The network implicitly represents the syntax of the input, and that's very impressive.
On the other hand, the code sequences have no meaning. They don't try to compute anything; they just are. Some of them may compile, but none of them do anything meaningful. It has extracted syntax, but fails to produce anything semantically meaningful.
You see the same thing with the recent examples of deep learning network learning to play games: they learn the syntactic structure of the in-game tasks, but the playing is not goal-directed; there is no concept of "winning", or of achieving anything by playing.
Now, I don't think those are insurmountable obstacles. Semantics and pragmatics, or motivation and autonomy is not magical pixie dust. But it does require a lot more structure than a fairly task-constrained training of a single network can achieve.
There's a reason brains are very highly structured systems, with lots of separate subsystems interacting in well-defined, highly constrained ways. Even a pre-term developing brain is anything but a blank learning slate, and smaller, simpler brains (insects, say) are if anything much more highly structured than larger ones.
Also, deep learning networks are paradoxically also too general. They can learn any well-defined temporal structure, but they are computationally very inefficient at implementing and executing any one structure in particular. I suspect that in many applications, creating networks like this will be only an intermediate step; a way to constrain and understand the problem before you implement it "for real", in a much more efficient manner.
Take an artificial leg as an example. It is only syntax: a time-varying, context-dependent input (nerve signals, sensor info) gets translated into a time-varying output (joint movement). But implement that directly in an neural network and the user would need a separate backpack with the computer to run it in real time. Instead you need to extract the functions the neural network has implicitly learned, and implement them, in a far more efficient manner, in something similar to a regular old-fashioned control loop.
Current neural networks are both far more capable, and far more limited, than many people realize. I suspect future work will need to focus more on the large-scale structure, and less on adding more nodes in the network or finding still-larger training sets. Fun times ahead.