Post has attachment
Six technologies for memory devices that replicate the function of biological neurons and synapses.

ReRAM is a type electronic switch based on a 'drift' memristor model "that exhibits non-volatility, i.e., will retain its resistance state even after the voltage is turned off."

Diffusive memristors: memristors based on a type of memristor that uses diffusive dynamics of active metals that "are able to emulate synaptic plasticity using their unique conductance behavior."

Phase change memory: "another high performance, non-volatile memory, in this case, based on chalcogenide glass compounds that change their resistance as they move from one phase to the other. "

Spintronics MRAM: "sometimes referred to as spin‐torque transfer MRAM (STT-MRAM) stores data magnetically but uses electrons to read and write it."

Ferroelectric field‐effect transistors: "use ferroelectric materials that can switch rapidly between two polarized states."

Synaptic transistors: "Unlike the other technologies mentioned here, specifically designed to simulate the behavior of neurons."
Add a comment...

Post has attachment
Michael Nielsen's book Neural Networks and Deep Learning is available online for free.
Add a comment...

Post has attachment
"Neurons carrying a gene called Arc -- involved in learning and memory -- exchange information like viruses do."

"Cells with the Arc gene crank out proteins that clump into capsids. Inside is messenger RNA (mRNA), which relays DNA's genetic blueprints to cells. The capsids travel to neurons, where Arc's mRNA is then transferred; the cells then also start releasing Arc capsids. Viruses use the same system to spread their genes throughout an organism."
Add a comment...

Post has attachment
DeepMind is going to demo an AI playing StarCraft II on Thursday, January 24.
Add a comment...

Post has attachment
Kai-Fu Lee profile on 60 Minutes. Kai-Fu Lee has funded 140 AI startups in China. 10 are $1 billion companies. One is the face recognition company Face++ (not associated with the US company Facebook). He thinks China has caught up with Silicon Valley but Silicon Valley isn't aware of it yet.
Commenting is disabled for this post.

Post has attachment
"Neural networks to production, from an engineer." "Guide contents: Acquiring & formatting data for deep learning applications, word embedding and data splitting, bag-of-words to classify sentence types (dictionary), classify sentences via a multilayer perceptron (MLP), classify sentences via a recurrent neural network (LSTM), convolutional neural networks to classify sentences (CNN), FastText for sentence classification (FastText), and hyperparameter tuning for sentence classification."
Add a comment...

Post has attachment
"TPUs are fast and cost-efficient, as tested in our smaller example networks. For these smaller networks and datasets we can see that the overhead of parallelization is high, but this is no different from using multiple GPUs. For the Adam optimizer, we see some strange results in the validation loss of networks trained on TPUs that are not visible in networks trained on a GPU or CPU."
Add a comment...

Post has attachment
TVM now supports runtime bindings for Golang. It already supported runtime bindings for Javascript, Java, Python, and C++. But I didn't know that because this is the first time I ever heard of it. I spent the last 20 minutes trying to figure out what TVM stands for. I can't tell you what it stands for, but I can tell you the names of the people who invented it (Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy). Either "TVM" doesn't stand for anything and is just a made-up sequence of 3 letters, or it's a big secret, because nobody seems to be telling. They even had a big conference last year (in Seattle) (how is it people have conferences for stuff before I've even ever heard of it?), and I looked at some of the conference slides, and nobody says what TVM stands for. My best guess is that they took LLVM, the name of the open-source compiler toolchain project (which originally stood for "Low Level Virtual Machine", but now has little to do with virtual machines -- LLVM is a compiler toolchain, not a virtual machine), and decided to replace the "LL" with "T" for "tensor". ("T for tensor" should be a song.) (And likewise, TVM is a compiler, not a virtual machine...)

Anyway, what TVM is is a compiler to map deep learning models on to various hardware systems (GPUs, TPUs, mobile phones, etc), and then do a boatload of optimizations specific to that hardware system. It is said to be competitive with state-of-the-art, hand-tuned libraries specific to those hardware systems. It's also ideal for hardware designers, such as FPGA designers, who want to be able to deploy deep learning models on their new hardware, and see if the new hardware is competitive with existing hardware.

"Current deep learning frameworks, such as TensorFlow, MXNet, Caffe, and PyTorch, rely on a computational graph intermediate representation to implement optimizations, e.g., auto differentiation and dynamic memory management. Graph-level optimizations, however, are often too high-level to handle hardware back-end- specific operator-level transformations. Most of these frameworks focus on a narrow class of server-class GPU devices and delegate target-specific optimizations to highly engineered and vendor-specific operator libraries. These operator-level libraries require significant manual tuning and hence are too specialized and opaque to be easily ported across hardware devices. Providing support in various deep learning frameworks for diverse hardware back-ends presently requires significant engineering effort. Even for supported back-ends, frameworks must make the difficult choice between: (1) avoiding graph optimizations that yield new operators not in the predefined operator library, and (2) using unoptimized implementations of these new operators."

"To enable both graph- and operator-level optimizations for diverse hardware back-ends, we take a fundamentally different, end-to-end approach. We built TVM, a compiler that takes a high-level specification of a deep learning program from existing frameworks and generates low-level optimized code for a diverse set of hardware back-ends. To be attractive to users, TVM needs to offer performance competitive with the multitude of manually optimized operator libraries across diverse hardware back-ends."
Siva
Siva
tvm.ai
Add a comment...

Post has attachment
AI Fiddle is a system to design, train, and share AI models from your browser.
Add a comment...

Post has attachment
NLP Overview is an overview of recent trends in deep learning based natural language processing (NLP). "It covers the theoretical descriptions and implementation details behind deep learning models, such as recurrent neural networks (RNNs), convolutional neural networks (CNNs), and reinforcement learning, used to solve various NLP tasks and applications. The overview also contains a summary of state of the art results for NLP tasks such as machine translation, question answering, and dialogue systems."

It is an open source project, with the source hosted on Github and maintained by Elvis Saravia (National Tsing Hua University) and Soujanya Poria (Nanyang Technological University).
Add a comment...
Wait while more posts are being loaded