Shared publicly  - 
 
Everybody should write everything in Go. This is how easy concurrent/"threaded" applications are... I passed a string for pedagogical purposes. :) No pthread... not stupid crap... just works! :)


package main

import (
"fmt"
"strconv"
)

func whatever(c chan string, thr int) {
for i := 0; i < 100; i++ {
fmt.Printf("%d\n",i)
}
c <- "Thread "+strconv.Itoa(thr)+" done!"
}

func main() {
c := make(chan string)
for i := 0; i < 4; i++ {
go whatever(c,i)
}
for g := 0; g < 4; g++ {
fmt.Printf("%v\n", <-c)
}
}

Holy f%#9... we live in the future. +The Go Programming Language
13
7
Simon Marlow's profile photoroger peppe's profile photoAmit Kumar's profile photoKevin Ballard's profile photo
29 comments
 
Golfing it in Haskell, with +James Cook's monad-loops:

import Control.Applicative
import Control.Monad.Loops
import Data.Either

whatever thr = mapM_ print [1..100] >> return ("Thread " ++ show thr ++ " done!")

main = mapM_ print =<< rights <$> forkMapM whatever [1..4]

+Simon Marlow How would you do this in monad-par?

(*edit: made whatever a one-liner, because G+ eats block formatting)
 
+Conrad Parker Interesting… is that actually creating forks though? I like Go because it doesn't really map directly onto an underlying concurrency primitive like a thread or process...
 
+Ryan Hayes monad-loops calls forkIO, which creates lightweight threads managed by the runtime (like Go).
 
strictly speaking the Go program used a single channel (c), so the Haskell equivalent would be:

whatever c thr = do
..mapM_ print [1..100]
..writeChan c ("Thread " ++ show thr ++ " done!")

main = do
..c <- newChan
..forM_ [1..4] $ \thr ->
....forkIO (whatever c thr)
..forM_ [1..4] $ \_ -> do
....s <- readChan c
....putStrLn s

I have heard from at least a couple of reliable sources (Googlers) that Go gets quite painful when things get complex due to synchronous message passing. Asynchronous message passing is much easier to deal with (e.g. Erlang, and Haskell's Chan/TChan).
 
+Simon Marlow Thanks for commenting on my post. :) I have a little stage fright. :) Anyway, Go does support bidirectional channels and they can be buffered or unbuffered... Or did you mean something different? :)
 
unbuffered channels are "synchronous" in the sense that both reader and writer must be ready at the same time. It's easy to deadlock if you're not careful. Buffered channels allow asynchronous writes, but only up to the buffer size, so that doesn't actually make things easier. Fully asynchronous channels, like you get in Erlang and Haskell don't have this problem, but they are unbounded so you have to be careful about filling them up (Erlang uses a clever scheduling trick to mitigate that problem, though).
 
+Simon Marlow You can write unbounded asynchronous channels in Go, it just requires a bit of code on your side. This was discussed recently on the go-nuts ML at https://groups.google.com/d/topic/golang-nuts/SgcUteZlmzQ/discussion.

One solution is to have a channel of size 1 that contains a slice of your desired type, and when you want to send you first do a non-blocking receive. If you pull a slice out, you append your value and shove it back in. Otherwise you just create a slice to wrap your value and shove it in. Very simple, just a few extra lines of code.

The other solution is to have an unbuffered (or buffer length 1 if you don't want to yield) channel, and a goroutine that reads from that channel constantly and manages a slice itself, sending elements from that slice out a second channel whenever it can. This technique is used in Kyle Lemmons's `iq` library at https://github.com/kylelemons/iq/blob/master/iq_slice.go.
 
Synchronous channels is what makes Go concurrency much less painful than the alternatives. It makes it much easier to reason about. You can get deadlocks anywhere, no code is foolproof, but this is much more rare in Go/CSP than with the alternatives.

+Simon Marlow I have no idea who you have spoken to, but the developers of doozer, which implements Paxos, which is about as complicated as things get, said they couldn't imagine writing it in anything other than Go. See http://blog.golang.org/2011/04/go-at-heroku.html

Also the developers of http://code.google.com/p/vitess/ which is the only Go project Google has made public so far seem quite happy with it.

And of all the people in the go-nuts mailinglist, I don't remember anyone complaining that deadlocks are an issue, or that synchronous channels require you to be 'extra careful'. Some people not used to the model end up writing deadlocks, yes, but seems quite rare and mostly by people used to async models.
 
Actually Uriel, Dmitry and I had a long conversation about this quite a while ago. The fact of the matter is that both synchronous and asynchronous message passing are good for a certain class of problems. In some places these classes overlap, but they're distinct for some problems.

The same can be said for mutexes and explicit locking.. there are situations where it is blatantly trivial to understand how to make them work and how to reason about them, whereas channels and messages can overly complicate the situation.

Haskell's concurrency model is often overlooked by people when looking at Go and Erlang due to not knowing enough about Haskell. It's a beautiful way to express certain types of problems, and that definitely includes some concurrent exercises such as this.
 
Is this thread only saying that Haskell supports the same functionality, or is there more? FWIW Go is more legible and I can actually use it on major commercial platform. Unless there is some presented benefit of that Haskell code why should I care?
I definitely agree with Uriel.
There also should be something said about using programming constructs that the average programmer doesn't understand.
 
I think the point of the Haskell examples is that the same style of concurrency is supported in Haskell, something that the original poster obviously wasn't aware of (given his response to the original code).

FWIW you feel that Go is more legible, but that's a matter of personal opinion. I also don't understand your claim about 'major commercial platforms', given that Haskell is an apt-get installable package on debian/ubuntu and most other major Linux platforms (possibly more than Go).

The original post was about expressing a concurrent idea using Go. It also made a claim that everyone should be using Go. It's not outside the realm of decorum that someone would respond with another language saying "Here's how we'd do the exact same thing" using the same concepts.

Your argument about things that "the average programmer doesn't understand" is precisely part of the problem. 99% of the articles and books about concurrency out there talk about semaphores, mutexes, and other explicit locks. The fact of the matter is that channel-based concurrency (let along actor based concurrency) aren't constructs that "the average programmer understands". That's why people find it more difficult to express some problems using these constructs. It takes time and experience using a language to better understand how to leverage what a language has to offer.

If you're going to advocate using channels, which I do.. there's nothing to stop you from advocating a pure and monadic way of expressing channels in a functional programming language.

You however seem to think that broadening horizons is a bad thing, and I cannot agree with that.
 
Broadening horizons is great, and it is why I did any study of Haskell. My CMU background taught me that functional programming languages are for broadening horizons rather than for writing software. I have found the same in practice.
My questions are to discern whether the Haskell advocates feel Haskell could be used much more often for commercial software. And you have to admit that GAE has some nice features that you don't get on a regular server.

Edit: I was a bit derailed here. I am interested in the functional advantages of Haskell, since those offering Haskell advice seem to be a little too excited that their version of the code contains less line.
 
I'm not so interested in a language advocacy debate, but I am interested in specific examples that are more easily or better expressed using synchronous channels. I think that would make a fascinating discussion. Can people point to some?

FYI, there's a synchronous message passing library for Haskell called CHP (Communicating Haskell Processes), built on top of STM: http://hackage.haskell.org/package/chp

and FWIW, I think that STM is the right concurrency primitive, rather than any sort of channels. (that is, STM with the retry and orElse operations, so you can express compositional blocking)
 
+Simon Marlow I don't know who you've been speaking to at Google, but I know most of the authors of large Go programs at Google and they've never said anything of the sort to me.

I've also reviewed a sizable portion of the Go code written at Google. A common trap for newcomers to the language is overuse of concurrency primitives. Frequently in first-timer reviews it'll be a case of "no, you don't need channels here. You're just making things more complicated." Maybe that's what your source is referring to.
 
+Uriel Étranger indeed, when writing any concurrent program it is possible to create deadlocks. I contend that Go's concurrency primitives make it easier to avoid deadlocks.
 
+Andrew Gerrand Easier than what?

I think it's a mistake to decide on a single way of doing things, like you and your colleagues at Google have done with Go (correct me if I'm wrong, I'm a little rusty, but channels are basically the only concurrent communication mechanism in go right?). The view that's been taken in Haskell (read: GHC) land is that there is no one best solution, so we provide very well designed alternatives for many different ways of communicating (MVars, Chans, STM, pure parallelism, the Cloud Haskell work [though this is less related]).

On a side note, are there any plans for Go to add some form of type safe polymorphic data types? It'd love to be able to have type safe Maps for example. I understand currently this is achieved using duck typing right? I remember when you gave a talk at ANU you got a similar question, and at that time you said something along the line that there were no plans to extend the language that way.
 
+Alex Mason easier the traditional threading/semaphore and event loop models we use in C++ and Java.

Go only provides one threading model: goroutines. For modelling communications between concurrent processes, channels are typically the best way of doing so. However, Go does provide traditional locking primitives that can be more convenient when (for example) protecting shared data structures. You can also write in an event-driven style if it suits the task at hand. But you can't just drop in a new concurrency model, as with Haskell.

I disagree that this is a mistake. Go has different goals to Haskell. Go is a small language that is easy to learn and use. Go code closely resembles the generated machine code, so it is straightforward to understand the performance characteristics of a program. Haskell is beautiful, but it is also difficult and unfamiliar to most programmers.

Maps are provided by the Go language itself, so you can have type-safe maps already. However, there's no way of constructing your own type safe container classes. In practice this isn't much of an inconvenience; it's usually a matter of implementing a simple interface on your data structure and doing a type assertion when you take a value out.

We have been looking at various ways of putting "generics" into Go, but so far we haven't found an approach that we're satisfied with. Such a change would have a dramatic affect on the language, and shouldn't be made lightly. It's still something we're thinking about.
 
+Alex Mason consider the value of providing a single good way of doing something. It may not be the best thing in all circumstances, but it means that anyone can look at your Go code and understand how it works. Go was designed to scale to tens of thousands of programmers and hundreds of millions of lines. At this scale, consistency trumps convenience every time.
 
+Andrew Gerrand So you mentioned that Go has locking primitives in addition to channels (buffered and unbuffered), because sometimes locking is more convenient. You didn't have to add locks, because locks can be simulated using channels, and vice versa, but sometimes it's more convenient or efficient to have the right interface readily available.

This isn't all that different from Haskell, where we have two primitive mechanisms - 1-place buffered channels (MVar), and transactional memory (STM). On top of these are built a wide variety of abstractions, but you could do the same in Go. For example, Haskell's unbounded asynchronous channels are built out of 1-place buffered channels (MVar), and I'm sure you could do exactly the same thing in Go. So I don't think the design is fundamentally different: Go has multiple synchronisation mechanisms; the choice of primitives is different from Haskell, but overlapping.

Perhaps the difference is more cultural. In Haskell people often build their own abstraction layers, because there's little incentive not to. Haskell is really good at abstraction, so these layers can be indistinguishable from another concurrency model - see for example the monad-par library, which is really just a library using the basic concurrency primitives underneath. At Google there seems to be a strong emphasis on writing code that others can quickly understand, which I imagine would discourage the use of abstractions that aren't widely accepted. That's my guess anyway. I'm not suggesting it's bad: certainly I have a hard time reading a lot of random Haskell code that I come across, because Haskell programmers often tend to go crazy with the abstractions.

BTW, for some context you might want to look at this earlier thread where I was nice about Go :) https://plus.google.com/107890464054636586545/posts/dVowddBT8AW
 
+Andrew Gerrand It's possible that +Simon Marlow was referring to +John Millikin 's second post in this thread: https://plus.google.com/107890464054636586545/posts/dVowddBT8AW

I wonder if the deadlock problems come from using channels in a style taken from other languages with asynchronous channels. I very rarely have a problem with deadlock in Go, but that's perhaps because over years of this kind of programming I've evolved some design patterns that can avoid it.

In particular, I will always use a synchronous channel unless I know exactly why I want the buffer space. (simulating concurrent programs in Spin, it's striking how a buffered channel adds to the state space of the program).

Also, I make sure that each concurrent element of the program can be reasoned about independently of the behaviour of other components. Channels are most often used for communicating between elements within a package, and it's there that synchrony is useful for reasoning about behaviour.

I'm not so interested in a language advocacy debate, but I am interested in specific examples that are more easily or better expressed using synchronous channels.

Here's a pattern using synchronous channels I've found useful in the past: http://play.golang.org/p/3RHayA0gXP
When DoSomething succeeds in sending the request, it knows that the the request is being processed and that it will receive a reply. This makes it easy for some other goroutine to call Stop asynchronously without worrying that requests will be dropped.
 
"I wonder if the deadlock problems come from using channels in a style taken from other languages with asynchronous channels."

From what I have seen, I think you are spot on, this are the same kind of people that tend to over-use buffered channels, and when something doesn't work with unbuffered channels, they just add a buffer in hopes that it will fix the problem.

And I fear the worst is yet to come, is scary to think of the damage done to the software industry's collective consciousness by Node.js
 
+roger peppe thanks very much for the example. There's one thing I'm struggling to understand. What is it that prevents the request in this sequence from being processed:

f.Stop()
f.DoSomething("foo")

it looks like there's still a small possibility that the request will be processed successfully, because in f.run() both branches of the select will be able to proceed.
 
+Simon Marlow that's true. in this example, Stop is not synchronous, but that's often not a problem. if you need to be sure that Stop means it's stopped, it's easy to make the Stop method wait until the runnning goroutine has signalled that it's done.

actually, for this particular problem (shutting things down), this package http://go.pkgdoc.org/launchpad.net/tomb can be helpful, but the "you know what state the receiving goroutine is in when you've succeeded in sending to it" property of synchronous channels is useful in many more general scenarios too, i've found.
 
Great, thanks.

So I agree that synchronous channels are useful here. However you pay a high price for this level of synchronisation: all the requests are completely serialised. It makes me wonder to what extent the use of synchronous channels leads to accidental concurrency bottlenecks in practice.

And it's funny you should refer to "the state that the receiving goroutine is in", because that's exactly how my Haskell version works: http://hpaste.org/69356. It has a synchronous Stop, so the semantics differ slightly from the Go version, it's about the same number of lines though.
 
Any asshole can construct a few benchmarks that show some language is faster. Haskell is a nightmare to program in. Period.
 
The benchmarks are ok, I think there are other more important things, depending on the needs of each developer, for example: the ecosystem and a good web framework (scala strategy, although I prefer the syntax of Haskell on FP), I hate verbosity but Revel Framework is quite interesting.
Add a comment...