Pipes - observation, comment, and question

I've had a chance to digest the Pipes package, and I have three basic thoughts I'd like to share.

The observation: a driving motivator for iteratees has always been making it easier to reason about resource usage. In support of this, a general property of enumeration-IO packages has been that scarce resources (Handles/sockets/file descriptors) will never be held for longer than necessary, and indeed it's basically been impossible to circumvent this (at least in "iteratee", "enumerator", and Oleg's code). However, by reifying enumerators (producers) to data, the pipes package no longer has this property. A Pipe may hold an open Handle until it's garbage collected, which may be much later than you would expect (or desire). I don't think it will be easy to regain this property without significantly altering some core design choices.

The comment: I believe that the decision to require that all data producers be explicitly drained is a mistake. This is mostly because at present, I don't see an easy way to short-circuit the necessary IO to consume e.g. a file if it's determined at an early stage that further processing is unnecessary. Also, out of all the programming mistakes I've made, and bugs I've introduced (both categories are unfortunately quite large), I don't believe I've ever made an error that would have been prevented by this requirement. However, this would be relatively simple to change and/or work around should the implementers desire to do so.

The question: for anyone who's used the pipes library for any significant work, how does performance compare to iteratee/enumerator/iterIO? An element-based implementation, such as pipes uses, is unquestionably more elegant than the block-based implementations of other libraries, but I've yet to see an element-based implementation that matches the performance of a block-based version. Changing this probably wouldn't be too difficult, but the result would definitely not look as pretty.

In conclusion, I'm very glad to see some new libraries expanding the design space, as most other packages hew fairly closely to Oleg's work. However, I do think it's fair to say that the "pipes" design makes some tradeoffs that may impact some use-cases heavily. I don't think the ultimate iteratee design, if indeed one exists, has yet been found.
Shared publiclyView activity