Shared publicly  - 
 
There's been a discussion about this on the Yesod mailing list, but I thought I'd open it up to the general Haskell community as well. We're finding it is very difficult for new users to get up-and-running with Yesod, because they need to:

* Install the Haskell Platform. That itself can be non-trivial at times, and certainly non-obvious for a beginner (do I use the one from the repos? but it's out of date...)
* Make sure they have updated build tools like alex (language-javascript requires version 3, but the HP ships with version 2).
* Actually install Yesod with Cabal. Due to dependency hell, this is very problematic.
* Sit and wait for everything to compile. I wouldn't have thought this is a big deal, but I've seen a few blog posts/talked with a few people who simply gave up when compiling took more than 10 minutes.

Any thoughts on easing this process? We're considering trying out Nix as a better way to install GHC+cabal+Yesod, but I'm not sure how that's going to go. We're also looking at creating new build tools to avoid dependency hell, but forcing Cabal to use "blessed" versions of packages that we know work well together.
4
Daniel Santa Cruz's profile photoGreg Weber's profile photoMichael Snoyman's profile photoKetil Malde's profile photo
67 comments
 
On Mac and Windows you could ship a native installer keyed to the latest Haskell Platform. It could include binary builds of all the packages at the versions you want. You'd plunk the whole blob down at someplace like /Library/Haskell/ghc-7.0.4/lib/yesod-bundle -- then register all the packages (if they aren't already), and finally, re-run Haddock to get the master index re-built.

For Mac and Windows I'd shy away from using Nix or any other non-OS supplied package manager: You don't want to force users into using one, since developer tend to be very finicky about such things.
 
And for Linux, the ideal is having 'apt-get¹ install yesod' just do the right thing. Or 'apt-get install haskell-platform' and 'cabal install yesod', but that doesn't seem to work well for you - which is quite unfortunate.

¹ Or yum or emerge or whatever your favorite Linux distribution prefers.
 
Perhaps supporting many platforms results in such huge difference in yesod installing experience.
On ArchLinux the steps are quite simple:

pacman -S ghc alex happy cabal-install
cabal update
cabal install yesod

That's it.

Yes, compilation step takes a few minutes. But i did it today on 3 machines, no problem.
 
I think we all agree that having a specific package created for each OS and distribution is the ideal, but we simply don't have the manpower to pull it off. That's why the Nix approach is attractive: they've already done the heavy lifting of getting something that works on all OSes and distributions.

Installing the HP on most distros these days is similarly pretty easy, excepting cases where (e.g.) Ubuntu has a 2-year-old version of GHC in its repo. It still doesn't solve dependency hell: far too often someone gets a bunch of confusing error messages when they type `cabal install yesod`.
 
+Aristid Breitkreuz In general, no, though due to the nature of Hackage and Cabal, we can't stop it. If someone uploads a new version of a dependency which confuses Cabal, we have to scramble to work around it immediately. This happens even if we're perfectly following the PVP, because (1) sometimes people make mistakes, (2) some of our dependencies don't follow the PVP, and (3) even with everyone onboard with the PVP Cabal still gets confused sometimes. A recent example:

http://groups.google.com/group/yesodweb/msg/b32c0e5e540f52d7
 
Yesod still works with GHC 7.0.4, right?
It's probably a good idea not to require more than what comes with the latest HP. It seems their focus is towards stability, not necessarily using the very latest version at all times.

On Ubuntu, I wish you good luck. They provides just good enough Haskell support so you can download an updated GHC from source and bootstrap.
If you want a Linux distro with really good Haskell support, I suggest Gentoo Linux.
If you're using the haskell overlay (which haskell developers should), then it's as easy as;

emerge yesod

The packages available have been tested and are known to work together.
Find details here; https://github.com/gentoo-haskell/gentoo-haskell
 
Less than you would think. Sometimes, Cabal will decide it can't install a newer version of a package, because an older version is already installed. So a build bot may be able to build things perfectly, whereas a user who installed some other package previously may not. That's why the idea of having a preapproved list of packages that work together is so appealing.
 
I'd like to have concrete examples of Yesod dependency hell when it occurs. I have been using Yesod as an example case for testing the modular solver quite often, because it has such interesting dependencies. But it's difficult for me to simulate real systems and update procedures. As for NixOS. I'm a Nix user and contributor myself, and we usually have Yesod running one or two days after a Hackage release. It's part of Nix's automatic build system Hydra, so binary packages are available, too. The NixOS link on the Hackage page points to the Hydra page.
 
Oh, and perhaps I should have said more explicitly to +Erik de Castro Lopo: yes, I think the modular solver should at least produce a valid solution more often than the topdown solver. You don't actually need the branch. The solver is merged in the HEAD. You still have to select the solver explicitly via the --solver=modular flag. Any feedback welcome.
 
+Lennart Kolmodin This isn't an issue of which distro I want to use, it's an issue that Yesod users are using every distribution out there. The answer to "How do I install Yesod on Ubuntu?" can't be "Install Gentoo."

+Andres Löh I'd be happy to give you examples via email (don't want to spam Google+ with this), but you might also be interested in subscribing to the mailing list and seeing the questions that come up. As for Nix... it's looking like a good option. I'm putting together a shell script now to automate the process, and would certainly appreciate input from someone who knows Nix: https://gist.github.com/1984981
 
+Michael Snoyman: Regarding the Nix script - there are several distro-specific packages on http://hydra.nixos.org/release/nix/nix-0.16 that would probably be preferable, but it's hard to auto-detect which one to use, of course. If you just install the yesod package to the default profile in the end, you'll get a GHC that's slightly awkward to use from a normal distribution. In particular, it doesn't work with cabal-install unless you pass additional flags. There's a better option. I'll try to explain this to you via email.
 
+Michael Snoyman Right, I took the opportunity for Gentoo to shine a moment :)
I'm using a Ubuntu machine too (or derivate thereof) so I've seen that it's probably not easy for newcomers.
I have yet to see the complex problems with dependencies that I hear people run into all the time.
Using Nix might be a short term solution, in the long run I hope that Cabal will become even better. We probably have a significant number of developers on Windows, making things easier for them too would be awesome.
 
Making Cabal itself more like Nix is certainly an option we're considering. While I'm certainly happy for Nix to get more users, it feels wrong to use weaknesses of Cabal to achieve this goal :)
 
I think this (esp. "people make mistakes") is good evidence of why the PVP doesn't really work. What we really need are specified interfaces for modules, with automatic querying of supported features. Sort of like a configure script in spirit.
 
People making mistakes is fine. Problem is that they cannot easily correct them.
In Gentoo, the dependencies the library/application developer specified are edited to better fit the real world. Dependencies are continuously either relaxed or made more strict, to build with new or old versions of libraries. To the user, it looks like it just works.
It's understood that what's going on with all dependencies might be difficult to grasp for the developer, especially if to include the dependencies' dependencies, and so forth. People will make mistakes.

What we need is for Hackage to have a similar set of super users as in Gentoo, which should be allowed to slightly alter published libraries so that the dependencies so that it all fits better together.

The problem with configuration scripts, and similar things, is that it's not declarative. We need to be able to translate the dependencies into the package systems used by Gentoo (hackport), Nix (cabal2nix), Arch (cabal2arch), Debian (cabal-rpm?), etc.
 
+John Lato Even with the exactly same interface we may have problems. Recently there was a 300x performance regression on base64-bytestring that affected Yesod pretty badly.
 
Generally, I agree with what +Lennart Kolmodin said earlier about stability. Yesod is developing at an amazing pace, but for people who always want the latest version of everything, there's a natural risk that things are going to break. It probably makes sense to identify intermediate more stable releases that target the latest platform and nothing beyond, and recommend that most users stick to that version. That may be difficult to achieve with the current speed of development, but should emerge naturally sooner or later. Having a book out there that describes a certain version and nothing beyond might certainly help with that :)
 
+Felipe Almeida Lessa that is a real problem. It's a specific example of what happens when the interface doesn't change but the semantics (or operational behavior) does. I don't have any idea of an automated solution for that.

+Lennart Kolmodin that seems like a rather labor-intensive process. I'd rather you spent your time fixing bugs or adding features than tweaking version dependencies (presuming that version-tweaking were unnecessary, which may not ever be entirely true).

Also, I think you misunderstood my point about configure scripts. What I would like to see is a system whereby after I write a package, Cabal figures out that each module needs (foo :: SomeFoo), etc., and where they're provided from. If bar-1.0 doesn't export a matching symbol 'foo', but bar-1.1 does, a tool should be able to figure out that my package needs bar-1.1. The point about configure scripts is that autotools allows you to query for specific functionality, whereas I would like to see language features allowing the coder to declare the use of specific functionality.

We need more than just names and types to do this properly, as Felipe points out. Probably some user-specified annotations will always be necessary. But suppose a new "bytestring" is released, with a few functions removed. If my code doesn't use any of those functions, I shouldn't have to manually update all of my dependency lists just to bump the upper bound when a (currently non-existent AFAIK) tool could figure that out for me.
 
+John Lato It is a labor-intensive process, indeed. But to me, it is bug fixing. I can assure you that every distro that even attempts to keep up with packaging popular Haskell packages has to do this. And they do it, and often without sharing the result with each other.
I'd rather like that we fix the problem at the source, try to correct the errors already in Hackage.

I'm not sure how some optional and possibly bug prone user annotations could remove humans from the equation.
 
+Lennart Kolmodin That's precisely my point. Bug-prone user annotations are all we have now. I think we can do better.
 
I'm trying out Nix, exactly because trying to install Yesod, Snap, Happstack was taking WAY too long. I had ghc 7.4.1 installed, and tried doing things just via cabal at first. Fail. Yesod compiled nicely, Snap didn't. One of it's deps doesn't compile with 7.4.1. So... on to cabal-dev. That worked, but it was very very manual, and having to re-install everything from source more than once is just getting a bit annoying. I'd be up for trying anything Nix related that you can come up with.
 
+Andres Löh How is it that snap builds fine for you? It depends on Crypto == 4.2.* but every version that satisfies these bounds fails to build on 7.4.1 due to the Eq/Show change. Snap does build fine if I fix Crypto manually, but there has been no response from the maintainer of Crypto on this front.
 
+John Lato The idea is compelling, and maybe already with small efforts it could become useful. It's probably common that you start using a library but forget to make the dependency stricter as you start using newer versions which brings new features. When you just add a dependency, it's likely that you make it stricter than required.To make it reliable, though, you'd need to do type inference with multiple versions available for each dependency. For example, how can it figure out if a type is an instance of a class or not? Also, what about code generating preprocessors which also are dependencies? And compiling packages with different flags? And dependencies like "if version X of library A, then export API 1 otherwise API 2".
I agree that such an application could find some mistakes, but I don't think that it's a silver bullet.
 
We seem to have an unfortunate trend - because GHC 7.4.1 is out, everyone tries to jump to it. I find it unfortunate that the default GHC install for for Ubuntu 12.04 is GHC 7.4.1.

The reason that Haskell Platform lags is that it wants to be a stable base of package versions. When a new GHC comes out, there needs to be time for packages to adjust and settle. This is why we (HP) lags ~5 mo.s behind. I understand that in this case, due to the circumstances of the 7.2 line, it seems like HP is "way behind" at 7.0.4 when head of GHC is at 7.4.x.

This is why larger projects, targeting wide use, and with many package dependencies, would be best to restrict themselves to the HP base and package versions that work with those packages. When installing Yesod, for example, the dependency on Alex 3 is a problem.
 
+Mark Lentczner I completely agree with you. And FWIW, while Nix offers ghc-7.4.1, the default version is usually the latest platform, thus currently ghc-7.0.4. And the latest version of Yesod builds fine with that one as well.
 
I agree with what +Mark Lentczner said as well. The plan has always been to used distros (and binary packages for Windows and Mac) as the way to distribute libraries to developers who want a more stable base to work on.

I realize that this is at odds with the furious pace we as a community are moving with, but I don't think this tension can be resolved. Things will eventually settle down, when people are willing to use older libraries and compilers in return for more stability.
 
Getting back to Michael's question, then: perhaps we should create our tools thinking about experienced devs only and leave the new ones to distro packages? Hopefully distros will be able to create nice packages once stabilized Yesod 1.0 is out. This is just an idea. What do you think, +Michael Snoyman?
 
+Felipe Almeida Lessa I think that's a great idea. Cabal/cabal-install currently work great for the casual use case of installing and using stable Haskell packages. IMO the biggest shortcoming they have is that there is very poor support for my workflow as a Snap developer (and I'm sure the Yesod guys would agree here) where we're constantly working with unreleased versions of multiple packages. The operative words here are "unreleased" and "multiple". The current toolchain works great if you're only working on a single project or all your dependencies have been released to hackage. cabal-dev helps with this problem, but it's not yet good enough.
 
+Doug Beardsley I have the same problem. I want to be able to have Cabal build a tree of packages, all in source form.
 
+Mark Lentczner I think this is not a workable proposal to have to restrict yourself to the HP and packages that build well with it (is there such a list in the first place ?). I find the packages available in the HP too restrictive and limited. Unless the HP is going to actively gobble more good and useful packages, I don't think it's realistic to expect people not to stray away from the HP.
 
I will see if I can come up with a scheme to package language-javascript with the output of Alex 3 included, so it is not needed for a routine install, only for actual development
 
So one common theme I'm seeing here that everyone (seems) to agree on: most users just want to have a stable set of packages that work together, and it's kind of inconsequential how that happens. Hackage as it stands is the wild west: anyone can upload any version of anything, and completely break things for everyone else. Distros each have to sort through this mess themselves, and people using cabal/hackage directly don't have any solution.

So here's a concrete proposal: we start creating profiles of blessed packages (like I mentioned at the beginning). A number of people will have control over the list, and the list will be versioned. We'll have some scripts that run over the list and ensure that everything in there can build together. Then in theory the distros will be able to take advantage of that list. We can also have different stabilities (beta, alpha, bleeding edge). We'll also need to either add support for these lists to cabal-install, or write a separate tool to use it.

These lists would be intended to be all inclusive, as opposed to the HP which is currently a very grueling process to get through.

The goal would be to have all major players involved in creating these lists, so that the "Haskell Blessed Packages 2012.0.1" includes Yesod, Snap, Happstack, hs2gtk, wx, criterion, and every other major system out there (apologies if I left anyone out).

I'm happy to coordinate this. I was already planning on putting this work together for just the Yesod community, if we can all collaborate and come up with a solution for everyone, all the better.
 
+Mark Lentczner +Vincent Hanquez As long as the HP stays the way it is now, I don't think people will stick to it. I believe it has literally been years since I've used it. The only time I use it is for bootstrapping cabal-install, which only seems to happen when I build a brand new system (and even then not always since distro packages may do it for me). Once I have a recent enough version of cabal-install, I use it exclusively and the HP ceases to be relevant for me. It might be relevant in a much grander view of compatibility, but my concern is to make sure my packages build when someone types "cabal install snap", and to fix them when someone complains that they don't install.

I think the only way the HP will have more adherence and impact is if it becomes much more intimately related to cabal-install, hackage, and the standard way people install Haskell packages. For instance, if the HP existed as a separate "production" instance of hackage, then I could see it having a lot more impact. I could envision doing something like...

$ cabal use production

...to restrict oneself to the HP repository. And then...

$ cabal use testing

...to switch to the bleeding edge (what's currently in hackage).

I don't mean to minimize the value of the work being done on the HP, but I think that as long as it is relatively invisible in the everyday life of prolific Haskell developers, people won't restrict themselves to it.
 
+Michael Snoyman You could start with the current version ranges in .cabal files. Just enumerate all possibilities and have a build-bot attempt to build them all. It would end up being fairly close to what maintainers should be doing already.
 
+Doug Beardsley I don't think the HP is intended for you. :) I'd say it's more for the mass of Haskell application developers that don't exist yet. It's also a common target for distros to target.
 
+Johan Tibell You're probably right, but in the current state of the world it's developers like us who are driving the "unfortunate trend" that +Mark Lentczner lamented. So there's a disconnect somewhere...
 
+Johan Tibell I'm not sure how can you recommend the HP to the mass Haskell application developers when there's lots of basic things missing from the HP. To name a few, xml, json, serialization, crypto, https, fastcgi. Those developers will have to go look outside the HP really quickly.
 
+Michael Snoyman I think that's a good idea to test. At least i could imagine that all distributions packagers could join efforts here.
 
+Vincent Hanquez The goal for HP (in my opinion) is for it to be like the Python standard libs. We're not there yet, and even if we get to that point you might need to install some extra libs (just like you do in Python.) However, you shouldn't have to install the bleeding edge of everything!
 
+Johan Tibell I think we're in complete agreement ;-) As a matter of fact, I went to check the python lib to find some examples of what was missing in the HP. I don't want to imply that everything should go in the HP, nor that the HP should be bleeding edge. However on the other hand, at the moment, the HP is too stable (borderline on frozen) and telling people that the HP should be enough for all their need is slightly unrealistic.
 
+Vincent Hanquez I agree unfortunately. The HP maintainers (Duncan and Don in particular) are too busy nowadays. We need some fresh people with time to do timely releases etc. I think we should release every 6 months like clockwork and start looking into adding some more libs.
 
I like the ideas in Hope, but most of it seems orthogonal to cabal-nirvana. For example, I want a solution to the build tools/system libraries issue, but nirvana isn't doing anything for that. I'd be happy to include some approach there.

The place where I see this intersecting is your breakdown of Hackage packages, with ratings and the like. I definitely thing we need multiple tiers of packages: everything that exists, then narrowed down to more and more refined. We already have that to some degree: we have the libraries included with GHC, then the Haskell Platform, and then all of Hackage. We just need some levels in the middle. nirvana is currently punting on this (it's just trusting me as the benevolent dictator for that list), and it looks like Hope has a more refined idea for how to make that list.
 
Would anyone please think about us, enterprise developers ? Blessing ? Nirvana ? It's really hard to sell yesod to managers and bosses with such talk.

How about certifying and cabal-certify ?
 
+Doug Beardsley : we have worked around most of the cabal limitations of working with multiple unreleased packages with our script/install script [1]. It uses cabal-src-install,which is really required unless you are using cabal-dev. I am abstracting some of this functionality out into a Haskell script and enhancing it to a certain extent with a package called cabal-meta, which is currently unreleased [2] . Its functionality is currently completely orthogonal to the discussion here.
[1] https://github.com/yesodweb/scripts/blob/master/install
[2] https://github.com/yesodweb/cabal-meta
 
+Michael Snoyman: As I tried to explain in the ticket, it isn't actually a bug. Please try if --preference does what you want. And you really might want to try with the HEAD version of cabal-install and then --solver=modular as well. I'm currently trying to move towards a new cabal-install release, so hopefully the next HP release will have it.
 
One more remark about the HP complaints (although it has been made in a similar way already by +Johan Tibell): it is not currently the idea that you restrict yourself to depend on HP libraries. There is obviously lots of essential functionality missing from them. However, as long as you depend on packages that are in the HP, it's a good idea to be compatible with the versions of the current HP. HP packages are depended on by lots of other packages, and not forcing users to upgrade any of these goes a long way towards avoiding dependency hell for these users.
 
Well, you can say it's not a bug, but I think adding an unused constraint causing a build to fail with a cryptic error message seems buggy to me. And even if the new version of cabal-install fixes some of these issues, we'll still be dealing with older versions for the foreseeable future, and I'm reluctant to rely on a preferences setting that may or may not be respected.

I agree that modifying downloaded files is not ideal, but I don't think we have much of a choice anymore. I'm willing to take any necessary hacks to get around the issues we've been struggling with for close to two years now.
 
+Michael Snoyman: I understand your frustration. I don't think very much of Cabal's error messages, but in this case, I don't think it is cryptic. And a constraint can't be unused "by definition", currently. But let's not argue about whether it's a bug or not, because I agree that you should be able to do what you want. Again, please try if --preference works. The old (topdown) solver doesn't backtrack, so a preference is nearly as good as a hard constraint anyway. The only reason why the modular solver would reject a preference is because there's no solution that can be found respecting it anyway.

I don't understand your point about old versions of cabal-install. If you believe you can ask everyone to install cabal-nirvana, then why couldn't you ask them to install the latest cabal-install? Also, while I agree that short-term hacks may be useful to improve the immediate situation, I'm still interested in improving the long-term situation.
 
Maybe I'm mistaken about cabal-install: I thought it was going to have dependencies on the newer Cabal library, which would require installing a newer GHC. If that's not the case, then I stand corrected.

The issue with --preference is that I can't predict or reproduce all the cases where dependency hell comes up. It may solve the issue in every one of my test cases, but will miss some corner cases, and we'll be back to square one. Modifying the 00-index.tar file seems currently to be the only sure-fire approach.
 
You're right. It does rely on a more recent Cabal, but that doesn't necessarily require installing a newer GHC.
 
I thought that installing new versions of Cabal caused all sorts of problems, same as installing a new version of containers or time. I just tested it, and it seems to have worked, so asking users to upgrade could be a solution. Thank you for correcting me.
 
The typical situation that causes problems is the following. You install package A-1 (for example, because it's included with the platform). Then you install lots of other packages (for example B) which happen to depend on A, and if they don't have any further constraints, A-1 will be selected for them. Later, you decide to upgrade A for whatever reason to A-2. Now you install new packages (C) that depend on A. Cabal will try to select A-2 for them, but you already have lots of packages (B) installed that depend on A-1, and these can't be used in a library C that also depends on A-2. So Cabal will start producing install plans that suggest reinstalling versions of B, changing their dependency from A-1 to A-2. This reinstallation then breaks all reverse dependencies that B has on your system at that point.

This can happen for Cabal as well as for containers, but containers has far more reverse dependencies on Hackage than Cabal, so it's much more likely to happen there. (BTW, why does yesod itself depend on Cabal?)

The HEAD version of cabal-install will warn you before such reinstalls are selected and also gives you the option to try to find install plans that avoid such reinstalls. However, that's not always possible. The only proper way is to switch to a Nix-style system for Cabal where several instances of a library with the same version, but different dependencies (and hence different hashes) can co-exist. (The example I mentioned above is also explained with pictures in my HIW 2011 talk at http://www.haskell.org/wikiupload/b/b4/HIW2011-Talk-Loeh.pdf )
 
Would cabal-dev be better than cabal in the long run?
Add a comment...