I think I'm gonna go ahead and put this here, as I don't really feel like a blog post and some of these thoughts are too long for twitter. SO HERE GOES. Also my apologies I just wrote a novel on this topic but I hope it helps move the conversation forward!
DISCLAIMER: I have previously been both a judge (student and main, 2010) AND a jury member (Design, Grand Prize, Mobile, 2011), I am a personal friend of Brandon's, and this year I worked on a Grand Prize nominee (FEZ), an honorable mention (Hundreds) and an UNJUSTLY OVERLOOKED MASTERPIECE (Scoundrel). I am not affiliated with nor am I a representative of the IGF at all, and could not participate this year due to submitting all those games. So, make of that what you will!
Let's talk anecdotal percentages real quick. As a judge (as opposed to jury member) in both the main competition and the student competition in 2010, I was super serious about playing every game I was assigned for at least 1 hour. That is a judge's responsibility, in my opinion. Other people have different views, which is fine. It's all voluntary, and different people see it different ways, whatever. As a game designer I felt that one hour per game was probably enough to get a really solid feel for it - either to begin to plumb its depths, or conversely to play it enough to discover in fact it HAD no depth, if that should be the case. 90% of the games that I played, the first few minutes of play were HIGHLY INDICATIVE of my overall impression. That is to say, if I had only dedicated about 5 minutes to each game, I still would have had highly accurate opinions regarding those overall experiences regarding their nominations and standing compared to their fellow entrants.
So, I would submit to you that this argument whether 5 minutes is adequate or fair is, practically speaking, about that other 10% of games. Because in a lot of ways that is what the IGF is about, I think. Finding those crazy games that nobody else found (which is getting harder and harder under the 24-7 spotlight of the internet enthusiast). About 5% of those games were so abysmal that I did not play them for the full hour, and I do not regret it. They were derivative crap that could only be appreciated through an ironic lens, if that. That might sound super harsh but man, how amazing is it that out of say 20 or 30 games only 5%, just 1 or 2 games, would be THAT bad? That's way out of proportion to any other selection of stuff that I can think of. But anyways, 5% were crap. Playing them beyond the first 20 minutes or so was an obvious waste of time.
Which brings me to the LAST 5%, which I think is the most important. These were the games that began with a whimper, and grew into something really special that totally charmed me. I feel like this is what a lot of us are afraid of, with judge dropouts or short judge sessions (even if the only evidence we have of these behaviors is one or two games worth of anecdotal evidence). And believe me, I am totally with you! I think Continuity, the sliding window platformer from the Student compo a few years ago, was a game that started very weak, in their IGF build at least, but around the half hour mark, it really stretched out and showed you what it could do.
The thing is, Continuity was awarded the Best Student Game of 2010.
The thing is, the IGF is full of AMAZING, newly discovered slow-burn non-traditional games every year. Somehow, this "broken" and "non-transparent" system always seems to find the most strange and wonderful games every fucking year. And every year there is an arguably well-meaning shitstorm on every imaginable form of social media available about how the IGF just doesn't work, and every year everyone somehow is unable to find a single example of a really outstanding game that was utterly ignored by the judges. Even this game, which the creators themselves say was unworthy of even entering the competition, was played for a full hour by one judge!!
SO. That is not a QED proof, that is just my anecdotal perspective on this thing: that every year, it seems like games that are self-admittedly not-worth-judging are under-judged, while glorious, thoughtful, slow and strange games receive buckets of nominations. In what way that system is "broken" I do not understand. REGARDLESS:
1 - If you're going to sign up to be a judge, do your job. The years when I was a judge I was retarded busy, and both my wife and I made sacrifices in order for me to judge my games. But we talked about it and decided it was worth it. If you sign up but then something comes up and you can't judge your games, man, TELL SOMEONE. The IGF relies on and trusts their volunteer judges, and most of them are fantastic. Don't ruin it for everybody, k?
2 - If you use the annual IGF controversy as a cynical way to get attention for the game you're about to lunch, please keep in mind that that particular tactic has a real human cost. You ruined a lot of real people's day in order for those blog hits. So, enjoy them. You may have lost everyone's respect, but at least you'll sell a few more copies on the App Store.
3 - The IGF will never be flawless. Not everyone who enters will be happy about it. However, the IGF can and does improve in concrete ways every year. Just a couple of years ago we didn't even know who the Honorable Mentions were! But there was a constructive public discussion about increasing transparency, and voila; an improvement to the system. This current bout of outrage, if we can stop being grumpy about things we made up, COULD result in yet another series of incremental improvements to a system that is, again, flawed, but FAR from broken. For example, there could be some simple checks in the database - did a game only get judged by half of its judges, because of a poor overlap in busy judges' schedules? Flag it, and assign it to some active judges! That can be systematized. Problems with iOS provisioning? That can be scripted too. As the number of games grows by over 100 submissions a year, scaling is going to introduce all sorts of problems. As a community, we can help by A) pointing out the shortcomings (BOY ARE WE GOOD AT THIS) and more importantly, B) making simple suggestions based on our knowledge of the system that could make it better for everyone next year.
4 - The idea that you are entitled to 8+ man-hours of intense, devoted playtesting for a measly $95 is a bit bonkers, isn't it? Keep in mind that A) I believe judges SHOULD play every game for at least an hour, B) somehow slow-paced games constantly fill the ranks of the IGF nominees, and C) somehow MOST of the most brilliant games seem to get nominated each year. With all that in mind... 8+ hours of devoted playtesting and detailed feedback for $95 is a really good deal. So good, in fact, that PERHAPS you are not in fact purchasing that when you enter. Perhaps you are just paying an entrance fee, similar to ALL OTHER HIGH-PROFILE COMPETITIONS IN ANY DISCIPLINE. Perhaps on top of that, due to a dedicated volunteer effort from the community, many judges play for an hour or more and leave very detailed feedback. Others, not so much. Perhaps entrants kind of need to keep all that stuff in mind.
Those are my thoughts. Be a good judge, make games worth judging, and help improve the IGF by making actionable suggestions. Recommendations have worked in the past and will work in the future and can help create a system that continues to struggle toward fairness despite the overwhelming quantity and disparity of entrants, system requirements and arcane security profiling required to operate a videogame contest.