Shared publicly  - 
 
Continuing the recent trend of me getting way pissed off over bullshit defenses of things more than I am at the original things.

Read this and then check out Brandon Boyer's comment.
http://therottingcartridge.wordpress.com/2012/02/22/whats-wrong-with-the-igf

Ok. Wow. Did I read this right? Did Brandon Boyer just waste a bunch of perfectly good words there trying to argue that the judges for a contest that awards thousands of dollars shouldn't be expected to actually play the games that are entered? What the fucking fuck?

I almost went through it point by point, but none of what he said defends the actual problem and I don't think people need to be shown why. They paid $95 to be fairly judged for a contest. They clearly did not get that. Acting like this is the entrants' fault, acting like it's not a problem that a contest with tens of thousands of dollars to award doesn't make sure they get more than a couple minutes from the judges?

I was just disappointed when it was them just fucking up. When they started acting like they didn't, when they started blaming the people they wronged, I got MAD.
8
2
Andrew M's profile photoDavid Papazian's profile photoMike Meyer's profile photoKenneth Backus's profile photo
14 comments
 
I shudder to think how many games weren't played at all.

edit: also from the next comment belonging to a judge "No one is obligated to play your game; you have to prove why you’re worth their time." so that answers my previous concern
 
With $95, and the amazing shower of fame that a nomination might bring, people expect fairness. "Fair" is highly subjective. There are too many emotions thrown around regarding the IGF (controversy as well as successes) so I don't think I'd ever want to enter, unless I was amazingly rich and felt like donating $95. Jokes aside, I would have to treat it like a donation / lottery or I'd feel too strongly about it.
 
I think I'm gonna go ahead and put this here, as I don't really feel like a blog post and some of these thoughts are too long for twitter. SO HERE GOES. Also my apologies I just wrote a novel on this topic but I hope it helps move the conversation forward!

DISCLAIMER: I have previously been both a judge (student and main, 2010) AND a jury member (Design, Grand Prize, Mobile, 2011), I am a personal friend of Brandon's, and this year I worked on a Grand Prize nominee (FEZ), an honorable mention (Hundreds) and an UNJUSTLY OVERLOOKED MASTERPIECE (Scoundrel). I am not affiliated with nor am I a representative of the IGF at all, and could not participate this year due to submitting all those games. So, make of that what you will!

NOVEL TIME!!

Let's talk anecdotal percentages real quick. As a judge (as opposed to jury member) in both the main competition and the student competition in 2010, I was super serious about playing every game I was assigned for at least 1 hour. That is a judge's responsibility, in my opinion. Other people have different views, which is fine. It's all voluntary, and different people see it different ways, whatever. As a game designer I felt that one hour per game was probably enough to get a really solid feel for it - either to begin to plumb its depths, or conversely to play it enough to discover in fact it HAD no depth, if that should be the case. 90% of the games that I played, the first few minutes of play were HIGHLY INDICATIVE of my overall impression. That is to say, if I had only dedicated about 5 minutes to each game, I still would have had highly accurate opinions regarding those overall experiences regarding their nominations and standing compared to their fellow entrants.

So, I would submit to you that this argument whether 5 minutes is adequate or fair is, practically speaking, about that other 10% of games. Because in a lot of ways that is what the IGF is about, I think. Finding those crazy games that nobody else found (which is getting harder and harder under the 24-7 spotlight of the internet enthusiast). About 5% of those games were so abysmal that I did not play them for the full hour, and I do not regret it. They were derivative crap that could only be appreciated through an ironic lens, if that. That might sound super harsh but man, how amazing is it that out of say 20 or 30 games only 5%, just 1 or 2 games, would be THAT bad? That's way out of proportion to any other selection of stuff that I can think of. But anyways, 5% were crap. Playing them beyond the first 20 minutes or so was an obvious waste of time.

Which brings me to the LAST 5%, which I think is the most important. These were the games that began with a whimper, and grew into something really special that totally charmed me. I feel like this is what a lot of us are afraid of, with judge dropouts or short judge sessions (even if the only evidence we have of these behaviors is one or two games worth of anecdotal evidence). And believe me, I am totally with you! I think Continuity, the sliding window platformer from the Student compo a few years ago, was a game that started very weak, in their IGF build at least, but around the half hour mark, it really stretched out and showed you what it could do.

The thing is, Continuity was awarded the Best Student Game of 2010.

The thing is, the IGF is full of AMAZING, newly discovered slow-burn non-traditional games every year. Somehow, this "broken" and "non-transparent" system always seems to find the most strange and wonderful games every fucking year. And every year there is an arguably well-meaning shitstorm on every imaginable form of social media available about how the IGF just doesn't work, and every year everyone somehow is unable to find a single example of a really outstanding game that was utterly ignored by the judges. Even this game, which the creators themselves say was unworthy of even entering the competition, was played for a full hour by one judge!!

SO. That is not a QED proof, that is just my anecdotal perspective on this thing: that every year, it seems like games that are self-admittedly not-worth-judging are under-judged, while glorious, thoughtful, slow and strange games receive buckets of nominations. In what way that system is "broken" I do not understand. REGARDLESS:

1 - If you're going to sign up to be a judge, do your job. The years when I was a judge I was retarded busy, and both my wife and I made sacrifices in order for me to judge my games. But we talked about it and decided it was worth it. If you sign up but then something comes up and you can't judge your games, man, TELL SOMEONE. The IGF relies on and trusts their volunteer judges, and most of them are fantastic. Don't ruin it for everybody, k?

2 - If you use the annual IGF controversy as a cynical way to get attention for the game you're about to lunch, please keep in mind that that particular tactic has a real human cost. You ruined a lot of real people's day in order for those blog hits. So, enjoy them. You may have lost everyone's respect, but at least you'll sell a few more copies on the App Store.

3 - The IGF will never be flawless. Not everyone who enters will be happy about it. However, the IGF can and does improve in concrete ways every year. Just a couple of years ago we didn't even know who the Honorable Mentions were! But there was a constructive public discussion about increasing transparency, and voila; an improvement to the system. This current bout of outrage, if we can stop being grumpy about things we made up, COULD result in yet another series of incremental improvements to a system that is, again, flawed, but FAR from broken. For example, there could be some simple checks in the database - did a game only get judged by half of its judges, because of a poor overlap in busy judges' schedules? Flag it, and assign it to some active judges! That can be systematized. Problems with iOS provisioning? That can be scripted too. As the number of games grows by over 100 submissions a year, scaling is going to introduce all sorts of problems. As a community, we can help by A) pointing out the shortcomings (BOY ARE WE GOOD AT THIS) and more importantly, B) making simple suggestions based on our knowledge of the system that could make it better for everyone next year.

4 - The idea that you are entitled to 8+ man-hours of intense, devoted playtesting for a measly $95 is a bit bonkers, isn't it? Keep in mind that A) I believe judges SHOULD play every game for at least an hour, B) somehow slow-paced games constantly fill the ranks of the IGF nominees, and C) somehow MOST of the most brilliant games seem to get nominated each year. With all that in mind... 8+ hours of devoted playtesting and detailed feedback for $95 is a really good deal. So good, in fact, that PERHAPS you are not in fact purchasing that when you enter. Perhaps you are just paying an entrance fee, similar to ALL OTHER HIGH-PROFILE COMPETITIONS IN ANY DISCIPLINE. Perhaps on top of that, due to a dedicated volunteer effort from the community, many judges play for an hour or more and leave very detailed feedback. Others, not so much. Perhaps entrants kind of need to keep all that stuff in mind.

Those are my thoughts. Be a good judge, make games worth judging, and help improve the IGF by making actionable suggestions. Recommendations have worked in the past and will work in the future and can help create a system that continues to struggle toward fairness despite the overwhelming quantity and disparity of entrants, system requirements and arcane security profiling required to operate a videogame contest.
 
Newsflash: Making a successful game includes doing the part that makes people want to install it and then play for at least 5 minutes.
 
I think it's an incredibly salient point that if judges sign up they do their job. When you have judges that don't play games - excepting the rare(?) cases when they can't run the things - it disrespects the creators, other judges, and the IGF organizers. Even Adam-as-judge played the abysmal derivative crap for at least a couple of minutes rather than not giving them their fair shake. No one (sane) expects judges to be perfect human beings, but at minimum they should be playing the games if they can, or yeah, notifying the IGF if they can't.

A lot of people seem to be defending the IGF, which leads to confirmation-bias-y arguments like "The IGF uncovers hidden gems all the times because they found them," but isn't the overall outrage more about the judges? People need to stop defending outright shitty judges.
 
Hey Michael! Yeah I mean I thought I was pretty clear about this, but I believe it is every judge's responsibility to play all their games as thoroughly as is reasonable, and to leave detailed feedback about the experience. Otherwise, I don't think they should sign up to judge.

From what I have heard and from our own experience this year, working with Testflight and the judges ended up being way harder than everybody expected it to be. At first it seemed like the obvious magic bullet, but that didn't pan out. I was on 2 iOS games this year, and there were different technical difficulties with each one, up to and including running out of provisioning slots. And we have used Testflight for our internal development easily and successfully for months.

So maybe take it easy with the scare quotes and implication that testing hundreds of mobile games is easy and that the IGF chair is just publicly lying about it?

Even in 2010, when there were barely any iOS games and Testflight didn't exist yet, I still had provisioning profile problems as a judge. So, I didn't get to play some of those games :( Some of that will always happen, probably, what with computers being computers, etc. But there have been some really good recommendations already about how to improve the iOS provisioning process to help remove obstacles between judges and their games which I think could make a big difference next year.
 
I like Boxleiter's perspective. The interesting side effect of disproportional playtime stemming from whatever is that there is an underlying competition (or filter) for marketing. That is to the IGF's benefit. If all your winners drop into obscurity because they overlooked the importance of getting your game out there, your festival would be brushed off. Having IGF prize winners moving on to strong releases self-fulfills the purpose of the IGF and respectively helps the indie game community.

The answer to whether the promotional side of indie game development should have any effect on IGF success is still not obvious to me.

I do think the judging interface should move to a more reddit style upvote/downvote so the judges could get a handle on which games were never played. Or even simply sorting a list by number of downloads in ascending order so you could see underplayed games.
 
Wow. Well, I think I am glad I called Brandon out on this. I don't know him, but while I do think he means well, he seems to have either not understood where entrants were coming from or what the things he said suggest to someone outside the IGF.

It must be nearly unavoidable that problems like these will happen. But when it happens, don't make excuses about how hard it is, or how busy your judges are and especially don't blame the entrants or the way their games work. It IS hard and judges ARE busy, but that isn't the entrants' problem! They paid the $95 under the clear understanding that their game would get seriously looked at by judges and have a fair shot in the contest. If that doesn't happen, whatever the reason, then the IGF failed to do it's job. The proper response is to own up to that, apologize, and continue to try and prevent it from happening again.

When you act like it isn't a problem, or that it is somehow the entrant's fault, you are sending the message that your contest is broken, intends to stay broken, and that games or developers like these are not welcome in it.
 
TestFlights needs an internet connection to work, it's not what we could call a solid proof.
 
After sleeping on it, I had a thought: Maybe the IGF should actually respond by promising LESS.

I think it would be fine if like only one judge had to play each game in a first pass to quickly cull the list down to something more manageable before they have lots of people look at them. Just make it clear that this is how the contest works and that feedback and getting seen by a number of judges is not what paying the entrance fee gives you.

The nature of the contest forces it to be a thing where not every kind of game belongs there. It's quite possibly better to acknowledge that and go with it. Be up-front about the inherent unfairness and how the system inherently favors certain types of games. If fewer people who never had a chance waste their entrance fee, that means that many fewer people feeling wronged.

Just an idea.
 
Having regularly organized competitions myself, the dilemma is this: the people who you want to volunteer their time to judge are the same people who have zero time. There is a balance to be struck and competing or not is your own choice but I think you should only be this prissy about it if you've ever tried organizing one yourself.
 
I called out the IGF last year (mostly privately/in person, not on the net) on pretty much the same issue, and things were looked into, and countermeasures put in place. Sure I might have "lost my chance" but I'm happy to have improved the IGF in some small incremental way.

I think discussion is good. Calling it drama is bad, boycotting the IGF is bad, saying people should STFU and deal is bad. Let's make it better, together, with calm and rational discussion. Or: Fucking chill, people.

More eloquently put into a blog post though: http://www.andymoore.ca/2012/02/igf-is-awesome-and-not-perfect/
 
Im very late on this, but it seems like the game in question (the one that started all of this) 'did' get a fair shake. The reason the IGF assigns 8 judges to your game doesn't mean necessarily that all 8 will play it. In fact, maybe the reason 8 are assigned is so that at least some of those 8 will play it. 4 judges playing a game, one for at least an hour feels to me like the game got a good shot. If one of those judges has really liked it, wouldn't they have recommended it strongly to their peer judges?

What it seems adam is arguing here is that the system may be a bit weird, but it does work. Sure judges miss some games sometimes, but there were a billion games entered this year, and by and large, the IGF does manage to find nearly every amazing special game that is entered. It feels like a lot of this discussion is avoiding that very practical fact to focus on the tiny detail that some judges aren't doing their jobs (which IS a problem), despite the fact that the system seems designed to work around that.

Its a volunteer judging system, it's never going to have 100% success. Would I like it to be better? sure. I submitted a bunch of games, at least one of which (Scoundrel, that adam mentioned) i really feel is a great game, but you know what? thats how things go in the real world. And what will I do about it? I'm going to submit it again next year, when it's more polished and more complete, and I think it'll have a much better shot.

I think if there was some global indie conspiracy, we'd have a problem here, but what we really have is a system thats trying hard to do it's best, and if theres any way it can do a better job I think brandon would be happy to hear it, but guys, come on, the IGF works pretty darn well.

I understand that theres this idea that brandon and adam and all the defenders of the process are avoiding the fact that some judges don't do their jobs, but that's because they're focused on the big picture, and expecting every single judge to do their job is absolutely crazy.
If we assume that there were 650 games this year in the main competition, and each one needs to be played by 8 judges
(650 * 8 = 5200)
and then each judge has 15 games (5200 / 15), thats 346 judges.

If anyone has any idea how to find Three Hundred and Forty Six highly respected industry professionals, writers, and indies who have not entered their games, who are all incredibly responsible, I'm sure we'd all love to know.
 
Even the judges not doing their job (or not doing it well) is kind of understandable. What gets my goat and really hurts the contest and leads to the continued anger is the way these problems are defended. Acting like the IGF is powerless to fix these issues, that they aren't problems, or that the entrants are to blame for their games not getting judged remains ludicrous and I think it is doing more harm than inadequate judging is.
Add a comment...