|
| Thread: A Judge's Dilemma | |
|
Gnoll_Mage
Responsible
Supreme Hero
|
posted March 07, 2007 09:40 PM |
|
|
A Judge's Dilemma
Hi all,
I've marked a few entries by now, and I have a slight problem, something like this.
Last year I got a little annoyed about the judging system. It didn't necessarily award the best towns the best marks - that's why I changed the system this time round. However, there has been an inevitable problem for me. Entries that I am not particularly keen on as ideas are scoring higher than others that I do like (alternately you can replace "like it" with "it would fit well into HoMM"). This is simply because the judging recognises quantity as well as quality. Now this entries-scoring-oddly thing is of course a bit of an exaggeration, but you can see how it might come about - if an entry's creature line-up is fantastic but there are very little `additions` such as history (which aren't strictly needed to evaluate an idea as to its suitability for the HoMM universe), they won't score as well as someone who put lots and lots of effort into what was ultimately a poor idea at the start. Again I will say that those are extreme cases, but smaller-scale versions of the scenarios have been occuring.
And it's not that I don't want to recognise hard work - indeed, this should be judged for, and the system won't be changed by any means - however, ultimately we are making towns to fit into the world of Ashan. However much is written on a forge-like town it simply won't fit as well as the average Naga town, or whatever.
For this reason, as I am marking all the entries, I am not just giving you the actual faction mark as dictated by my own sheet (!), but also a personal `how much I'd like to see this in the game` grade (using the very bad, bad, poor scale at the moment, although this might change - I may for example create a crude rank of my favourites). Sometimes these are noticeably different; I'd encourage other judges to do a similar thing. The Cavern is a good example here - there are chunks missing but I quite like the town in itself.
And whilst I'm here, this is a taster of my marking board so far (I haven't marked everyone yet), which could change at any time - hence me not putting in crude marks:
Normal judging:
 First place: Bastion (really enjoyed reading this one TDL)
Second place: Guild
Third place: Stampede
`Personal` scoring:
First place tied: Bastion, Cavern
Second place: Guild
____________
|
|
GenieLord
Honorable
Legendary Hero
|
|
posted March 07, 2007 09:58 PM |
|
|
I can understand what are you talking about.
The question is what can we do about it.
|
|
Gnoll_Mage
Responsible
Supreme Hero
|
|
posted March 07, 2007 10:03 PM |
|
|
Well as I said providing a personal ranking seems a good idea - but of course that counts to nothing.
This all harks back to something I said at the start of the contest - I believe I gave a link to Gamespot and mentioned their `tilt` system. For our method, this was never implemented - strangely, I don't feel I could put it in, yet I do feel that the judging is a little too objective (as was expressed in the previous post) - hence another dilemma...
____________
|
|
alcibiades
Honorable
Undefeatable Hero
of Gold Dragons
|
|
posted March 07, 2007 10:26 PM |
|
|
Well, I'm not sure I understand the problem. As I understood the spreadsheet, you were going to give your scores not based on quantity, but on quality. Thus, if you rated creatures between 0 and 25 points, you should give points based on things like a) how well conceived they are, b) how well they fit each other and the theme, c) whether they add to the game in terms of gameplay, etc. Thus, the mark you provide on each section should reflect not only quantity (of course you'll get 0 if it's not there), but also quality, and overall rating should come out to match your all-round oppinion.
EDIT > But yes, a faction can have a really really nice creature line-up (and only that), and score 100 % in that category, and still not come out as the best, that's true. But that was the decision that was made prior to the contest, that one wanted to include all the surrounding stuff also in the rating, and I think it's too late to change that now.
____________
What will happen now?
|
|
Gnoll_Mage
Responsible
Supreme Hero
|
|
posted March 07, 2007 10:38 PM |
|
|
Of course it's too late, but as I said I wouldn't want to change it. It would be wrong not to recognise that stuff. It just means that sometimes some towns are done an injustice a little - which is why I'll be giving the other markings too.
I realise that the sheet recognises quality (I did say I was exaggerating above), but it also recognises quantity. Besides, I think quality is the wrong word - something can be good quality but still not be to my liking. And how much I like a town's line-up can only be recognised in the creature section, even though this has a big effect on my overall impression of the faction.
It's certainly a difficult topic and the solution for my eyes is to have the secondary ratings. At least that way I don't feel guilty about a town I like being too low.
____________
|
|
Fofa
Famous Hero
Famous? Me?!
|
|
posted March 07, 2007 10:46 PM |
|
|
Quote:
Normal judging:
First place: Bastion (really enjoyed reading this one TDL)
Second place: Guild
Third place: Stampede
`Personal` scoring:
First place tied: Bastion, Cavern
Second place: Guild
Bastion being in first place is no surprise. But I'm still surprised that I'm second. 
People seem to really like my Guild town though; at first, they primarily perferred my Prison town (or that's just me).
____________
Creator of the Guild and Prison towns for ICTC.
|
|
Elvin
Admirable
Omnipresent Hero
Endless Revival
|
|
posted March 08, 2007 01:36 AM |
|
|
I too am judging the towns according to what they offer, how complete they are and how they fit in the setting- my personal preference has little impact. Maybe some parts such as history and skill percentages eat away more at some towns than I'd like but hopefully we'll have a better system next time. I cannot really say that the current one is disappointing at all, there are just a few minor things.
____________
H5 is still alive and kicking, join us in the Duel Map discord server!
Map also hosted on Moddb
|
|
EliteKill
Promising
The Starless
|
|
posted March 08, 2007 06:20 AM |
|
|
|
How about penalizing points for incomplete towns, or for entries that are just good in one area?
|
|
alcibiades
Honorable
Undefeatable Hero
of Gold Dragons
|
|
posted March 08, 2007 09:11 AM |
|
|
I think the idea was the opposite, actually - that the current system penalizes incomplete towns quite hard (maybe even too much).
____________
What will happen now?
|
|
Geny
Responsible
Undefeatable Hero
What if Elvin was female?
|
|
posted March 08, 2007 10:26 AM |
|
|
I haven't seen the judging sheet, so I don't know exactly how you judge. However, sometimes when judging something as complex as full towns there is a small percentage (5%-10%) left to the judges' opinion. That section is used to mark the overall feeling or a special creativity (or the lack of it) of the faction. If you don't have this section already I think it would be wise to add it. Moreover, as far as I know, only the final grades count, so any grades given so far are actually subject to change if you'll add a minor correction to the judging sheet.
____________
DON'T BE A NOOB, JOIN A.D.V.E.N.T.U.R.E.
|
|
alcibiades
Honorable
Undefeatable Hero
of Gold Dragons
|
|
posted March 08, 2007 01:06 PM |
|
|
Quote:
I haven't seen the judging sheet, so I don't know exactly how you judge. However, sometimes when judging something as complex as full towns there is a small percentage (5%-10%) left to the judges' opinion. That section is used to mark the overall feeling or a special creativity (or the lack of it) of the faction. If you don't have this section already I think it would be wise to add it. Moreover, as far as I know, only the final grades count, so any grades given so far are actually subject to change if you'll add a minor correction to the judging sheet.
We tried to take that into account. I don't know where the official link to the spreadsheet is, but you can see it here. Basically, we tied the points up so that 25 % goes into Creatures, 25 % into History and Race, Skills and Abilities gets 15 %, Town gets 10 %, Heroes get 8 %, Miscelaneous Basics gets 7 %, and then there's 10 % to add as bonuses where one sees fit. That was the compromise we reached back then that we agreed would cover best the aspects of the factions.
____________
What will happen now?
|
|
Elvin
Admirable
Omnipresent Hero
Endless Revival
|
|
posted March 08, 2007 01:14 PM |
|
|
Just a thought...
Maybe it would be better if the points were focused on units, buildings, skills/hero then history. All other things could be used as bonuses to avoid reducing the grade.
____________
H5 is still alive and kicking, join us in the Duel Map discord server!
Map also hosted on Moddb
|
|
alcibiades
Honorable
Undefeatable Hero
of Gold Dragons
|
|
posted March 08, 2007 01:20 PM |
|
|
Well, like I said, personally I think it's too late to change it, but I'm not in the judge group, so I suppose that's not my place to say. Personally, I don't feel History is the most important thing for a faction, but it was decided back then that this category should have so much weight because it would focus not only on the History itself, but also be a main proxy for how good the faction fitted into the Heroes 5 universe in terms of racial definition and relationships to the other factions and the general mythology. These things are pretty crucial in my oppinion - more so than the History, which when all comes to all, has no significance to the actual game.
____________
What will happen now?
|
|
Ted
Promising
Supreme Hero
Peanut Exterminator
|
|
posted March 08, 2007 04:38 PM |
|
|
just to say, i think that an award for the creation of the town timeframe,
like evil_warriors town, it was made only 1 day before the deadline, so i think it should get a bonous, but GLs was made ages before the contest so it should get a penalty (nothing big, like 1-5% maybe?)
just a thought
____________
Visit my Site!
|
|
alcibiades
Honorable
Undefeatable Hero
of Gold Dragons
|
|
posted March 08, 2007 06:18 PM |
|
|
No, that doesn't make sense. Daystar and I posted the Conflux three days before deadline, but that does not meen we didn't start work long before that, as you probably know.
____________
What will happen now?
|
|
GenieLord
Honorable
Legendary Hero
|
|
posted March 08, 2007 08:48 PM |
|
|
Quote:
just to say, i think that an award for the creation of the town timeframe,
like evil_warriors town, it was made only 1 day before the deadline, so i think it should get a bonous, but GLs was made ages before the contest so it should get a penalty (nothing big, like 1-5% maybe?)
just a thought
Fofa posted her towns 11 months ago.
Still, I don't think it will be fair to give her large penality.
And those who posted their factions two days before the contest started, it's their problem. 
They knew about it, they had enough time. They could post it a long time before.
It's not like we kept on this contest as a secret and they just found it about it...
|
|
Ted
Promising
Supreme Hero
Peanut Exterminator
|
|
posted March 08, 2007 08:50 PM |
|
|
fair enough, Alc has got a point, besides, i did write 1-5% increase/decrease, so it would effect it, but its not nessicary
____________
Visit my Site!
|
|
Gnoll_Mage
Responsible
Supreme Hero
|
|
posted March 08, 2007 11:20 PM |
|
|
Quote:
I think the idea was the opposite, actually - that the current system penalizes incomplete towns quite hard (maybe even too much).
That's correct.
As I hope you can see, it's clearly difficult getting the ultimate balance between all the different factors - and not only is this the case, but it also seems that marking in one way is mutually exclusive with others. For example, there's not only the problem of balancing the multipliers of town, race, skills etc. (`solvable`, as such), but also that it is impossible to mark a faction on both its promise as an idea and on its completeness - you have to focus on just one; or so it seems. This is why the two-rankings system seems the best idea.
____________
|
| |
|
|
