Assessment for Judging

Assessment for Judging
by on (#213888)
Being a teacher, assessment methods (rubrics, grading scales, and the like) are all of particular interest to me.

How are you each approaching judging?

What factors are you including in the overall categories, etc.?
Re: Assessment for Judging
by on (#213921)
It's hard to tell. I usually begin from top to bottom. I usually pick up muy most favourite in each category and give it full marks if I think it deserves it, or almost full marks. Then I pick up my next favourite, and measure how much "worse" it is so I give it the same, 1 less or 2 less or ...

When I'm done I reconsider my decisions. Maybe I was too harsh with my least favourite. Maybe the entries awarded the less points should have higher marks.

So it is quite relative. I don't know if this is a good or bad method, but it's the method I use most of the time in every task where I need to qualify or simply pick an arbitrary order to stuff.

On top of that, I think each one has a different approach to what topic or category is most important. For example, I don't care if a game is original as long as it is fun. Well, I do care that it is original, but that's less important. I don't care if it's technically top-notch as long as it looks good and plays great, and feels polished and complete.
Re: Assessment for Judging
by on (#213924)
One way, though it requires a bit of prequisite legwork, is to set up atomized and universally functioning merits, and assign them a value (nominally 1). Each merit which you find true rewards a point (or a half, or whatever value you've assigned to that merit). Since some categories can reward more points than others, you'd either have to come up with more merits or attribute them with more points each.

the apple was:
-rinsed (1pt)
-good for baking (1pt)
it had:
-an appetizing colour (1pt)
-a nice scent (1pt)
-a distinct flavour (1pt)
-a nice texture (1pt)

Maybe if the apple had a really nice scent, it might be worth 2pts.

Or the merits could be accumulative, perhaps like this:
-the student has grasped the elementary concepts needed to pass the course and continue with the next (threshold 1pt)
-the student has been able to reflect on the concepts and relate them to other related fields of study (threshold 3pt)
-the student has achieved a confident and profound understanding of the subject. (threshold 5pts)
Re: Assessment for Judging
by on (#213927)
I ordered them by how much I like them, then adjusted their scores until the numbers came out the way I wanted. Unfair? maybe.

I also subtracted points for bugs at submission time. Sorry. I know alot of them had been fixed before voting, but deadlines are deadlines. I'm a stickler.
Re: Assessment for Judging
by on (#213932)
dougeff wrote:
I also subtracted points for bugs at submission time. Sorry. I know alot of them had been fixed before voting, but deadlines are deadlines. I'm a stickler.

I think it was mentioned somewhere that we should be rating them based on the version available at the deadline/submission, and not based on later improvements, so I agree with this.
Re: Assessment for Judging
by on (#213947)
That's why I haven't still commited to introduce any of the enhancements / corrections suggested for my entry. I will do when the results are posted, for the cart.
Re: Assessment for Judging
by on (#213998)
i am also holding off on updates until voting is completed, and will be voting on submissions, not edited versions.
Re: Assessment for Judging
by on (#214169)
I feel bad voting on games I haven't been able to beat.

Is it appropriate to PM requesting savestates for endings or that show essential content a developer might want included in the judging of their game?

Did anyone get stuck in Grunio? If so, I'd gladly PM a password list and a savestate that shows the all-carrots ending.
Re: Assessment for Judging
by on (#214182)
I only got about two thirds in on Grunio, at that point I just couldn't figure out any solutions. No need to send a password list though.
Re: Assessment for Judging
by on (#214190)
There's a cheat in Miedow you can use to skip levels. It's not that hard to find (specially if you check the sources ;) ), but if somebody is interested just PM me.

The ending sequence (which goes from the moment Cheril "awakes" after level 10 to the end and which is playable) is quite nice to experience.
Re: Assessment for Judging
by on (#214240)
I am unfamiliar with voting on homebrews or the contest in general. Can anyone vote? When is the voting? Are there rules explaining the voting and contest process?
Re: Assessment for Judging
by on (#214242)
Although judging is not open for anyone (essential for preventing it from becoming an internet popularity contest, IMO), I'd still love to hear your assessment of our entry at least.
Re: Assessment for Judging
by on (#214268)
I think it would be nice if people posted their own ratings just for fun, specially if commentary was done. I always like to know what people think about my design choices because, as some of you will agree, working for some time in your project basicly blindfolds you.
Re: Assessment for Judging
by on (#214461)
Thanks for the insight into your approaches.

Judging was definitely enjoyable. It's nice to be motivated to take the time to play and analyze these games more than I may have gotten around to otherwise.

Ɓukasz and I assessed separately and then planned to submit the average of our scores. I was caught a bit off guard by being restricted to whole number scores. At first, I'd just planned to round up any category entry that fell between whole numbers. But then I realized some games were getting total score boosts of 1.5 points or more (as they had multiple categories being rounded up). So, then we decided to round each (non whole number) category up or down to ensure that no game received more than a half point advantage from rounding.

As for my half of our judging, I assessed each game independently of each other, using the attached sheet which broke down most categories into smaller subcategories and a scale for each.
NesDevMTeeScorecard.pdf [23.08 KiB]
Downloaded 286 times

Anyway, I printed these and penciled them in fairly messily during my last play session, but if anyone wants my individual assessment of their entry, I'll gladly type up a legible copy with notes when I get a chance and shoot it your way.