We appreciate that you have invested your time and resources participating in the SC2 Challenge, and we look forward to reviewing your Proposal. To ensure that you will be treated fairly, we are offering this explanation of the judging process. Once you have completed your Proposal, up to five (5) Judges will be assigned to review your recommendations. Judges will offer both a composite score and a comment for each of three (3) distinct traits. The judging process is designed to offer feedback and transparency. We hope that this explanation will clarify how your score is calculated and the way in which we seek to level the playing field for everyone.

You are welcome to test drive the actual judging tool that will be used to score your Proposal by clicking here.

Each of the three (3) traits will be scored on a 0-5 scale (in increments of 0.1). Examples of possible scores for a trait are… 0.5, 3.5, 5.0

The most straightforward way to ensure that everyone is treated fairly would be to have the same Judge score every Proposal; unfortunately, due to the number of Proposals, that is not possible – it would take too long and require too much of one person.

Since we are not having the same Judge score every Proposal, the question of fairness needs to be carefully explained. It makes sense that one Judge scoring a Proposal may be very harsh, and give everyone a 1.0, while another judge may be very easy and give everyone a 5.0. So, how do we make sure that no one is penalized (or given some unfair advantage) because of the Judges that they are assigned?

The first Judge is a lot more lenient than the second Judge, who gives much lower scores. If your Proposal was rated by the first Judge, it would have a much higher total score than if it was assigned to the second Judge.

Thankfully, we have a way to fix this problem. We make sure that no matter which Judges you are assigned, you will be treated fairly. To do this, we utilize a mathematical technique relying on two measures of distribution, the mean and the standard deviation.

The mean takes all the scores assigned by a Judge, adds them up and divides them by the number of scores assigned, giving us an average score. So, if a judge is easy, he will have a much higher average score than a harsh judge.

Formally, we can denote the mean like this:

\[ \overline{x} = \frac{1}{n} \sum_{i=1}^{n} x_{i} \]

The standard deviation measures the "spread" of a Judge’s scores. So, maybe two Judges both give the same mean (average) score, but one gives out a lot of zeros and fives, while the other gives a lot of ones and fours. We can see how it wouldn't be fair to you if we didn’t consider this difference.

Formally, we denote standard deviation like this:

\[ \sigma = \sqrt{\frac{\sum_{i=1}^{n} (X_{i}-\overline{X})^2}{n-1}} \]

So, to ensure that the judging process is fair, we rescale all the scores to match the judging population. In order to do this, we measure the mean and the standard deviation of all scores across all Judges. Then, we change the mean score and the standard deviation of each Judge to match.

We rescale standard deviation like this:

\[ x_{i} = \frac{x_{i}}{(\sigma_{judge}/\sigma)} \]

Then, we rescale mean like this:

\[ x_{i} = x_{i}-(\overline{x}_{judge}-\overline{x}) \]

Basically, we are finding the difference between both the distributions for a Judge and those for all of the Judges combined, then adjusting each score so that no one is treated unfairly according to which Judges they are assigned.