How Does the Tournament of Simulations (ToS) Work?
Despite the image, we’re not currently accepting nominations for the 2018 Tournament of Simulations (ToS). But that will happen very soon… so stay tuned!
In 2016 we published an article on how to win Squiddies that explained in great detail how the process works. Even though that piece briefly discussed ToS and gave what is still the best advice regarding ToS (Just Enter!), we thought it would be helpful to pull back the curtain on this competition as well.
So here’s everything you need to know about the Tournament of Simulations:
Purpose
The primary purpose of ToS is to honor outstanding sims/role plays/games on an annual basis. The secondary purpose is to provide visibility of and access to these fantastic games to the broader simming and online role playing community. ToS is open to all role plays, regardless of club affiliation, prior awards won, or community status. All forms of play are eligible (e.g., play by chat, email, forum, post, etc.) and all genres are eligible (e.g., original, Star Trek, X-Men, etc.). ToS is where everyone comes in and competes on a level playing field.
Process
So how does it all work? Every year we ask the entire community to nominate their games. Anyone loosely affiliated with a game can nominate it. You can be the host, a regular player, or even someone in leadership of the club that a game belongs to. We de-conflict any double or triple nominations.
Nominations are typically made through email and confirmations should be received within 48 hours. To keep the judges’ workloads manageable, each play-by-post/forum/email sim only submits 10 consecutive posts to be graded while each chat sim submits a single session log. The nominators also have the option to submit short pre-story and post-story write-ups for context, but these are not graded.
Each nomination is then assigned to three disinterested volunteer judges for evaluation. Since some judges may generally score high while other judges may generally score low and some judges may have large ranges while others have small ranges, all raw scores are normalized based each individual judge’s mean & standard deviation of scores to give us adjusted scores. The three adjusted scores are then added together to give us adjusted totals.
Next, all judges confirm to the other judges that their scores were recorded & calculated correctly without indicating who specifically they graded to maintain anonymity. The administrator (that’s me!) is the only one to know who evaluated whom. And since I’m the administrator, I do not serve as a judge.
There’s no set number of prizes, therefore we look for natural breaks in the distribution of the adjusted totals to determine which games win and which games don’t. The natural breaks also determine which winners are designated as Great, Excellent, or Outstanding. Genre and format play no role. Typically, about 1/3 to 40% of all nominations receive an award, but again, this varies from year to year. Finally, the winners are announced!
Criteria
So what are the judges grading on? It’s actually pretty simple. It’s changed some over the years, but we seem to have settled into these three categories through the last few tournaments. Each game is graded on a 1-10 scale (0=worst sim ever, 5=typical sim out there, 10=best sim ever) in three categories for a maximum raw score of 30.
- Story: fresh story free of clichés and overused concepts; interesting plot developments
- Characters: original characters who show growth that adds to the story, rather than takes away from it
- Readability: combination of grammar, pace, and member participation
Here’s how all that plays out. As a reminder: (Raw Score – Judge Mean) / Judge Standard Deviation = Adjusted Score
Adjusted Score 1 + Adjusted Score 2 + Adjusted Score 3 = Adjusted Total
Here’s a made up example.
Judge 1:
- Mean: 14.6
- Standard Deviation: 5.389
| Sim | Story | Characters | Pacing | Raw Score | Adjusted Score |
| A | 7 | 3 | 5 | 15 | 0.074 |
| B | 9 | 8 | 7 | 24 | 1.744 |
| C | 5 | 5 | 5 | 15 | 0.074 |
| D | 4 | 3 | 4 | 11 | -0.668 |
| E | 2 | 5 | 1 | 8 | -1.225 |
Judge 2:
- Mean: 22.8
- Standard Deviation: 1.720
| Sim | Story | Characters | Pacing | Raw Score | Adjusted Score |
| A | 8 | 8 | 7 | 23 | 0.116 |
| B | 9 | 9 | 8 | 26 | 1.860 |
| C | 7 | 7 | 8 | 22 | -0.465 |
| D | 8 | 6 | 8 | 22 | -0.465 |
| E | 6 | 8 | 7 | 21 | -1.046 |
Judge 3:
- Mean: 11.8
- Standard Deviation: 2.926
| Sim | Story | Characters | Pacing | Raw Score | Adjusted Score |
| A | 5 | 3 | 4 | 12 | 0.068 |
| B | 6 | 5 | 5 | 16 | 1.436 |
| C | 4 | 3 | 4 | 11 | -0.273 |
| D | 4 | 4 | 5 | 13 | 0.410 |
| E | 2 | 2 | 3 | 7 | -1.641 |
Totals:
| Sim | Raw Total | Adjusted Total |
| A | 50 | 0.259 |
| B | 66 | 5.040 |
| C | 48 | -0.664 |
| D | 46 | -0.723 |
| E | 36 | -3.912 |
Now imagine 30 or so sims with scores like that from. We’re looking for natural breaks in the final adjusted distribution to determine who gets awards and which ones. A spreadsheet showing all of the scores with the names of the judges and non-winning sims redacted is sent to all participants and anyone else who asks for it. We also ask the judges to provide two pieces of feedback for each sim they grade: one thing they do well & and one thing that could be improved. As with all advice, it’s subjective. Take it for what you will.
Whooptydoo, but does it all mean, Basil?
You now know how the system works, which should help you tailor your nominations. However, back to the second paragraph, the best advice is to just enter! The clubs that have had the most sims win over the last few years, guess what? They’re also the clubs that have entered the most sims. Go figure! And if you don’t win, keep trying! You’ve literally got nothing to lose: We only announce the winners. No one except you, the judges, and I will even know that you entered.
Our Judges are Awesome
That goes without saying! Our judges literally volunteer hours of their time for absolutely nothing in return. Without their selflessness year after year, ToS simply wouldn’t be possible. Every year I find them to be more than fair, discerning, and engaging. They’re fantastic.
Furthermore, I encourage everyone to take a round from time to time as a judge. First, we need judges to make ToS work. Second, it’ll give you an even better understanding how ToS works. And third, it’ll further expose you to some great sims out there! I mentioned disinterested earlier: To ensure that there are no conflicts of interest in the judging, we only assign judges to grade sims that are from different genres from their own sims and the sims from their clubs.
And that’s it! That’s ToS! Good luck!
