Ø Judges are volunteers who usually are successful in their own professions. They may have FIRST backgrounds by virtue of being mentors, or may be ex team members. However, sometimes they are from the companies that sponsor teams and may know very little about FIRST programs. There typically will be between 10 to 20 Judges at an event. They usually will not have worked with each other before. Their task is to work together to collectively identify the most deserving recipient for each award at that event. They spend most of their time interviewing teams in the pits and discussing their findings with other Judges, hence the unfortunate term "Pit Judges". Judges are typically assigned to one of the 2 award classes: Technical or Team Attributes. This assignment is usually based on their preference or expertise.
Ø Judges do not have much time to observe the teams and robots while they are competing. The Match Observers act as the Judge's eyes and ears on the field. They track the performance of each team and provide their observations to the Judges. The information can include the robot's performance, team members' behaviors, enthusiasm, etc. Judges find this input to be very important while making some award decisions.
Ø All Judges and Match Observers are managed by a "Judge Advisor". Among other things, the Advisor organizes and guides the Judges, assigns tasks and facilitates discussions. This person is the linchpin who keeps everything running smoothly. Most importantly, the Advisor is responsible for keeping the process on schedule.
Ø Advisors sometimes have Judge Assistants. They too can use this application to help keep the Judges on schedule.
To be effective, the Judge Advisor must define a process for judging and operate it. Different Advisors may have their own preferences for the specifics of the process. One such process that is supported by the application is described below.
Judging is usually spread over two days (actually, the second day is usually only half a day). The Advisor will break up the available time into a number of time periods. For example, the periods could be Day-1 morning, Day-1 afternoon and Day-2 morning.
The Advisor will then distribute the awards between these time periods. This reduces the workload on the Judges and allows them to focus on a smaller set of award criteria during each time period. Award decisions are expected to be completed in that period. For example, "Engineering Excellence" and "Innovation in Control" could be judged in the first period by the Technical Judges, while simultaneously, the Team Attribute Judges could be judging the "Entrepreneurship" and "Imagery" awards.
Depending on the number of Judges at an event, they may be grouped together, typically in pairs. This allows rookie Judges to learn from their more experienced partners. Each group would stay together while they interview teams. The group will be assigned to one class of awards: Technical or Team Attributes. Typically, half the groups will judge Technical awards and the other half will judge Team Attribute awards. There will typically be 3 to 5 Judge groups for each of the 2 award classes.
Next, the Advisor will assign a set of teams to each Judge group. Each group can reasonably interview about 10 to 12 teams in a time period. Judges may lose track of team details if they are assigned too many teams. For example, if there are 40 teams and 4 Judge Groups for each award class, then each group could be assigned 10 teams. Note that there could be events with 4 Judge Groups for Technical awards but only 3 groups for Team Attributes. Also, all Judge Groups may not have exactly the same number of teams assigned to them. The Advisor will make the assignments based on the experience of the Judges, team affiliations, personal preferences, etc. The application will help verify that each Judge team has been assigned a subset of teams to interview in each time period for a subset of awards, and that all teams have been interviewed for all awards at the end of all the time periods.
All the judge assignments are usually done before the event starts. However, some last minute re-assignments may be necessary if Judges do not show up as expected. The application will help the Advisor with all creating the assignments and doing re-assignments, when necessary.
When the event starts, the Judge teams interview the teams assigned to them during each time period. Judges will ask questions that will help them decide whether the team deserves the particular awards that are being judged during that time period. There will be some standard questions, but Judges may have their own preferences for questions. Judges will take notes on distinctive aspects of the robot or team, and also note down interesting responses to questions. Some Judges may want to take pictures to remind them of the robot during award discussions.
When Judges in a group have their individual mobile devices, the application will synchronize judging status information, but will keep notes and pictures private.
Each interview should typically take between 5 to 10 minutes. All teams must be interviewed within the allocated time period. The Judge group may have to return if the team is currently queuing for a match or is in the practice area. Sometimes, the team may be busy repairing the robot and may request the Judges to return in a few minutes.
After all assigned teams have been interviewed, each Judge group will be asked to nominate 1 to 3 candidates as "finalists" for each award. All finalists are then re-interviewed by the Judge groups that did not interview them initially. For example, if Judge group A picked teams 1 & 2 and group B picked teams 12 & 13 and group C picked team 22, then group A would re-interview teams 12, 13 & 22, group B would re-interview teams 1, 2 & 22, and group C would re-interview teams 1, 2, 12 & 13.
|
Judge Group |
Initial Nominations |
Award Finalists |
Re-interview |
|
A |
1, 2 |
1, 2, 12, 13 & 22 |
12, 13 & 22 |
|
B |
12, 13 |
1, 2 & 22 |
|
|
C |
22 |
1, 2, 12 & 13 |
To reiterate, each period has two phases: interviewing and re-interviewing. The awards being considered will be the same in both phases. However, the teams that each Judge group visits will be different.
Some Judge Advisors may schedule a third phase in the period. This will be for discussion. After the re-interviewing phase, all the Judges will confer and discuss the candidates to make the final award selections. All Judges have now interviewed all finalist candidates and are able to compare the different teams' merits. The winner for each award is usually decided by a vote with some balancing being done to ensure that no single team wins too many awards. Depending on the Judge Advisor, this discussion may be scheduled within each period or may be deferred till a later time. The notes taken during the interview will help the Judges make their decisions. The application will make it very convenient for the judges to view their notes during this phase, by making them available “offline” as well.
The interview and re-interview process is repeated in each time period. The assignments of Judge groups to teams may be different for each time period. Assignments may also change due to a Judge having to leave unexpectedly. The interview questions will vary based on the criteria for the awards.
There is a separate award called the Judge's Award. It is different because it is voted on by all Judges (both Technical and Team Attributes) at the end, after all other awards have been decided on. This award can be made for either Technical or Team Attribute qualities.
At all times, the Judges decisions are influenced by various factors:
· the team's confirmed awards in the current event ("spread the wealth" policy)
· the team's awards in previous events (to a lesser extent)
· the team's current performance on the field (usually for Technical awards)
· the team's behavior, attitude and enthusiasm
· the team's overcoming of special challenges (dog ate robot, school caught fire, ...)