Game Manager: User Guide for Judge Advisors
Game Manager: User Guide for Judge Advisors
Introduction
This guide describes how GMS facilitates the administrative tasks of Judge Advisors at an event. Separate guides are available for Judges for Judge Match Observers, and for setting up GMS. GMS is not a replacement for the technical expertise of any roles. Privacy of judging information is a critical attribute of GMS. Judging is supported for FRC, FTC and FLL.
The work flow for a Judge Advisor typically consists of the following sequence of steps before the event starts. Each step is detailed in subsequent sections.
- Add or modify awards for that event, if needed (typically for Off Season events)
- Customize rubrics and checklists for awards for use by judges, if desired
- Distribute the judging of awards into periods (optional)
- Add or update information about the judges volunteering at the event
- Form groups of judges for each period
- Assign teams to each Judge Group in each period
During the event, judging assignments are sent to each Judge’s mobile device. The Judge Advisor can do the following:
- Start a count down timer indicating the time left in the period. This timer is displayed on all judges’ mobile devices and on the Match Observer’s screen.
- Track the progress of each Judge Group
- Receive comments from Match Observers and optionally share them with Judges and Robot Inspectors
- Enter judge’s choices for team finalists at the end of the interview phase
- Start a re-interview phase for the period
- Enter judge’s selections for team finalists after re-interviewing
- Facilitate discussion to select an award winner
- Share information with the Lead Robot Inspector
The GMS system comprises a server that runs on a laptop and an app for mobile devices. Judges and Robot Inspectors carry mobile devices (tablets or phones) that communicate with the server. Judge Match Observers have a separate system that is described at http://gms.pejaver.com/UserGuide-MatchObserver.htm.
This document needs to be improved. Also the re-interview functionality needs to be improved. All this will happen if there is sufficient interest.
Awards
Awards are usually set by FIRST for In-season events. For Off-Season events, the Judge Advisor can define new awards. This is a good time for creating off the wall awards, like MacGyver Award for Innovation in Material Usage, Martha Stewart award for Best Pit Design and Display. Click on the Edit Awards button on home screen or select View -> Awards. Select an Award to Edit or Delete from the Select Awards screen. You should not need to delete awards. To make a copy of an award, click to select it and then click Add Award.
If you see strange awards, verify that the correct FIRST series (FRC, FTC or FLL) is selected under Configuration.
Award Rubrics
Each award has a rubric, a set of guidelines for that award. Each guideline can have a list of choices and each choice can have a value. These will be detailed in the following sections. The JA can modify/edit the award rubrics for the Judges. Match Observers have a separate rubric that can also be modified.
Rubrics help Judges gather consistent data and record precise observations during interviews. They need to be worded appropriately to be effective. The Award guidelines will be displayed on the Judge's tablet.
Each guideline can be an Instruction or a Question. A Question ends with a question mark (?), and can have a few Options as responses. If no options are specified following the question mark, the default options will be ‘Yes’ and ‘No’. Instructions should not end with a question mark. Blank lines can be inserted as separators.
The Rubric for the Creativity award is shown in the example below. Note the two guidelines that are marked in blue and in red.
The rubric will be displayed on the Judge's tablet as shown below. The Judge can pick one of the available options by clicking on it. Long press on an option to clear it.
Note the guidelines marked in blue and red. This generates a corresponding screen like the following image below, on the judge’s device.
Judges download rubrics for each period when they log in and accept their assignments. If multiple awards are being judged in a period, then the rubrics are combined and presented to the judge on one screen.
Each answered Question will have a value, which can be used to compute an average value for the Interview. Unanswered questions are ignored and do not contribute to the average. Each Option has a value. If selected, this becomes the value of the question if the value is selected. Option values can be explicitly specified as integers or decimal values and can even be negative. If no value is specified for an Option, the default value is the index of the Option (starting with 1).
Examples are:
Team demonstrates respect and Gracious Professionalism to all?
1, 2, 3, 4, 5
Creative component must work reliably?
1:2, 2:4, 3:6, 4:8, 5:10
Team must submit an Engineering Notebook with an Engineering Section?
No:-9, Yes:9, Dog ate it:3
In the " Gracious Professionalism" question, the value of the question will be between 1 and 9 depending on the option picked. These are the 'default' values. The " Creative component" question has double weight. The " Engineering Notebook" question rightfully dings a team (-9) for not having a required item, and offers partial credit (3) for trying.
Team Ratings for Awards
Judges can evaluate a team for multiple awards during an interview. An Interview Rating is assigned for each award; it indicates a Judge's opinion of the team's suitability for that award. These values are merged with ratings from all other Judges and helps identify finalists for awards. There are 4 steps for calculating a team's final award rating.
1. Interview Ratings are a value in the range of 1 to 5, for example, 3.2. Depending on GMS configuration setting (JA's choice), Ratings can be either manually entered by the Judge, or automatically computed from the Award rubric responses. This configuration applies to all Judges and Match Observers at the event.
For manual rating, the Judge gets to select a button for 1 to 5. The automatically computed Rating is the average value of all Rubrics in an award interview. Rubrics without a response are not counted in the average. The averaged value will be scaled to a 'standard' range of 1 to 5. This value is sent to the server for further processing.
2. On the GMS server, the individual ratings are first normalized. Each Judge may have a tendency to consistently rate teams high or low. For each Judge, the ratings for all interviews are averaged. On a scale of 1 to 5, the average value should be 3.0. If it is different, all the scores for this Judge are multiplied by a common factor (3.0/average). The resulting values are limited to min and max values of 1 and 5. Interview topics that were not scored are not taken into this calculation.
3. Next, if the Judge Group has more than one Judge, the group average is computed. Each Judge that reported their opinion has an equal weight.
4. Finally, if multiple Judge Groups interviewed the team for the award, an average is computed of all the group ratings. Each group will have an equal weight, regardless of the number of Judges in the group.
This final rating is used to sort candidate teams for each award. Note that this rating is only a suggestion and need not be followed.
Judging Periods
In this step, judging activity is divided into periods and the awards to be judged in each period are assigned. There can be up to a maximum of 5 periods at an event. The JA can configure in as many as are needed, though 3 would be typical. Periods that are ‘empty’ will be ignored.
For each period, the JA can also choose a name that is more meaningful than ‘Period 2’ (for example, “Saturday Morning” or “Regional awards”). However, the JA must specify whether the period is on Day 1 or Day 2, because that helps match up the judge’s availabilities. A period can run into multiple days, which would happen if only one period has been chosen for the entire event. This should be configured carefully; otherwise judges may not appear in selection lists as expected. If neither day is selected, then this feature will be disabled for the period and judge availability will not be checked. Each period can have two phases: interview and re-interview. The time durations for each phase can also be specified.
The next step would be assigning awards to each period. Awards are separated into Technical ones and Team Attribute Awards. In the image below, the tables on the left show the awards that are yet to be assigned to periods. The tables on the right show the awards that have already been assigned and will be decided in the selected period.
If the tables on the left are empty, click on “Show Awards from All Events” to show the predefined FIRST awards. New awards can also be created for local events.
Once the awards to be judged in Period 1 have been selected and assigned, the JA should select period 2 by clicking on the number at the top. This process can be repeated until all awards have been distributed between the periods and the tables on the left are empty. If any changes have to be made, that can be done easily by moving awards to different periods by de-assigning an award and reassigning it to another period. The configuration should be saved by clicking Save and clicking Done to return to the main screen.
Judge Groups
To form Judge Groups, the Judge Advisor has to first select an event. Note that the Event Period selector is now active in the Select Judge Groups screen.
The available judges in the selected period show up on the upper left table. The table typically shows judges that have volunteered for the selected event. However, judges from other events can be displayed by clicking on the ‘All events’ button. The JA can remove a judge (in case they do not show up at the last minute) from an event in the Modify Judge screen. Judges are shown only if they are available on the days of the period. (Each period has been configured to run on a specific day (or days) and judges have indicated which days they are available. Also, if the days for a period have not been selected in the Select Period screen, then judge availabilities will not be checked.) The JA can change the name of a period from this screen, but cannot change the period timing (i.e., which days). That can only be changed in the Select Period screen.
By highlighting one or more judges from the upper left (unassigned judges) and clicking on “Add Judge”, moves them to the upper right table (assigned judges). The JA can add or remove judges as desired till they are comfortable with the group. When an event has been selected, double clicking on a judge’s name will allow the JA to assign it. (This is different from what happens if an event has not been selected.)
The JA can select a name for the group and whether they will judge Technical or Team Attributes awards in the period. Note that defaults are provided for the Group name and type. This can be overridden if desired.
Now by clicking on “Form Group”, a new group can be created in the lower right table. This new group is associated with the current period. The team affiliation column lists a union of all the teams of the member judges. The Experience column shows the maximum and minimum years of experience of the different judges in the group.
Groups can be edited by selecting a group from the lower right table (Assigned Judge Groups). The judges in the selected group will be added to the upper right table. Multiple groups cannot be selected for edit. The JA can also re-form the same group with a different name. The new name shows up only in the current period and one may have to rename the group in each subsequent period. Pressing ‘Form Group’ will update the group being edited. Press the ‘Clear Edit’ button to stop editing and create new groups.
To add or remove a Judge from a group, click on the Judge Group in the bottom right table. The group members will appear in the top right table. Add a Judge from the table of available Judges (top left) or remove the Judge from the table on top right. You MUST re-form the group by clicking "Form Group". Then click on Save.
It is recommended that all assignments for Period 1 be completed before moving on to period 2. Though the same groupings are usually used in all periods, judges can be grouped differently in each period. The lower left table (unassigned Judge Groups) contains available groups that have been formed for other periods. If a group is formed for a different period, and the member judges are unassigned in the current period, then the group will show up in the Unassigned Judge Groups table. These groups can be conveniently assigned to the current period by clicking on the Assign button next to that table. De-assigning a group takes it out of the period. The judges in the group are made available for re-grouping and their names will show up in the Unassigned table.
The JA can conveniently “Clone” all Judge Groups from a different period. This works only if the judges are available in the current period. The pairings of the judges can also be modified after they have been cloned.
Judge Assignments
This is where the JA assigns the teams to Judge Groups, for each period. GMS ensures that that all teams are interviewed for all awards at the end of the time periods.
The upper left table lists the available, unassigned Judge Groups for the period. The upper right table lists the assigned Judge Groups. The lower left table lists all the teams that have not yet been assigned to a Judge Group. The table on the lower right shows the teams that are currently assigned to a specific Judge Group. The goal is to have empty tables on the left.
The JA should start by selecting an unassigned Judge Group (the program will not allow the selection of more than one). The lower left table is updated to show teams that have not been assigned to a Judge Group for that particular award type. The Judge Advisor can assign a maximum of 15 teams in the lower right table. A number lower than 15 can also be assigned, depending on the number of judge groups and the total number of teams participating at that particular event. Then by clicking on the “Assign” button those teams will be automatically assigned to the selected Judge Group. The Judge Group is moved to the upper right table and the teams are removed from the lower tables. Note that each team must be assigned a Technical Judge Group and a Team Attribute Judge Group.
It would be best to finish assigning one period before moving on to the next one. Assignments from a previous period can also be cloned into this period to help speed up the process! When cloning assignments, one must have previously cloned Judge Groups for that period. It would be best to periodically save all assignments and changes made.
Tracking
Once all the periods have been defined, and all the awards have been scheduled, and judges have been grouped and tasked to decide the awards, the Judge Advisor is now ready to start the event. Click “Start Server” on the Configuration screen (unless the server was Auto Started). This will first allow configurations to be copied to the mobile devices.
Click on Tracking. The Judging Progress Tracking screen will appear: See image below.
The next step is to select the Current period. The period name will be displayed on the upper left. The Technical and Team Attribute Awards that will be decided during this period will also be displayed.
As can be seen in the image above, there are two charts: one for Technical and another for Team Attribute awards.
Judge Group Name buttons (on the left) will show the name of the judge group. Group names will be white until the latest configuration has been picked up by the associated mobile device. Clicking on the white GroupName make it appear that the Judge Group has downloaded the configuration and allows the following colors to be displayed. This is useful if a Judge Group does not have a mobile device.
The colors for the Judge Group are decided based on the number of teams that the group has left for interviewing. Green: On schedule, Yellow: Falling behind, Red: Way behind schedule. The average time available to interview each team is calculated by dividing the time left in the period by the number of teams left to be interviewed. If this time is less than 6 minutes, then the group’s color will be red. Interview times between 6 to 8 minutes will cause the group to be colored yellow, and more than 8 minutes will be green. The time thresholds can be configured via Tools à Configuration à Application.
Hovering over the Group Name displays the names of the group members (judge names).
There is a ‘Lock’ button next to each Group Name. It must be pressed after a Judge Group finishes interviewing all their teams. It indicates that the Judge Group is ready to enter their finalist selections.
The teams assigned to each Judge Group are listed following the button. Hovering over a team button will display a hint containing the team nickname and location. Clicking on a team displays the Team Details page with team information, including Match Observer comments. The team’s award history will also be displayed if so configured in the configuration screen.
While button unlocked, the team button is color coded as follows:
Ø Green: Interview complete
Ø Light Blue: Interview incomplete
Ø Yellow: Queuing soon (interview incomplete)
Ø Red: Queuing or on field (interview incomplete)
Ø Brown: Practice field or busy (interview incomplete)
A particular team’s color may not be the same here and on the PitMap screen (or on judge’s screens.) If a team has been interviewed by the Technical group but not by the Team Attributes group, they will be green only in the technical chart. A team may request a delay for the technical interview but may be ready for a Team Attributes interview. If the team is queuing and has not yet been interviewed, then they will be red in both charts.
The LockIn radio button must be pressed on the Judge Advisor’s Tracking screen after a Judge Group finishes interviewing all their teams. This allows finalists for that particular Judge Group to be entered. If all teams are not green (Interviewed) then a pop up will come up on the screen requesting confirmation. If the radio button is pressed again (unchecked), the row will revert back to its previous state.
All teams in the row will turn to white when the LockIn button has been pressed by the Judge Advisor in the presence of the Judge group. Once the Judge Group has decided on their finalists they can convey that information to the JA. The JA can then click on only those teams which will then be selected for re-interview (they turn green). The JA should decide how many teams should be selected for re-interview. Clicking on a selected team will de-select it and revert it to white.
While button locked, the team button is color coded as follows:
Ø White: Not selected for re-interview
Ø Green: Selected for re-interview
Re-interviews
Clicking on the Re-Interview button takes the period to the re-interview phase. A “re-interview phase” message is displayed next to the period name. The Re-interview button changes to Interview. Period number, period name and the awards being judged do not change. The screen will look something as follows:
The team assignments will change for all Judge Groups. Each Group gets to interview teams that they did not interview in the first round. The list of teams to interview should be smaller.
The new assignments are available for distribution to mobile devices. As in the Interview phase, the GroupName will turn to white until the updated configuration is picked up by the mobile device.
The remaining teams will be color coded as before. Each Judge Group’s selection for winners will appear in the group’s list in Dark Blue. They need not be re-interviewed by that group. They are present so that they can be re-chosen as finalists if desired by the group. The other teams are color coded as before.
At the end of the re-interview phase, the JA will lock the rows again. Each Judge group must select a finalist for discussion. They may stick with their original selections (in Dark Blue) or pick new teams. The new selections will be colored green. They will be transferred to the Discussion screen.
Clicking on the Interview button displays the screen as it was at the end of that phase. It may be useful for reviewing. Clicking on the ‘Re-interview’ button again displays the current screen. It is possible to display screens of previous periods for review, and also toggle between the Interview and Re-interview phases. It is also possible to click on future periods to review assignments and awards. Color coding will be disabled for all periods except the current one.
After re-interviews are complete, the JA can click on the Discussion button to enter the Discussion phase of the interview. This will display the Discussion screen.
At the end of the discussion for a period, the JA must select the next period and click the ‘Set Current Period’ button. This will start the Interview phase of the next period.
Award Discussion
The Discussion screen summarizes the finalists for each award. During discussion, awards can be switched back and forth between teams. Different scenarios can be tried out.
Clicking on an award displays the award description and rubrics. The screen below shows the discussion at the end of period 1. Only a few awards have been judged. Towards the end of the event, all awards would have been judged and ‘serious’ discussions can begin.
Each team button is colored as follows at the JA station:
- Purple: team has been nominated for the award. Only one team can be nominated. Clicking on a nominated team will un-nominate it.
- Red: the team has been nominated for a different award and should not be considered. However, clicking on the button will nominate the team for the award and the button will turn purple.
- Green: the team has been explicitly nominated for an award on the Interview tracking screen.
- Yellow: team has not been nominated, but has ratings worth considering.
Final winners can be printed from this screen.
© Rajaram Pejaver, July 2013