Appendix: The Match Observer Role
Appendix: Match Observer Requirements
Appendix: Advantages of using GMS
Overview
Judge Match Observers have a hard task during FRC events. They need to:
- track robots during matches and record multiple salient points about their performance,
- share and merge notes with their MO partner(s),
- keep all notes secure and away from teams,
- reorganize notes to answer Judges' questions about a specific team's on-field performance,
- analyze team performances to compare a specific game aspect with other teams,
- do all the above from the scoring table, where the view is often blocked by field staff.
GMS is a cool tool that can help MOs with their tasks. GMS uses Android devices to:
- use customized checklists to help rapidly record pertinent robot characteristics during each match. MOs simply press buttons and can also record arbitrary comments about a team's performance,
- merge notes with MO partner(s) at any time using Android Beam or Bluetooth. This avoids the need for all MOs to go to the Judge's room to answer their questions,
- keep notes safe and private by encrypting them,
- automatically reorganize and summarize notes by teams. This eliminates having to shuffle paper while Judges wait for an answer,
- quickly analyze team performances based on the merged notes to promptly answer Judge's questions with firm numbers and stats.
- use mobile devices so that MOs can sit high in the stands and get a great view of the field.
MOs can use GMS even if no one else at the event is using it. In such situations, GMS will download the event schedule and team alliances from the web. After this, GMS can run "offline" and will not need on-field WiFi or Internet access. Notes can be shared with other MOs via Android Beam or Bluetooth. All comments are tagged by the originator’s authenticated name.
If GMS based judging is used at the event, then MO notes are automatically distributed to others who need it. Judges and Advisors can view the notes without disturbing the MOs, who may continue to observe ongoing matches.
While observing matches, MOs can record notes:
1. via buttons in "checklists" to rapidly note most common observations. For example: "Robot did not move during Autonomous Period", or "Team got excessive penalties" thereby indicating un-gracious behavior. Checklists make the MO’s task convenient and also consistent. They are easily customizable.
2. as free-form text comments. These comments can be typed in or transcribed from voice by speaking into a headset microphone. (The background crowd noise will typically be cancelled.) The text will be prefixed with the MO’s name whenever it is displayed.
MOs can highlight teams to remind themselves to look out for something special about the teams.
There are no limits on the number of MOs working simultaneously. Notes from all MOs can be merged into one set for analysis.
GMS Match Observers may be positioned anywhere where they have a clear view of the field. They just need to be comfortable since they will be watching most of the matches. They need not sit at the busy scoring table, nor be near a power plug. A reserved spot in the stands would be ideal.
Quick Start
Follow these steps to try out this app:
- Install GMS on an Android device. Find GMS in the Google Play store by searching for "GMS FRC". GMS will work best on devices that have a display size of 5" or more.
- Select "Use Web" and login using any name and password. Internet access is required during setup; it will not be required later. The password is used to protect MO notes.
- From the choice of events, select "Indiana Robotics Invitational" as a demo. The other events do not yet have schedules posted and are not suitable for demos. Match schedules are typically posted the day before the event.
- The "current" match and the teams on the field will be displayed. Click "Start Observing" and enter notes for a team. Swipe the screen sideways to enter notes for other teams on the field. Click the Android Back button to return to the Home screen.
- Click on the Next or Previous buttons to select other matches. Find a different match with the same team and enter more notes.
- To view a summary of the data collected, click on the team's number and then "Team Notes". You should see a summary of your free-form comments and the checklist buttons pressed. See the section on Data Analysis for more things to try. Contact Raj if you have any questions.
Screens and workflow
This section describes what the Match Observer sees and does. The following screens are from a 7 inch tablet.
For stand-alone mode, select a user name and password, select the "Match Observer" role, "Use Web", and login. WiFi access to the Internet will be needed. Select an event. If the event has published the Match Schedule, you will see the "Home page". To avoid needing WiFi access after this, logout and login again in "Offline" mode.
If GMS is being used for judging at the event, then use the assigned user name and select the "Online" radio button.
The Home page shows the current match and the teams that are on the field. The current judging period and the time left in that period are displayed. The “Time Offset” is the current delay in the current schedule. Additional team details are available by clicking on the team buttons. You can also see all comments made on a specific team. Descriptions of common features, like messaging, notes, searching, etc. are described in the GMS documentation.
MOs can review and edit comments on previous matches by clicking on the Previous or Next button. Press the large button at the bottom "Start Observing" to start entering comments. This will display the following screen in landscape mode.
Match Observers can view up to 4 teams simultaneously, though no more than 3 are recommended. The other teams come into view when the screen is swiped sideways. Select the number of teams displayed on a page by clicking Menu à Settings à Teams per Page.
Free form comments can be entered into the upper part of the team area. Text can be entered using the soft keyboard, Swype, or via voice transcription. A noise cancelling headset is recommended for voice transcription.
If a previously played match is being reviewed, comments made by other observers for that match will be displayed. These comments cannot be modified or deleted, but additional comments can be added as explanations.
The checklist can be scrolled vertically. Check off the button selections that apply. Buttons can be unchecked by long-pressing on them.
Exiting this screen automatically advances the current match number.
Team Page
Clicking on a team button displays the Team Page. This screen summarizes information about the selected team which may be useful while watching them. Additional award history is available by clicking on Details button. Private notes on each team can be entered and viewed on the Team Notes page. Click on the Match Notes button to see all available match comments made on a specific team.
Merging Notes
Notes are merged automatically in Online mode when GMS is being used for judging.
In other modes, data entered by MOs can be easily merged between devices, using one of the following options:
This is really easy and is the preferred way when both devices have Android Beam capability. Data is transferred from the sender to the receiver. The transfer must be confirmed by both parties. Both parties must be logged-in to GMS, and the GMS app must be in the foreground. At the end of the operation, the receiver will have merged data and notes from the sender.
All Beam operations follow the same sequence. There are two steps:
- For the first step, hold the two devices together (back to back) until the Beam tone is heard. Touch the screen only on the sending device. The Android Notification bar will show an uploading or downloading arrow icon. It may take a while for the beam to complete if Picture Notes are involved. The arrow icon will stop moving and a Beam complete tone may be heard when the transfer completes. If the wrong device is accidentally touched, just repeat this step and do it right this time.
- The data is now on the receiving device but it needs to be imported into GMS. There should be a Beam downloaded arrow icon in the Android Notification bar. For step 2, pull down the Notification bar. The Beam icon should say: "Beam Completed. Touch to view." Touch it and select "GMS Beam Importer" from the chooser that pops up. The sync process is now complete. All checklist buttons and notes (including pictures) are copied over. The receiver may be logged out if new schedule data is imported.
The Beam operation is useful in the following cases:
- Data is sent from the one device (Master) to the other during setup. The receiver becomes a clone of the sender.
- Data is sent from the one device to the other during syncing.
- Merged data can optionally be synced back so that both MOs have a copy of each others' notes.
As a safety feature, "Terse / Expert" mode should be enabled only on the Master device. Select Menu: Settings and check on Expert Mode. This skips importing checklists and match schedules from the sender and prevents the Master device from getting corrupted by a potentially messed up sender.
2) Syncing with Bluetooth
This function is similar to Syncing with Beam, only the steps are different. The sender and receiver must have Bluetooth capability. Both parties must be logged in into GMS. Click the "Share" icon (in the bar on top) to get to the Bluetooth Share screen. The receiver should first click on the Receive button to make that tablet visible to the sender. The receiving device will display its name and MAC address. The sender should then click Transmit. After a few seconds, the available receivers within range will be displayed on the transmitter. Wait for the target receiver to appear, verify the MAC addresses, and select it. Both parties may now need to verify Bluetooth pairing. Lastly, the receiver must click a button to accept data from the transmitter. The "Terse / Expert" setting is used as described earlier in Beam.
Data Analysis
After merging, the data is available on the receiving device for analysis. The merged data is organized by teams. A team can be selected by touching the team number on the pit map or on other screens. This displays the Team page.
The screens here show a demo match with simulated data. Note that ‘moe’ and ‘molly’ on the screens are the names of two fictitious MOs. All comments are imaginary and do not relate to any team living or dead. The data on the screens scroll vertically and horizontally.
In Match Notes, all free form comments are collected at the top. Each comment is annotated with the match number and the comment writer.
Each checklist question is summarized below the comments. It lists the counts for each choice for that team. The line below shows the total counts for all teams.
When properly constructed, the checklist can provide useful “quantitative” information. It can be used to answer questions like: “how effective was team XYZ in catching the ball?” It can provide an answer like:
In the 10 matches they played, team XYZ attempted to catch the ball in 8 and were successful in 6; In comparison, other teams in the field attempted to catch the ball 100 times and were successful 21 times.
This indicates that team XYZ has a 75% (6 out of 8) success rate with catching compared to the event average of 21%. The “10 matches” may not be relevant since they may not have had alliance partners capable of throwing a ball.
GMS can drill down even further and provide useful information. If asked the question: “which team was most effective in catching the ball?” touching a checklist summary on the tablet will display a list of all teams sorted by their effectiveness for that checklist question. Each column can be sorted in ascending or descending order by touching the column header.
Pit Inspecting information is summarized on the "Team List" page. Each team is listed as a row. There is a column for each Pit checklist. The column can be sorted by touching the column header.
Checklist
Checklists help MOs gather consistent data and record precise responses. They need to be worded appropriately to be effective. It helps to know how the gathered information will be used while wording the checklists. A trial run will help, and of course, checklists can be fine tuned after each event.
The questions in the checklist can be left unanswered, or a button can be selected. Long press to clear a radio button.
Checklists should not be changed during an event: all collected data may be lost. But if you absolutely need to, you can add (append) items at the end. You can also add options to existing items. Do NOT delete or modify any items, including blank lines.
Checklist Syntax
Each checklist item is specified in one line. Each line can be a section title, checklist item or an instruction.
- A section title is specified by enclosing the title between ‘<<<’ and ‘>>>’. This title will be displayed in bold on the mobile device.
- A checklist item is statement to which the MO should select a response. The responses are listed following a question mark (?). Each response is used to label a single choice radio button. If no options are specified following the question mark, the default options are ‘Yes’ and ‘No’.
- An Instruction should not contain with a question mark.
- Blank lines can be inserted as separators.
The following fragment of checklists generate the screen shown below.
<<<Autonomous>>>
Initial Activity? Dead, Moved, Mobility Zone
Did it detect the Hot Goal? No, Yes
Did it score in Auto? Attempted, Low Goal, High Goal
Extra credit? Attempted, Low Goal, High Goal
<<<TeleOp Period>>>
General efficiency? 1, 2, 3, 4, 5
Main Strategy? Defensive, Passer, Shooter
The checklists must be put into a file and copied to the GMS directory on the Android device. Their contents will be automatically transferred to other MO devices during setup using Beam or Bluetooth.
Lastly, there can be special section in the checklists file. The checklists listed after a header called "<<<Match>>>" are used as match checklists. These collect data on each match, independent of teams and alliances. This section must be last, following all other sections.
Installing Checklists
Modified Checklists are installed on the server in Online mode when GMS is being used for judging. Update the file called moRubrics.txt in the event folder.
In other modes, the customized checklists must be installed on the Master Android device. If it is not already there, create a directory called com.pejaver.gms on the Android. A simple way of creating this directory is to start GMS. The file must be called match_rubrics.txt. GMS will read the files only when logging in to an account. If you make changes to the files, you will have to logout and login again.
Copy the customized checklist files into this directory. One way is to connect the device to a PC using a USB cable. If you don't see the directory from Windows, restart the Android device (not a GMS bug!) Connect the Master device to the PC using a USB cable. You may need to install USB drivers for the device on the PC. You may also need to press a button to allow Android to open a USB connection.
You can also create or edit the files directly on the Master tablet.
Repeat this procedure for MO devices that do not have NFC Beam/Bluetooth, since the checklists are not copied automatically to them.
Security Aspects
All notes will be encrypted on the mobile devices using AES-128. Data sharing via Bluetooth is disabled.
Each 'free form' comment is tagged with the originating MO’s name and the match time. After the match, comments cannot be modified or deleted. MOs can add comments to clarify previous comments.
In general, MO comments are useful and could be shared with all other staff at the event. RIs need to know about robot performance on the field: like robots with loose bumpers, parts falling off, etc. However, some JAs may wish to restrict sharing of Match Observer comments with certain other roles. This can be accomplished by configuring: Tools è Configure è Application: “Block Access to MO Comments”. The roles that are checked will be blocked L. Access can be blocked to: LRI, Robot Inspectors & Inspection Manager, Judges & JAs, Others. Match Observers will not be blocked; they need to see each other’s comments so that they can expand on them.
Appendix: The Match Observer Role
Match Observing has traditionally been a judging function. Observers are typically assigned by the Judge Advisor out of the pool of available Judges. The information they collect is vital to the judging process. For example, they provide information on
· the reliability of a robot during matches,
· the practical effectiveness of an innovative mechanism on the robot,
· the gracious behavior of a team (or lack thereof)
· whether a robot is collecting excessive penalties on the field
Judges are usually busy interviewing teams in the pits and deliberating on awards. They do not get to watch many matches. Match Observers act as the eyes for the Judges and Judge Advisor. They are often asked to report on their observations during Judges’ deliberations for awards. For example, the Judges may want to award the Quality Award to a team whose robot was meticulously crafted. However, the award may be inappropriate if a Match Observer reports that the robot kept failing on the field. As another example, the Judges may have been impressed by a team’s description of an innovative method for a robot to jump over a fence. They need the Match Observer to confirm that the method actually worked on the field. Also, a team’s un-graceful or un-professional conduct on the field would be another important observation for all awards.
MOs usually spend most of their time watching matches. They may be called to the Judges’ room during deliberations to provide their observations. For example, they may be asked: how well did team XYZ’s feature work? Or: which robot was best at jumping over the fence? While MO observations are used to support an award assignment, they usually do not initiate the award discussion (where an MO comes in and says “I really like team XYZ’s shooter. Please consider them for the ABC award.”)
Observing six robots squirming on the field is not easy. A robot tends to do something notable just when the Match Observer is not watching it. Ideally, a Match Observer would work closely with a partner. They could share the load in some way, for example, one MO focuses on the 3 robots in the Red alliance while the other watches on the Blue teams. If available, even more eyes on the field would be useful. FIRST recommends 2 MOs for events with up to 68 teams. If more teams are present at an event, FIRST recommends 4 MOs. One extra MO will be useful to relieve active MOs while they take a break.
Appendix: Match Observer Requirements
A Match Observer’s tasks are significantly different from that of Pit Judges, and so are the required backgrounds and skills. Given the shortage of qualified Judges, the proposition being made here is that the MO role can be staffed from a different pool of volunteers.
The table below compares the requirements. Red rows indicate very different requirements, green rows indicate similar requirements and yellow indicates “similar but different”. The table has more red and yellow than green. Your opinions may vary. J
|
Judges |
Match Observers |
|
Judges interview teams. Judges should have people skills to interact with teams and elicit information from them. |
MOs do not interview teams. |
|
Judges do not watch many matches. They do not need to be intimately familiar with the game rules or the robot rules. In the past, Judges were mostly team mentors. Now increasingly, the Judges are representatives of sponsors, and they are not as familiar with FIRST, the game or the rules. |
MOs watch robots competing in matches. MOs must be interested in the game and must be familiar with the game rules. Their task is somewhat similar to that of RIing. They need to know when a robot is doing well and not doing well. Being familiar with the teams and team history may be useful. |
|
Judges should have a stature to grab a team’s attention while interviewing them. |
MOs can blend in amongst the crowd of screaming students in the stands. |
|
Judges should have the life experience to participate in deliberations and balance the distribution of awards to teams. They need to have negotiation skills. Some of them should have writing skills to be able to construct interesting award scripts. |
MOs do not usually participate on award deliberations because they are away from the Judge’s room most of the time. Even when they are in the room, they cannot meaningfully contribute to the deliberations because they may have missed some of the previous arguments. |
|
Judges are usually more mature and may not have the interest or the stamina to watch many matches. |
MOs are usually younger, technically inclined and genuinely interested in the game. They should be mobile enough to move about in the stands. |
|
Most Judges will not enjoy being MOs. |
Most MOs will not enjoy being Judges. |
|
Judges training should include the interview/ re-interview process, the intent of awards, team interaction, etc.
The training is different from MO’s. |
MO training should help them make comments that are consistent between matches and across teams. They should be told what type of comments and observations which will be useful while judging each award. The training is different from Judge’s. |
|
Both Judges and Match Observers need to be mature, impartial and objective. |
Both Judges and Match Observers need to be mature, impartial and objective. |
|
Judge’s notes and discussions are considered sensitive and should be protected. |
MO’s observations are generally based on matches that are viewed by many others. However, they may contain sensitive notes and should be protected. |
Appendix: Advantages of using GMS
GMS was designed based on actual field experiences. Using it has several advantages:
- The MO Checklist helps record consistent data on all teams.
- The MO does not have to shuffle papers or re-organize their notes on their laptops in order to report. The organized reports are available constantly and automatically.
- MOs will not miss matches while responding to Judge’s questions. With the older system, MO de-briefing requires all MOs to be present to answer questions, since there is no way to merge all their observations. GMS merges all comments and presents it in an organized way.
- MOs need not visit the Judge’s rooms and see the sensitive information that may be on the walls.
- When GMS is being used for judging, Judges get instant information, they do not have to wait for the MO to show up. All the comments and quant data are available to Judges at their fingertips. This reduces the need to drag the MO to the Judge’s room.