Mt. Olive
Pilot Description
The first pilot of GMS at Mt. Olive (Week 1) was a success. Overall, the feedback was positive.
Functionality
The following GMS roles were trialed:
· Robot Inspection: Lead Robot Inspector, Robot Inspector and Inspections Manager
· Judging: Judge, Judge Advisor, Judge Assistant
· Match Observing
· Match Queuing
· Game Announcing
Equipment
The following equipment was used:
· Tablets (28) for various roles
· Laptops (3) for the JA & LRI stations, and backup.
WiFi
The school’s WiFi system (a Cisco product) was used. Though it provided excellent coverage, it had some intrinsic routing problems. The problems were mitigated sufficiently for the duration of the event.
There was network congestion, but the bandwidth was adequate for GMS.
Feedback
Judges
· Many Judges felt that the GMS training was inadequate. Some suggested a 2 hour session.
· Some Judges felt that using the tablets during the interview slowed them down. They had the impression that data should be entered into the tablet during the interactions with the team. (Training could have corrected this misunderstanding.)
· One Judge Group wanted GMS to allow them to interview teams that were not assigned to them in that period.
· There was a suggestion to use 3 person teams for GMS. One person as tablet scribe, other two to ask questions. (Again, with training, this should not be necessary.)
· The Interview responses should be shared with the JA. Notes pictures/videos should be optionally shareable with the JA and other Judges. A Judge would explicitly have to designate a picture for sharing. (Great idea that can be supported in next season.)
· The Match Observers liked the app. There was a suggestion that a larger tablet (10”) may be needed for MOs.
Robot Inspectors
· Overall, the comments were positive. There were many suggestions for improvements.
· There was a claim that GMS was attacked using a WiFi Denial of Service tool. Besides being anti-social, any such attack would be equally effective on the robots on the playing field.
· There was an issue where Inspection Checklists were not being transmitted correctly in certain circumstances.
· There were suggestions for pre-configuring the tablet auto-rotation, timeout and font size options.
Queue Managers and Game Announcer
Again, the comments were quite positive. There were suggestions for improvements.
Lessons learned
JA and Judge training:
· We did not adequately 'train' the Judge Advisor for GMS. Dina used the system briefly to configure GMS guidelines but may not have had enough time to think about how they would be used.
· For Judges, we had allocated 30 mins at the end of the Judge's dinner meeting to introduce GMS and assign tablets to them. The feedback was that this was totally inadequate. I did not get to explain many of the features. Only 2 judges used the system effectively.
· For future events, we should work closely with the Judge Advisor to establish a strategy for using GMS. The JA should be trained on how GMS can help. They should take time to think about the Judging Guidelines that they wish to use. The JA should ‘adopt’ GMS as their tool.
· Participating Judges need to be trained at 3 levels:
o general Android tablet usage (display configuration, navigation, camera, typing, etc.)
o basic judging (how to use the guidelines, when to press the Completed button, notes)
o advanced topics (reviewing notes, Match Observer comments, voice transcription, etc)
This will take at least 90 mins and should be in an isolated environment (away from non-participating judges.) The JA should be involved. Judges should be allowed to take the assigned tablets home. The training should be conducted at least 3 days prior to the event so that they can get a 'feel' for the tablet and play with GMS. Training for Voice Transcription also takes some practice.
Training for RIs & other roles:
· Though no training was provided, the RIs, Queue Managers and the Game Announcer caught on quickly and used the system effectively.
For future events, we should schedule a phone conference prior to the day of the event. For RIs, this call should address the RI procedures, like: team state progressions, checklists, what kind of notes to take, team signatures, etc. The Lead RI and a few key RIs should download the PC application and play with it. For other roles, the call would cover role specific procedures and general GMS features.
Selection of GMS users:
· Many of the selected Judges were not familiar with tablets and could not use them effectively. GMS and tablet technology is not yet intuitive enough so that this set of Judges could be adequately trained in a few days. The event was not suitable as an introductory experience to a tablet. On the flip side, the Match Observer kids were a treat to watch. They had a great time and were the most effective users.
For future events, we should try to select users who are experienced with 'smart devices'.
WiFi:
· We were not able to start setting up the tablets till 3:00pm on Friday. At that time, we discovered that half the devices were not connecting properly to the school's WiFi system. We kept re-trying and by 6:00pm we had 22 tablets configured. The remaining tablets never connected. The Judge laptop connected around 11:00am on Saturday. Once connected, the WiFi system provided excellent coverage and adequate bandwidth.
For future events, the WiFi network should be set up and adequately tested well before the event, not on the day of the event.
A school (or venue) WiFi installation can be successfully used if reasonable safeguard are taken to ensure security and bandwidth availability.
Device Management:
· The tablets cycle through the following steps: initial charging, configuration, issuance, overnight re-charging, data wiping and return checking. Each user was given a charger and asked to re-charge the device overnight. All tablets were returned in good shape; one stylus was lost. However, we noted that tablets were getting passed around and exchanged, which leads to a loss of accountability.
For future events, tablets should have a color coded sticker on the outside with the name of the person to whom it has been issued. It would be nice to have one person responsible for issuing, maintaining and tracking the devices during the event.
Match Observers:
· We had 3 host high school volunteers that operated Match Observer tablets during the qualification matches. They provided useful information, though it may have been biased in favor of the host team !! Unfortunately, this information was not provided to the Judges.
For future events, the JA should review the selection of MOs and encourage Judges to use this information.
Schematic:
Springside-Chestnut Hill
Pilot Description
The pilot of GMS at Springside-Chestnut Hill (Week 3) went well. The Event & Volunteer coordinators were absolutely fantastic with their support.
Functionality
The following GMS roles were used:
· Robot Inspection: Lead Robot Inspector, Robot Inspector and Inspections Manager
· Judging (2 Judge teams used tablets)
· Match Observing (with official Judges)
· Match Queuing
WiFi
One WiFi access point was set up near the arena. This provided excellent coverage for all points in the pits and arena. The Judge’s room did not have a good signal. Though we had the necessary wiring and equipment in place, we did not install a AP there.
Feedback
Judges
· The Judge Match Observers liked the app. They felt that it was much better than the spreadsheet that was traditionally used.
· There was a suggestion that an index card describing GMS navigation short-cuts be placed on the inside cover of the tablet case. (Great idea. This shall be done.)
Robot Inspectors & Queue Managers
· Overall, the comments were positive. Most of them said “It’s about time…”. There were many suggestions for improvements.
Lessons learned:
JA involvement:
· It is becoming clear that a GMS pilot for judging is meaningless unless the Judge Advisor supports it. Half-hearted involvement will lead to negative impressions of GMS. We missed a great opportunity at SCH to pilot GMS judging with 'rookie' Judges; there were many who could have benefited from GMS. The two Judge teams that used GMS seem to have liked it.
Training for RIs & other roles:
· RI training was provided at the LRI desk at 3:00pm on Thursday (prior to the 4:00pm opening). Most RIs attended the training. A few arrived late and missed the training. However, they caught on quickly and used the system quite effectively. The LRIs were familiar with GMS.
For future events, we should schedule a phone conference prior to the day of the event. For RIs, this call should address the RI procedures, like: team state progressions, checklists, what kind of notes to take, team signatures, etc. This would be especially useful for ‘rookie’ RIs.
Laptops:
· I got delayed in traffic on Friday morning. Since I had the laptops with all the inspection data on it, there were some anxious moments at the RI desk.
For future events, we should leave the laptops secured at the event site. We should also remember that most other people commute to work on Fridays L
WiFi:
· Communications in the Judge’s room and the Volunteer’s break room were spotty due to a cinderblock wall, even though they were very close to the AP. Opening the door fixed the problems, but that was not a feasible solution.
· The school WiFi facility had been turned off for the event. This probably would have been useful and better than the tiny AP we used. We will be using the site’s existing WiFi in subsequent events.
For future events, the WiFi network should be tested with GMS. Simply checking the radio signal levels is not adequate. Besides the Judge’s room, providing good coverage in the Volunteer’s break room is vital. Many volunteers hang out in the break room and expect GMS to work there while they have lunch.
Lenape-Seneca
Summary
The pilot of GMS at Lenape-Seneca (Week 4) went well on the average. The system worked flawlessly for the Match Observers. Two Judges lost their notes on the tablet; the cause is being investigated. All tablets were returned in good shape.
The following GMS roles were trialed:
· Judging (4 Judge teams used tablets)
· Match Observing (4 MOs)
A report of the previous pilots are at:
Feedback:
Judges
· The Judge Match Observers liked the system. One of them has signed up to do it again at the next event (Bridgewater).
· The one Machine / Technical Judge who used GMS liked it. He had used it before at the Beta test at Ramp Riot.
· The other three Team Attribute Judges felt that they could not type notes into the tablet as easily as they could write on paper. They preferred GMS to take pictures and to compare team ratings. Losing their notes did not help either.
Lessons learned:
WiFi
· The school Guest WiFi system was used. This provided excellent coverage for all points in the pits, arena and even some portions of the parking lot outside. Even though it was wide open with no access control password, there were no problems with bandwidth or interference.
· Interestingly, the school WiFi was active at 5 GHz and was using channels that were conflicting directly with those used by the robots. No one seems to have checked and the robots didn’t mind at all.
For future events, the event site’s existing WiFi should be used whenever possible (after testing, of course.)
Judge Training
· GMS was introduced to the Judges during their Dinner meeting on the day before the event. I did most of the presentation while they were eating. A few Judges stayed back after the meeting for additional training. Many thanks to the ‘benevolent’ Judge Advisor who ‘enabled’ GMS and made all the difference.
For future events, this format worked well. However, we could ask the JA to schedule their training a couple of days before the event. In this case, I was totally available on the day before the event, which will not be the case in Bridgewater.
Judging
· Other than the data loss, GMS worked well. It was able to model the Judge assignments and re-assignments made by the Judge Advisor. I operated GMS as the ‘Judge Assistant’ and got several ideas for future improvements (for next year.)
For future events, I have added an ‘automatic archive/backup’ mechanism to save info on a 1 minute granularity.
Bridgewater-Raritan
Summary
Technically, the GMS pilot at Bridgewater-Raritan (Week 5) went well. Everything worked as desired, the school Guest WiFi system functioned reliably, and all tablets were returned in good shape.
The following GMS roles were active:
· Robot Inspectors
· Match Observers (2 Judge MOs)
· Queue Managers (5)
· Game Announcers (2 GAs)
Inspection Report:
Inspection at Bridgewater Raritan went quite well. We had 46 teams, including 2 teams that had not shown up at previous events and had never been inspected. On Friday, we had 6 RIs and 1 Inspection Manager. By Sunday, we had only 3 RIs, but all was OK.
On Friday, we got all teams Safety Checked in time for practice sessions. A few teams wanted to get Quals Checked so that they could get extra practice on the fill line, so we started doing them. By Friday night, 7 teams were not Quals checked: 2 teams had not shown up, 1 had software issues, and the others were rebuilding. (GMS has a now 'rewind' capability that allows me to review the event state at different times.)
Qual matches started on Sat at 9:30am. All teams were present. Both late arrivers missed their first match. We asked a couple of senior teams to help the newbies. After that, all was smooth sailing. Before Elimination, we re-checked the top 30 teams for bumper integrity and weight.
We chose to go full commando and did not have any paper backups to GMS. Inspection stickers were not used since Queuers could view inspection states on their tablets. However, we gave all teams a commemorative sticker for the event. The Queuers liked the inspection state information that GMS provided.
The referees also had GMS and posted several Reinspect requests, mostly for frame perimeter and bumper issues. We were able to address the issues and reset the team's inspection state before their next match. RIs were waiting for the team at their pit even before the robot returned from their matches.
Most of the RIs had used GMS earlier and did not need GMS training. We had 2 rookie RIs (Siri+Jon), who enthusiastically caught on pretty quickly.
GMS had a glitch at the end of Quarter Finals when FMS momentarily put up a Match Result in a format that I had not expected. (GMS reads FIRST web pages to get scores & rankings.) This slowed down GMS responses significantly, though they did not stop. I turned off result tracking and service was restored within a few mins. Inspection was not affected, but one Queuer switched their tablet to offline mode.
Feedback:
· The Match Observers spent some time previewing their system and were able to customize their checklist to their preferences. They both liked the system. One MO had trouble typing comments quickly enough into the tablet. I will add a Bluetooth keyboard to help in these situations.
· The pits were split between 2 areas. RIs mentioned that GMS helped reduce running back and forth between the rooms.
· Feedback has been uniformly positive. There were requests from Queuers for added features. Many of these were for features that were already present in GMS, which points to inadequate training or to a non-intuitive UI. I sincerely thank everyone that participated.
Lessons learned:
· You can drag a horse to the water but you can’t make it think.
Bethlehem 2014 MAR Champonships
Summary
The final pilot for GMS at the MAR Championships (Week 7) went well. Someone tried to attack GMS, but it survived and hung in there.
The following GMS roles were active:
· Robot Inspectors
· Judge (one brave soul)
· Queue Managers (5)
· Game Announcer (1)
Inspection Report:
Most of the inspectors were experienced with GMS. The two inspectors who were new to GMS had inspected at previous MAR events using paper. Both were quite impressed with the paperless system.
WiFi:
We set up our own WiFi network. There were 3 APs bridged together. WiFi coverage was seamless and extended well into the Dean’s List Judges room at the far end of the arena. This was the first trial of the planned WiFi configuration, and it went very well. See http://gms.pejaver.com/WiFi.pdf for details.
Hack Attack:
Someone broke into the WPA2 system and tried to hack into GMS between 11:00am to about 3:00pm on Friday. GMS was using SSL and it rejected all improper connection requests. Though the attack slowed the GMS server, no data was lost and no functionality was impaired. The attack may have gone unnoticed if it wasn’t for the warning messages that were displayed on tablets.
MAC Address filtering was enabled on Saturday. Either this move stopped the attacks or the attacker (probably a student) got tired of it. We were able to trace the attack to a Motorola tablet, but no further.
There were many Access Points active in the pits. Despite rule T22, one team even made it part of their theme (“Fly PanAm”). None of the APs caused any harm.
Judging:
The Judge Advisor declined to use GMS. However, a valiant FIRST staffer somehow enjoined him to allow Judges to use tablets. Unfortunately, judging had already started by this time and only one Judge actually used the system.
Other uses:
The Game Announcer used GMS throughout the event. Match Queuers also were pleased with the system.