While the process is not completely infallible, training and structured processes can ensure that assessors make the most accurate ratings possible.

Posted by  Alissa Parr, Ph.D.

This summer, I had the opportunity to go to the US Open, which is one of the four major tennis championships occurring every year. I was completely impressed by the skills of each of the tennis players, and the amount of dedication that is required to be able to compete at that level. As I was trying to keep pace with the rallies between the players, I also became pretty impressed by the level of accuracy from the line umpires. The players’ serves were averaging between 100-120 mph, and, went as high as 145 mph. At that rate, the ball seems to be a blurred line to me and I would find it difficult to determine whether the balls were in or out. However, without fail, the line umpires would make calls that very rarely were incorrect.

The role of tennis line umpires parallels those of hiring managers and assessors in selection systems. If a selection system includes a tool that is more interactive in nature (e.g., structured interview, role-play, in-basket exercises), assessors must make ratings or judgments on how the candidate performed in that exercise on a set of competencies. In order to improve the accuracy and reliability of the ratings assessors make, they must participate in training to better understand the structure of the exercise/tool, the competencies being assessed, and the process for making ratings. This training tends to be standardized so that each assessor receives the same information and will in turn make ratings similar to other assessors. In essence, this is how we ensure that all assessors are calibrated. Similar to assessors, line umpires must undergo training in order to be certified and also must accrue experience in order to judge bigger tournaments. Overall, this helps to improve the accuracy and reliability of their calls.

One of the other interesting things about tennis, and some other sports, is the advent of a technology that allows for a review of plays. Hawk-eye is the computer system that visually tracks the ball and defines the most likely path as the ball is moving. According to the makers of this technology, the system performs with an average error of 3.6 mm, which is basically equivalent to the fuzz on a tennis ball. Tennis players have the ability to challenge questionable calls. However, interestingly enough, about 75% of the original calls by umpires are upheld according to some statistics provided by the US Open this year. Even though this shows that line umpires are not perfect, the training they have and consistent process in place facilitate a very high rate of success. However, it’s not only the training that facilitates the success of assessors and line umpires. It begins with the exercise and tool itself. When the exercise and tool is structured and standardized, guidelines can be in place to facilitate more accurate ratings of performance. In the case of interviews, structured interviews have much greater validity than unstructured interviews. By keeping to a consistent process, interviewers/assessors can refer the scoring guidelines to make ratings for the standard process. Additionally, using a structured process can reduce the possibility of bias in making ratings. Similar to this process, tennis has a standard set of rules that line umpires must know in order to make their rulings. They need to know what defines whether a ball is in or out of play and when serves should be retaken. These consistent guidelines facilitate the ability to make accurate calls.

While assessors don’t exactly have this type of technology in place, there are times when assessment scores are questioned by candidates or other individuals. In turn, there are steps that assessors can take to show support for ratings. For example, during an interview, it’s important to take notes. Additionally, role-plays can be audio or video recorded. All of these pieces of evidence can provide a reference to review and support the original ratings provided. While the process is not completely infallible, training and structured processes can ensure that assessors make the most accurate ratings possible.