Online proctoring: report March 27

Introduction 

Now all education is taking place online, the question of how to arrange remote testing naturally arises. Currently, online proctoring seems to be one of the ways to enable large-scale summative remote testing. Alternative forms of testing and examining are also important and relevant. As chairman of the webinar, Silvester Draaijer, illustrated this statement: “At the VU, our first message is: Try to find a different form of testing than the classic(al) written final. If there is really no other way, the classic written final may qualify for online proctoring."

We won't discuss the alternative forms of assessment in this session. In this session, we mainly focus on the online proctoring part and the question: “How realistic is it to arrive at a substantial volume of examining with online proctoring during the coming weeks?”

That is why we have invited a number of institutions, including VU Amsterdam, TU Eindhoven, Wageningen Univeristy&Research, and Nyenrode Business University- which are currently conducting or upscaling pilots - to share with us which global approach they are following. In doing so, we'll address questions about fraud-proofing, risks, stability, and size. We also look at what has to be arranged, what the costs are, and how you can arrange it legally.

VU Amsterdam 

The VU has started using ProctorExam. It is used for groups no larger than 130 students because logging on is a precise business. Students need to know exactly when to log on and which buttons to press. Two employees of the back office who normally supervise digital tests are available for help. They try to answer all questions that come in from students with login problems. Every 15 minutes 50 students start because that is about the maximum number of students they can support at a time. Larger tests must, therefore, be divided into parts.

VU Amsterdam also uses the “Record and Review” functionality. This means that ProctorExam employees check the recordings after the exam for suspicious behaviour. The regular proctors then look at the recordings before official reports are made. The tests are administered via TestVision. The proctor environment is separate from this testing environment. Therefore, this requires a mirrored student administration and mirrored test conditions. Due to the COVID-19 crisis, VU Amsterdam is now working hard to see how they can improve the scalability, user-friendliness, and integration with their testing application.

Eindhoven University of Technology

Eindhoven University of Technology (TU/e) has rapidly implemented Proctorio, and even managed to create an API link with Cirrus and ANS. With a simple check mark, those test applications can indicate that a test is an online proctored test. Unlike ProctorExam, Proctorio automatically detects fraud, based on an intelligent system. Using ANS-Delft security settings in Proctorio can be made per exam . In Cirrus, this must be done at the institution level (all exams must use the same settings).

TU/e has conducted various tests in which students have deliberately committed fraud to see how well the system is able to detect fraud. About 90% of the fraud cases resulted in the so-called "flag". The screen, video, internet traffic, keyboard strokes and microphone are recorded, and based on an analysis, potential fraud is "flagged" in the timeline; afterwards you can view the recordings. According to TU/e you will still have to check this and separate suspicious behaviour from ordinary behavior. For example, drinking from a bottle of water can be detected as suspicious.

Just under 5% of the students ran into technical issues during the pilot phase. It is especially important to properly instruct students, to have a proctored practice test and of course to be ahead of technical issues. That is why they also offer an opportunity for students to practice with proctoring beforehand. Also, they continuously monitor within the organisation how much trust there is in the system. All levels are involved, including the University Council, programme directors, the educational board, deans and chairmen of examination boards. The Executive Board has yet to declare approval (this has happened [edit]).

Wageningen University&Research

Wageningen University&Research (WUR) has a lot of experience with online proctoring. For the past five years, they have been testing PSI within a full master's programme. That means small numbers for WUR, which makes it technically and process-wise easy to handle. The group size has so far been a maximum of 85 candidates, but they can't escape scaling up. This will soon be the case, be when a new examination period starts at the beginning of May.

WUR does not engage in live proctoring. A team is deployed to flag videos afterwards, checking for fraudulent behavior, using the so-called "Record and Review" function.

Experience shows that when starting a proctored test, the number of questions from students is limited. However, those pilots were with smaller groups. WUR, therefore, prepares a support desk for the upcoming exam weeks. As for the students... From next week onwards they can take the practice exam as often as they want.

Nyenrode Business University

In addition to TU/e, Nyenrode Business University also uses Proctorio. They do this together with Cirrus. There are no problems in terms of scalability, both on the Proctorio and Cirrus side. They also exclusively proctor afterwards, and based on flags that are automatically placed by the system.

Proctorio has eight different adjustable parameters that determine the sensitivity with which the software detects fraudulent behaviour. Nyenrode cannot yet make optimal use of this, in the sense that for the time being only 1 profile can be chosen for all tests. The integration with Cirrus still had to be made, which has already limited the functionalities slightly. Their big advantage is that they already benefit from two linked systems and do not have to worry about scalability.
The recordings made are stored until the evaluation is complete, and then data is deleted after three months.

And more information

A lot of information was shared during the session, including information from a number of other institutions. This information and more details per case can be found in this report (Dutch), courtesy of Nyenrode Business University.

Conclusion 

This session made clear there is still a lot of work to be done at the various universities and universities of applied sciences to organise online exams. An initial preliminary conclusion seems to be that institutions working with Cirrus benefit from a direct link with Proctorio. ANS Delft also works very well in combination with Proctorio. Proctorio seems to be scalable for the time being. Links with ProctorExam may still become available. Of course, this will have to be put into practice in the coming weeks. In addition, institutions that test with Remindotest or TestVision are also currently investigating how quickly these software systems could be properly linked with proctoring software. It, therefore, makes sense to keep tabs on developments in online proctoring in the coming weeks. A short survey among the participants of this session shows a large majority would like to be updated about the developments again in a week's time. So we will take care of that!

The core team of the SIG Digital Assessment consists of Sharon Klinkenberg (University of Amsterdam, chair), Silvester Draaijer (VU Amsterdam), Eky Fioole (Avans University of Applied Sciences), Vincent Kalis (Tilburg University), Michel Kao (HAN University of Applied Sciences), Ludo van Meeuwen (TU Eindhoven), and Annette Peet (contact person from SURF)