Assessing the assessment: the NRGD and Cito are working on a transparent evaluation process
Since its inception, the NRGD – independent and established by law – has been developing the register of court experts. The NRGD creates and manages that register for the criminal justice system. The register is steadily expanding, adding ever more fields of expertise, from DNA to pathology and from handwriting to digital forensics. The NRGD works continuously to further improve the quality of the processes involved in such an assessment. A key aspect of this is to make transparent how the assessment is conducted and the path by which a final qualitative evaluation is reached.
NRGD Director Michel Smithuis: ‘The NRGD evaluates professionals. Our evaluation may have major consequences for the individual expert – both professional and personal. When your work is judged to be qualitatively inadequate, it can be a big blow. We’re very aware of that. We therefore work with great care. That also means that we want to make the assessment process as transparent as possible. We go through this process again and again, because what was good five years ago is not necessarily still good now, or else it can be done even better.’
A good example is the advisory assessment form from 2019, which sets out the specific points on which the work of experts is assessed. Policy officer Els De Jong (NRGD): ‘The core of our assessment is the qualitative evaluation of the reports that an expert submits together with the application. These reports are an expert’s input in a criminal case. The asssessors examine all the reports in depth and comprehensively, with a keen eye for anything that stands out in the individual reports. A standard model or checklist is not much use to us.’
How do you make such a tailor-made process transparent? The NRGD looked for new ways to make the qualitative assessment transparent and comparable, and sought advice from Arnhem-based Cito, the educational assessment organisation known for developing and administering tests and central exams for schools. Kimberley Lek, PhD, educational researcher at Citolab, and Willem-Jan van Gendt, Business Consultant at Cito Zakelijk, were involved in supporting the NRGD in this project.
Willem-Jan: ‘The NRGD is actively engaged in quality improvement. Together with the NRGD, Cito has looked at the procedures and protocols, and assessed them against recent academic literature.’ We’ve examined the whole assessment process from the application stage. ‘This cooperation resulted, amongst other things, in a supervision procedure to arrive at an assessment tool that further enhances the transparency of the assessment. Training courses and hands-on sessions were used to design this tool.’
Weighting
The research focuses on transparency and not on individual experts, but these experts are following the progress of the project with interest. The aim is to involve them in the research. The Netherlands Institute of Forensic Psychiatry and Psychology (NIFP) has been informed about the initial results and responded positively. In addition, Cito provided a training course on assessment methodology for assessors. The training included topics such as reliability and pitfalls, which attracted a great deal of interest. The NRGD and Cito are considering the possibility of repeating the training course, or of exploring the subject in greater depth in a peer-to-peer review and reflection session with assessors.