Proceedings Pilot DNA Activity Level

News item | 10-03-2021 | 09:10

The pilot-assessors agreed on the quality of these reports and that the assessment could be done in a uniform manner. Based on the feedback of the assessors, the Standards were adjusted for an optimal assessment. The NRGD is hopeful that this proceedings document together with the Standards published by the NRGD will aid the community to create a standardized and stable field of expertise worldwide.

Attendees

Bart Blankers, Bas Kokshoorn, Duncan Taylor, Jonathan Whitaker, Ronald van Vuure, Sue Pope, Tacha Hicks, Nick Laan, Karien van den Doel, Nathalie Raats.

Introduction

In the last decade DNA activity level propositions have been addressed in the Dutch court of law more and more often. This was reason for the Netherlands Register for Court Experts (NRGD) to investigate if this field of expertise could be standardized and added to the register. Since then, multiple expert meetings, a SWOT analysis[1] and the creation of a white paper have resulted in the Standards for DNA Activity Level.

On the first of September 2020 the Standards for the field of DNA Activity Level[2] were published by the NRGD. The NRGD created these Standards together with DNA Activity Level experts who are currently working in this field. The Standards are believed to be the minimum requirements for experts to adhere to before they can call themselves an expert before the Dutch court of law. The NRGD assures the expertise of these experts by assessing them on their competencies based on their own written reports by a group of peer assessors. Reporting on DNA findings given activity level propositions is rather new. Subsequently it was unknown if and how these types of reports could and should be assessed. Therefore the NRGD organized a pilot assessment were a group of seven experienced experts and assessors gave there feedback on a number of reports on evaluations of findings given activity level propositions (further referred to as ‘Activity level reports’).

Method

The NRGD obtained nine Activity Level reports from five different institutes who are currently reporting on findings given activity level propositions. Each report was double anonymized, meaning that all names and private information concerning suspect, victim/complainant was removed but also all public information concerning the expert and the institute. Accordingly, the experts and assessors could assess the reports in the most objective manner possible. Each assessor received four reports for which they were asked to fill in a so-called Advisory. Evaluation Form (AEF). These forms are typically used by NRGD assessors. Prior to the meeting, the assessors send their feedback to the NRGD for analysis.

The AEF has eight different subjects the applicant is assessed on:

• professional development;
• case strategy;
• investigative material and data;
• investigative methods and interpretation;
• legal context;
• reporting;
• professional attitude;
• concluding and/or additional comments.

Of these eight subjects 1. and 5. were not evaluated due to the lack of information on both the case and the expert, which fall outside the scope of this pilot.

The goal of the meeting was to answer the following questions:

For each subject the assessors could fill in their comments in a table divided as follows:

	Strong points	Weak points
Main
Minor

The goal of the meeting was to answer the following questions:

• Do the Activity Level reports meet the requirements of the Standards?
• Can Activity Level reports be assessed in a uniform manner?
• Can Activity Level reports be assessed based on the current Standards and assessment form?

During the meeting each report was summarized by the chair and then discussed by the assessors. First, one assessor would state their findings of their evaluation followed by commentary of the other assessors. After each report was discussed, there was a general discussion concerning the assessment procedure including the Standards and AEF.

Results

In the pre-analysis done on the AEF’s by the NRGD, a number of issues were discovered which required attention beforehand. First of all, no prosecutor fallacies[1] were found by the assessors in the reports, which is a very important and positive point. In contrast, many assessors commented on almost every report either that the request was not clear, the issue was not clear or that the propositions were not clear. Nevertheless, the commissioning party usually determines these points and cannot always be altered/improved by the experts themselves.

A number of comments from the assessors were not addressed by the NRGD as they fall outside the scope of this pilot. These were comments concerning:

• the qualifications of the expert; normally the assessors receive additional information concerning the qualifications of the expert;
• lack of information from commissioning parties;
• no references to the framework documents; they were removed by the NRGD as they mainly referenced to the expert’s institute;
• issues related to law and jurisdiction;
• prior DNA analysis and sampling strategy, not performed by the experts themselves; sometimes experts cannot influence this part of the criminal investigation anymore.

The points mentioned above are normally part of the assessment.

Instead of analyzing the reports in these proceedings, the subjects of discussion will be discussed below. These subjects are:

• the propositions;
• verbal vs numerical likelihood ratio;
• assumptions;
• use and assessment of Bayesian Network;
• reedback on prior sampling strategies and DNA analysis;
• reference to and evaluation of the framework;
• changes to the Standards;
• changes to the AEF.

The propositions

Eight out of the nine reports contained propositions on activity level. Only one report did not contain specific propositions as it was written as an Activity Level statement instead of an Activity Level evaluative report.

In most reports the propositions were stated at the beginning and at the end of the report which was preferred by the assessors. Only few reports stated the propositions only at the end, which was inconvenient for the assessors and was regarded as a minor weak point. Furthermore, in some reports the propositions stated at the end were formulated differently from those stated in the beginning of the report. This change did not influence the conclusion in the report, however it was considered as a minor weak point.

For one of the reports it appeared that the experts created the investigative question themselves based on the information they received, however this was not directly mentioned in the report and it was unclear why the expert did this.

More than once, it was unclear who created the propositions based on the case information; did the experts create them themselves based on the case information or did the commissioning party give the propositions to them? Ideally, the experts formulate the propositions themselves based on the given case information and generally accepted criteria[2].

One proposition contained a double hypothesis, consequently it was unclear given which proposition the likelihood ratio was reported.

Based on these findings the assessment committee agreed that it should be explicitly stated if the propositions are formulated by the experts themselves as a proxy or by the commissioning party.

Furthermore, the committee advised to add a point in the assessment form: indication on what are the criteria that the propositions should adhere to.

Verbal vs numerical likelihood ratio

For various reports there was a discussion concerning the verbal and numerical likelihood ratios and the assigned probabilities that were reported. First, the expert should ideally start with a numerical LR or probability and provide an explanation by using verbal equivalent and not the other way around. In some reports, activities were assigned verbal probabilities without the corresponding numerical value. If numerical values are used then they should be included in the report. Furthermore, in some cases where a specific numerical value was mentioned, it was unclear how this was reached or how the final verbal term correlated to that specific numerical value.

In some reports it was very clear which numbers correlated to which verbal values based on a standard system. But even in these reports the experts used terms like very small changes of transfer and small probabilities which do not directly correspond to a numerical term. Normally in the court only verbal terms suffice for the understanding of the report, but for a thorough assessment of the expert, these numerical values should be included with the report, or in additional case file documentation provided.

Assumptions

In Activity Level reports, the expert needs to make assumptions concerning the evidence in question. Each report we received was very transparent with the assumptions that were made. However, it was not always clear on which information the assumptions were based and why the assumptions were made. In Dutch cases, if the DNA source is not contested then it is usual to assume the match is true. This assumption was confusing for the foreign assessors as no details (numerical values) were mentioned concerning the DNA analysis or that the match was not contested. However, the assessors considered these as minor weak points because this information is normally available to all involved parties in the criminal case. Nevertheless, it is preferred that this information is added to the application.

Use and assessment of Bayesian Network

Almost every expert used a Bayesian Network for the analysis and interpretation of the evidence. However, it was not always mentioned when a Bayesian Network was used. Furthermore, only two reports actually contained the Bayesian Network in the appendix. The assessors considered these reports very useful for the assessment as it gave a lot of insight on how the Bayesian Network was constructed and how this led to the results reported. In contrast, all this additional information is very complex and can create more confusion for court. The assessors conclude that the inclusion of the Bayesian Network when used is not necessary for the court but the use of the Bayesian Network should be disclosed in the methodology section of the report for transparence purposes. In addition, the Bayesian Network is necessary for the assessment of the expert and should be added to the application.

Consequently, we come to the question; should an Activity Level expert be able to construct and use a Bayesian Network? In other words, can an expert be registered for DNA Activity Level if he cannot construct Bayesian Networks? The assessors agreed that the competency of constructing a Bayesian Network should not be a requisite for registration, but if an expert does use Bayesian Networks, then the expert should be assessed on it. The assessors advise to add this point to the Standards:
If the applicant uses Bayesian Networks in their Activity Level interpretations then they should include at least one report in the application incorporating a Bayesian Network. All additional information concerning the Bayesian Network should be included in the application.

Finally, the question arose to which extent they needed to assess the Bayesian Network used. The assessors agreed that they should not assess the Bayesian Network itself, but how the expert applied the Bayesian Network (or any other statistical model) to their case. Therefore, the assessors should check the relevance and application of the Bayesian Network to the case, by assessing if the Bayesian Network is structurally sound (logically correct) and if the construction addresses the propositions, assumptions and other case information correctly. If no Bayesian Networks are used, then it should be clear how the conclusion in the report was reached from the report or additional case information provided. Aspects of conditionally dependent findings should be addressed.

Feedback on prior sampling strategies and DNA analysis

For two reports the assessors had questions concerning the sampling strategy of the evidence in question. For these reports it appeared that the Activity Level expert did not perform the sampling strategy themselves and could therefore not do additional physical investigations. Having more information concerning the evidence could possibly influence the outcome of the Activity Level analysis significantly. However, this effect was not mentioned in this report which was considered to be a major weak point. In conclusion, if a different or an additional sampling of the evidence could essentially change the final result of the Activity Level analysis, then it should be mentioned in the report.

Reference to and evaluation of the framework

For this pilot it was very difficult to determine if the framework for the reports was sufficient because most of the references were deleted by the NRGD. In general, if papers are used, references should be given and ideally, the data should be mentioned, giving the assessors the ability to check if the reference was applied correctly. However, the assessors should not go so far as to review the article itself. Furthermore, the scientific basis for Activity Level reporting is growing every day and therefore impossible for an assessor to keep up with.

The applicant should be assessed on the ability to correctly apply the framework to the investigation. The framework itself should not be under discussion for the assessment. The assessors agree that discussion concerning framework should be reserved for the court of law.

The question was raised if it should be accepted that experts use their own experience instead of using scientific references. Although scientific references are preferred, the applicants should be assessed on how they assign their probabilities and not on the ability to reference scientific articles.

Changes to the Standards

During the pilot only a few changes to the Standards were advised by the assessors, some of which have been mentioned above.

For both the experts and the assessors, the Standards should be more prescriptive on what must be seen in a report such that the assessors are able to do a proper assessment. The Standards could refer to the ISO norm for forensic reporting[3] and additionally mention the following topics specifically for Activity level reporting:

• propositions, including the criteria for propositions;
• pre-assessment;
• context information;
• assumptions and limitations;
• methodology;
• assignment of probabilities given the activities;
• bayesian Network if used;
• meaning of the LR.

As mentioned above, it was advised to add this point to the Standards:
If the applicant uses Bayesian Networks in their Activity Level investigations then they should include at least one report in the application incorporating a Bayesian Network. All additional information concerning the Bayesian Network should be included in the application.

Changes to the AEF

The assessors had trouble to assess the reports based on the current AEF of the NRGD. They advised to change this form incorporating the specific aspects of Activity Level reporting. For example, point 3 (Investigative material and data) from the AEF is logical to assess for Source Level DNA analysis. Point 3 focusses on how the DNA material was obtained and analyzed. In contrast, for Activity Level analysis, DNA analysis has already been done, often by another expert. Accordingly, the applicant cannot be assessed based on the work of someone else. In comparison, point 3 should focus on the methodology used for the Activity Level analysis.

Finally, it was unclear to many assessors, to what extend they were required to assess the report. As mentioned before, assessors should not review scientific articles but only check if they have been applied correctly. Similarly for the evaluation of the Bayesian Network, the assessors should not assess the methodology of the Bayesian Network itself, they should check the relevance and application of the Bayesian Network to the case, by assessing if the Bayesian Network is structurally sound (logically correct) and if the construction addresses the propositions, assumptions and other case information correctly. These limitations should be made clear for the assessors in the AEF.

Conclusion

The goal of the NRGD in standardizing DNA Activity Level is to support the court of law in the best possible way. To do so, the NRGD organized this pilot to determine if reports with Activity Level analysis could be assessed. For this particular meeting the NRGD set a number of goals which were all reached. The Activity reports met the requirements set in the Standards. The assessment outcome for almost each report was uniform as assessors gave similar feedback concerning each report. Feedback was discussed among the assessors where feedback was contradictory, which either resulted in agreement or the disagreement did not influence the outcome of the assessment. The Activity reports can be assessed based on the Standards and the assessment form. However, these documents did require a few revisions. Finally, the NRGD is hopeful that this proceedings document together with the Standards published by the NRGD will aid the community to create a standardized and stable field of expertise worldwide.

-------------

[1] Strengths, Weaknesses, Oppertunities and Treats analysis

[2] The NRGD calls this field DNA Activity Level, but in fact the experts will not address the activity itself but only the evidence. They will assess their results given the alleged activities and report the value of the DNA in the context of the case. The expert will not provide an assessment on how likely the proposition is. Determinnig the likelihood of a proposition i the domain of the court, as one needs to combine all the information of the case in order to make such a statement.

[3] When experts give an opinion on the (alleged) facts or propositions given the results, this is called a prosecutor’s fallacy. The name of the fallacy is a little unfair, as experts, journalists, lawyers and prosecutors commit this fallacy, which is also known as the transposed conditional.

[4] Propositions should not be set by the court as propositions have to follow scientific criteria. A basic criterion for propositions is that they should be formulated in such a way that it is reasonable for the scientist to address a question of the form - ‘what is the probability of the observations given this proposition and the framework of circumstances’? Thus, to formulate propositions one needs to know the (expected) results. However, prosecution, police and defense are in the best position of course to give the information on the alleged facts, and based on this the scientist will formulate the propositions that allow the assessment of their results.

[5] ISO/CD 21043-5