Evaluation procedure

Goals

The aims of the evaluation metrics are twofold:

• measure the degree of accuracy of the endocardial surface. This will be done through global and local measures of similarity with the reference contours;
  
  
• measure the degree of accuracy of the derived clinical indices.

Distance error metrics

To measure the degree of accuracy of the endocardial border, three standard metrics will be used:

• Modified Dice similarity index: the modified Dice similarity index, D*, is computed as a measure of overlap between the surface volume (V) from automatic method and the reference surface volume (V_ref), giving a measurement value between 0 (full overlap) and 1 (no overlap).
  
  
  This value will be computed for both End Diastolic D*_ED and End systolic D*_ES instances.
  
  
• Mean surface distance: the mean surface distance, d_mean, between the surface (S) from automatic method and the reference surface (S_ref) defined as:
  
  where d(S,S_ref) is the mean of distances between every surface voxel in S and the closest surface voxel in S_ref, while d(S_ref,S) is computed in a similar way. This value will be computed for both End Diastolic d_mean,ED and End systolic d_mean,ES instances.
  
  
• Hausdorff surface distance: the Hausdroff distance, d_H, measures the local maximum distance between the two surfaces S and S_ref. This value will be computed for both End Diastolic d_H,ED and End systolic d_H,ES instances.

Clinical indice metrics

To measure the ability of the algorithms in extracting relevant clinical indices, modified correlation (corr*=1-corr), bias and standard deviation (std) measurements will be computed from End Diastolic Volumes (EDV), End Systolic Volumes (ESV) and Ejection Fraction (EF) measurements. The following notation will be used

• Modified correlation computed from EDV measurements: EDV_corr*
  
  
• Bias computed from EDV measurements: EDV_bias
  
  
• Standard deviaton computed from EDV measurements: EDV_std
  
  
  

Ranking strategy

The challengers scores will be ranked according to the following measurements:

• Global distance errors measure
  
• Global clinical indices measure
  
• Global error measure
  

It has to be noticed that each individual measure (for example d_H,ED) will be normalized by the maximum value of the corresponding measure among the participants. By doing so, each measure will be defined between 0 (best score obtained if the result perfectly fits the reference mesh) and 1 (worst case among the participants).

The ranking will be performed on the global error measure obtained by each participant.

MIDAS mesh distance visualization

Each participant will have the possibility to visualize the distance between his uploaded result mesh and the corresponding reference one thanks to WebGL technique. A typical visualization that will be provided through the dedicated MIDAS website is presented bellow (the colormap provides a direct information on distance between two meshes).