Benchmark results of RNA aligners (May, 2008)
|Method||Accuracy of alignment||Accuracy of predicted secondary structure (MCC)|
|RNAalifold||(internal)||CPU time (s)|
|Structural alignment methods|
|X-INS-i-scarnapair (Uses MXSCARNA)||McCaskill||0.880||0.769||0.736||0.708||0.731||-||340|
|X-INS-i-foldalignglobalpair (Uses global alignments by FOLDALIGN)||McCaskill||0.884||0.783||0.738||0.705||0.734||-||9,700|
|X-INS-i-foldalignlocalpair (Uses local alignments by FOLDALIGN)||McCaskill||0.859||0.768||0.730||0.706||0.729||-||14,000|
|X-INS-i-larapair (Uses LaRA 1.3.2a)||McCaskill||0.869||0.783||0.738||0.712||0.741||-||(2,300)|
|FFT-NS-2||0.832||0.674||0.678||0.663||0.669||1.2||←Default of mafft|
|ClustalW version 2 (Iteration=tree)||0.798||0.641||0.649||0.641||0.652||-||22|
|ClustalW version 2 (Default)||0.795||0.646||0.640||0.641||0.648||-||2.6|
A similar observation is made with the comparison between LaRA and X-INS-i-larapair. These two methods use the same pairwise structural alignment algorithm, LaRA.
The combination of X-INS-i and FOLDALIGN is slightly more accurate than the combination of X-INS-i and SCARNA. However, the latter is much faster than the former and the difference in the accuracy is quite small. From the practical viewpoint, we selected SCARNA as the default for X-INS-i.
The difference between global and local options of FOLDALIGN is probably because the dataset is globally alignable.
Several alignment methods (RNA sampler, Murlet, MXSCARNA and MASTR) return secondary structures internally predicted, while other methods (MAFFT, LaRA and StrAl) do not. At present, the advantage of internal prediction is not clear, since the accuracies of predictions by external methods (Pfold, McCaskill-MEA and RNAalifold) are comparable to or rather higher than those of internal predictions (shown in the '(internal)' column). This observation is consistent with the Murlet paper.
Each alignment by each method was subjected to three RNA secondary structure prediction programs, Pfold, McCakill-MEA and RNAalifold. Accuracies of the predicted structures were assessed with Matthews Correlation Coefficient (MCC), calculated from TP, TN, FP and FN by the compare_ct.pl program. The curated structures in the Rfam database were assumed as correct.
Various combinations of alignment programs and RNA secondary structure prediction programs were examined as in the figure below.
For each alignment method, the accuracy values of all of the 52 alignments were plotted as a function of average percent identity and then a curve was plotted using a cubic spline. The curves of RNA sampler (the best of existing methods) and MASTR (only for internal prediction) are also shown for reference.