We will be offering an R workshop December 18-20, 2019. Learn more.

Pre.cluster

From mothur
Revision as of 12:08, 3 February 2010 by Pschloss (Talk | contribs) (New page: The pre.cluster command implements a pseudo-single linkage algorithm with the goal of removing sequences that are likely due to pyrosequencing errors. A version of this algorithm was ...)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

The pre.cluster command implements a pseudo-single linkage algorithm with the goal of removing sequences that are likely due to pyrosequencing errors. A version of this algorithm was developed by Sue Huse and will be published in a forthcoming paper in Environmental Microbiology. The basic idea is that abundant sequences are more likely to generate erroneous sequences than rare sequences. With that in mind, the algorithm proceeds by ranking sequences in order of their abundance. Then we walk through the list of sequences looking for rarer sequences that are within some threshold of the original sequence. Those that are within the threshold are merged with the larger sequence. The original Huse method performs this task on a distance matrix, whereas we do it based on the original sequences. The advantage of our approach is that the algorithm works on aligned sequences instead of a distance matrix. This is advantageous because by pre-clustering you remove a large number of sequences making the distance calculation much faster.


Default settings

The pre.cluster command expects a fasta-formatted file and a names file and that the sequences are in the same order in both files. Both of these files can be generated by the unique.seqs command. For example, if you are following along with the Sogin data analysis example and have aligned, filtered, and unique'd your sequences, then enter the following to perform the pre.clustering command:

mothur > unique.seqs(fasta=sogin.unique.filter.fasta, name=sogin.names)
mothur > pre.cluster(fasta=sogin.unique.filter.unique.fasta, name=sogin.unique.filter.names)

Will result in the following output:

0	21821	86
100	20286	1621
200	19824	2083
...


This output indicates, by column, the number of sequences processed, the number of sequences that will be found in the final dataset, and the number of sequences that have been clustered away. This should accelerate as the function runs. Two files are created - a *.precluster.fasta and a *.precluster.names file containing the new sequence and names file for further processing.


Options

diffs

By default the pre.cluster command will look for sequences that are within 1 mismatch of the sequence being considered. With the diffs option you can change this threshold. For example:

mothur > pre.cluster(fasta=sogin.unique.filter.unique.fasta, name=sogin.unique.filter.names, diffs=2)



fasta only

As shown above, pre.cluster expects you to provide a name file so that it can acquire the abundance information from each sequence. If you do not provide the name file the command will automatically run your data through unique.seqs to generate to get the information it needs.


Caveat emptor

Something to keep in mind is that when you set the number of mismatches to 2, you are allowing that the maximum difference between sequences within a cluster to be 4 (2 from the dominant sequence in one direction, and 2 in any other direction). This difference of 4 bases, could cloud your ability to distinguish signal from noise. For example, with this Sogin datasets, the sequences are ~60 bp V6 pyrotags. A difference of 4 bases is 6.7%! Alternatively, when using diffs=1, the difference of 2 bases is 3.3%. The assumption of the algorithm is that these mismatches are noise; however, it doesn't make sense to then analyze your data at 3% by either level of diffs. Remember that this method does not actually remove the noise, it just clusters sequences that are likely to be noisy. To remove the noise you would need to use a program like Chris Quince's Pyronoise. Considering using PyroNoise may not be practical for many people, the pre.cluster option may be your best bet.

Finally, in the Huse et al. paper they use an average neighbor clustering algorithm once the pre.clustering is done. We have mixed feelings about this. Their choice of clustering algorithm was made because it gave the output that they were expecting based on the composition of a mock community. There is always a risk of over-training a method to suit your data and then applying it to "real" data. The risk is that sequences that are legitimately different would get clustered together. The advantage of the furthest neighbor algorithm is that we know an OTU represents a collection of sequences that are no more than X% different from each other. The same cannot be said for the average or nearest neighbor algorithms.