Protein Structure and Function. Application of Bioinformatics Methods - John Rigden 2014

Prediction of Membrane Protein Structure
Prediction of Transmembrane Protein Topology
Datasets, Homology, Accuracy, and Cross-Validation

When developing any prediction method, it is crucial to use high-quality data for both training and assessing the reliability of the results. Extracting a training set from available Databases is a highly labor-intensive task that requires making numerous important decisions. For example, consider searching the PDB database using the keyword "transmembrane". The resulting hits will include both genome-encoded transmembrane Proteins and non-native transmembrane proteins, such as the bee venom peptide 1BH1, which disrupts the structural integrity of bilayers, or the bacterial colicin ICII, which is used to form Pores in the outer membranes of competing Bacteria. Moreover, databases frequently contain errors that introduce distortions into the method. For machine learning-based Methods, such distortions do not significantly impact study results, whereas for smaller datasets, this issue becomes much more significant.

Class="center">Structure/structure.files/image025.jpg" width="416"/>

Fig. 4.6. Eleven proteomes analyzed using the MEMSAT3 transmembrane/globular protein discrimination method. For proteins identified as transmembrane, a full TM topology prediction was performed. The x-axis represents the transmembrane helix number, and the y-axis represents the number of proteins.

Another issue that requires consideration is Sequence Homology within datasets, which in most cases accounts for 30–40% sequence identity. Since data on TM protein structures are currently generating heightened interest, this figure is likely somewhat higher than the one typical of globular protein datasets. Despite the increased risk of overfitting, it is essential to use training sets of sufficient size. All machine learning-based methods are characterized by numerous free parameters and, consequently, the potential for overfitting. This means that instead of identifying a particular sequence pattern, the algorithm may simply "memorize" it along with all possible errors it might contain. An overfitted method typically reproduces training Examples with high accuracy, while proving largely ineffective for previously unseen examples. To avoid overfitting when evaluating the accuracy of a prediction method, it is crucial to use training and test sets with low homology.

In all cases, It is important to apply rigorous cross-validation to data reliability. Cross-validation is a statistical method that involves splitting the original dataset into smaller subsets. Each such subset is tested against a model trained using the remaining subsets. The process is repeated until all subsets have been tested. When predicting TM topology, Two Types of cross-validation are most commonly used. In K-fold cross-validation, the dataset is divided into K subsets. One of the K subsets, containing several sequences, is used as the test set, while the remaining K-1 subsets serve as the training set. The Procedure is repeated K times, with each of the K subsets being used exactly once as the test set. The resulting K scores are then either combined or averaged to yield an overall evaluation. A more rigorous, albeit computationally more expensive, type of cross-validation is leave-one-out cross-validation (LOOCV), also known as the jackknife method. This approach involves using a single sequence from the dataset as the test set, while all other sequences make up the training set. The procedure is repeated until every sequence has been tested once. This method is essentially a special case of K-fold cross-validation where K equals the number of sequences in the dataset.

Several studies have attempted to compare the accuracy of various transmembrane protein topology prediction methods (e.g., Melen et al., 2003); however, the methods themselves have been significantly improved since then. It is now generally accepted that the best prediction methods achieve correct topologies for 80–93% of proteins, although evaluating methods is quite difficult in the absence of independent cross-validation using a common benchmark dataset. Methods may exhibit high predictive performance when tested on a specific dataset—for instance, one containing a small number of signal Peptides—while their performance may drop when applied to a dataset rich in signal peptides. Methods can also be optimized using a dataset containing many weakly hydrophobic TM helices, in which case they become highly prone to over-detecting TM helices in the analyzed datasets. To date, the highest-quality datasets, in which topologies were determined exclusively by structural methods and sequence homology was reduced, contain no more than 150 sequences (Lomize et al., 2006b). The insufficient consistency among these data, coupled with a lack of necessary cross-validation data, thus implies that differences in the accuracy of various prediction methods may stem from differences in the training and validation datasets rather than from notable differences in the quality of the approaches themselves.