Summary table of models + methods

Introduction

Throughout the course, we will go over several supervised and unsupervised machine learning models. This page summarizes the models.

Method	Strengths	Limitations	Example Use Cases	Implementation
Simple Fill	Simple and fast Works well with small datasets	May not handle complex data relationships Sensitive to outliers	Basic data analysis Quick data cleaning	Python
KNN Imputation	Can capture the relationships between features Works well with moderately missing data	Computationally intensive for large datasets Sensitive to the choice of k	Medical data analysis Market research	Python
Soft Impute	Effective for matrix completion in large datasets Works well with low-rank data	Assumes low-rank data structure Can be sensitive to hyperparameters	Recommender systems Large-scale data projects	Python
Iterative Imputer	Can model complex relationships Suitable for multiple imputation	Computationally expensive Depends on the choice of model	Complex datasets with multiple types of missing data	Python
Iterative SVD	Good for matrix completion with low-rank assumption Handles larger datasets	Sensitive to rank selection Computationally demanding	Image and video data processing Large datasets with structure	Python
Matrix Factorization	Useful for recommendation systems Can handle large-scale problems	Requires careful tuning Not suitable for all types of data	Recommendation engines User preference analysis	Python
Nuclear Norm Minimization	Theoretically strong for matrix completion Finds the lowest rank solution	Very computationally intensive Impractical for very large datasets	Research in theoretical data completion Small to medium datasets	Python
BiScaler	Normalizes data effectively Often used as a preprocessing step	Not an imputation method itself Doesn’t always converge	Preprocessing for other imputation methods Data normalization	Python

Model Type	Strengths	Limitations	Example Use Cases	Implementation
Logistic Regression	Simple and interpretable Fast to train	Assumes linear boundaries Not suitable for complex relationships	Credit approval Medical diagnosis	Python
Decision Trees	Intuitive Can model non-linear relationships	Prone to overfitting Sensitive to small changes in data	Customer segmentation Loan default prediction	Python
Random Forest	Handles overfitting Can model complex relationships	Slower to train and predict Black box model	Fraud detection Stock price movement prediction	Python
Support Vector Machines (SVM)	Effective in high dimensional spaces Works well with clear margin of separation	Sensitive to kernel choice Slow on large datasets	Image classification Handwriting recognition	Python
K-Nearest Neighbors (KNN)	Simple and intuitive No training phase	Slow during query phase Sensitive to irrelevant features and scale	Product recommendation Document classification	Python
Neural Networks	Capable of approximating complex functions Flexible architecture Trainable with backpropagation	Can require a large number of parameters Prone to overfitting on small data Training can be slow	Pattern recognition Basic image classification Function approximation	Python
Deep Learning	Can model highly complex relationships Excels with vast amounts of data State-of-the-art results in many domains	Requires a lot of data Computationally intensive Interpretability challenges	Advanced image and speech recognition Machine translation Game playing (like AlphaGo)	Python
Naive Bayes	Fast Works well with large feature sets	Assumes feature independence Not suitable for numerical input features	Spam detection Sentiment analysis	Python
Gradient Boosting Machines (GBM)	High performance Handles non-linear relationships	Prone to overfitting if not tuned Slow to train	Web search ranking Ecology predictions	Python
Rule-Based Classification	Transparent and explainable Easily updated and modified	Manual rule creation can be tedious May not capture complex relationships	Expert systems Business rule enforcement	Python
Bagging	Reduces variance Parallelizable	May not handle bias well	Random Forest is a popular example	Python
Boosting	Reduces bias Combines weak learners	Sensitive to noisy data and outliers	AdaBoost Gradient Boosting	Python
XGBoost	Scalable and efficient Regularization	Requires careful tuning Can overfit if not used correctly	Competitions on Kaggle Retail prediction	Python
Linear Discriminant Analysis (LDA)	Dimensionality reduction Simple and interpretable	Assumes Gaussian distributed data and equal class covariances	Face recognition Marketing segmentation	Python
Regularized Models (Shrinking)	Prevents overfitting Handles collinearity	Requires parameter tuning May result in loss of interpretability	Ridge and Lasso regression	Python
Stacking	Combines multiple models Can improve accuracy	Increases model complexity Risk of overfitting if base models are correlated	Meta-modeling Kaggle competitions	Python

Model Type	Strengths	Limitations	Example Use Cases	Implementation
Linear Regression	Simple and interpretable	Assumes linear relationship Sensitive to outliers	Sales forecasting Risk assessment	Python
Polynomial Regression	Can model non-linear relationships	Can overfit with high degrees	Growth prediction Non-linear trend modeling	Python
Ridge Regression	Prevents overfitting Regularizes the model	Does not perform feature selection	High-dimensional data Preventing overfitting	Python
Lasso Regression	Feature selection Regularizes the model	May exclude useful variables	Feature selection High-dimensional datasets	Python
Elastic Net Regression	Balance between Ridge and Lasso	Requires tuning for mixing parameter	High-dimensional datasets with correlated features	Python
Quantile Regression	Models the median or other quantiles	Less interpretable than ordinary regression	Median house price prediction Financial quantiles modeling	Python
Support Vector Regression (SVR)	Flexible Can handle non-linear relationships	Sensitive to kernel and hyperparameters	Stock price prediction Non-linear trend modeling	Python
Decision Tree Regression	Handles non-linear data Interpretable	Can overfit on noisy data	Price prediction Quality assessment	Python
Random Forest Regression	Handles large datasets Reduces overfitting	Requires more computational resources	Large datasets Environmental modeling	Python
Gradient Boosting Regression	High performance Can handle non-linear relationships	Prone to overfitting if not tuned	Web search ranking Price prediction	Python

Model Type	Strengths	Limitations	Example Use Cases	Implementation
K-Means Clustering	Simple and widely used Fast for large datasets	Sensitive to initial conditions Requires specifying the number of clusters	Market segmentation Image compression	Python
Hierarchical Clustering	Doesn’t require specifying the number of clusters Produces a dendrogram	May be computationally expensive for large datasets	Taxonomies Determining evolutionary relationships	Python
DBSCAN (Density-Based Clustering)	Can find arbitrarily shaped clusters Doesn’t require specifying the number of clusters	Sensitive to scale Requires density parameters to be set	Noise detection and anomaly detection	Python
Agglomerative Clustering	Variety of linkage criteria Produces a hierarchy of clusters	Not scalable for very large datasets	Sociological hierarchies Taxonomies	Python
Mean Shift Clustering	No need to specify number of clusters Can find arbitrarily shaped clusters	Computationally expensive Bandwidth parameter selection is crucial	Image analysis Computer vision tasks	Python
Affinity Propagation	Automatically determines the number of clusters Good for data with lots of exemplars	High computational complexity Preference parameter can be difficult to choose	Image recognition Data with many similar exemplars	Python
Spectral Clustering	Can capture complex cluster structures Can be used with various affinity matrices	Choice of affinity matrix is crucial Can be computationally expensive	Image and speech processing Graph-based clustering	Python

Method	Strengths	Limitations	Example Use Cases	Implementation
PCA	Dimensionality reduction Preserves variance	Linear method Not for categorical data	Feature extraction Data compression	Python
t-SNE	Captures non-linear structures Good for visualization	Computationally expensive Not for high-dimensional data	Data visualization Exploratory analysis	Python
Autoencoders	Dimensionality reduction Non-linear relationships	Neural network knowledge Computationally intensive	Feature learning Noise reduction	Python
Isolation Forest	Effective for high-dimensional data Fast and scalable	Randomized May miss some anomalies	Fraud detection Network security	Python
SVD	Matrix factorization Efficient for large datasets	Assumes linear relationships Sensitive to scaling	Recommender systems Latent semantic analysis	Python
ICA	Identifies independent components Signal separation	Non-Gaussian components Sensitive to noise	Blind signal separation Feature extraction	Python

Method	Strengths	Limitations	Example Use Cases	Implementation
Apriori Algorithm	Well-known and widely used Easy to understand and implement	Can be slow on large datasets Generates a large number of candidate sets	Market basket analysis Cross-marketing strategies	Python
FP-Growth Algorithm	Faster than Apriori Efficient for large datasets	Memory intensive Can be complex to implement	Frequent itemset mining in large databases Customer purchase patterns	Python
Eclat Algorithm	Faster than Apriori Scalable and easy to parallelize	Limited to binary attributes Generates many candidate itemsets	Market basket analysis Binary classification tasks	Python
GSP (Generalized Sequential Pattern)	Identifies sequential patterns Flexible for various datasets	Can be computationally expensive Not as efficient for very large databases	Customer purchase sequence analysis Event sequence analysis	Python
RuleGrowth Algorithm	Efficient for mining sequential rules Works well with sparse datasets	Requires careful parameter setting Less known and used than Apriori or FP-Growth	Analyzing customer shopping sequences Detecting patterns in web browsing data	Python

Technique	Strengths	Limitations	Example Use Cases	Implementation
Accuracy	Simple and intuitive Effective for balanced datasets	Misleading for imbalanced datasets Doesn’t reflect true positives/negatives	General classification problems Comparing baseline models	Python
AUC-ROC	Effective for binary classification Good for imbalanced datasets	Can be overly optimistic in imbalanced data Not threshold-specific	Medical diagnosis classification Fraud detection models	Python
Precision	Focuses on positive class Reduces false positives	Ignores false negatives Not useful alone in imbalanced datasets	Spam detection Content moderation systems	Python
Recall	Identifies actual positives well Minimizes false negatives	Ignores false positives Can be misleading if positives are rare	Disease outbreak detection Recall-focused tasks	Python
F1-Score	Balances precision and recall Useful for imbalanced datasets	May not reflect true model performance Depends on balance of precision and recall	Customer churn prediction Sentiment analysis	Python
Cross-Validation	Reduces overfitting Provides robust model evaluation	Computationally expensive May not be ideal for very large datasets	General model evaluation Comparing multiple models	Python
The Validation Set Approach	Simple and easy to implement Good for initial model assessment	Can lead to overfitting Dependent on the split	Quick model prototyping Small datasets	Python
Leave-One-Out Cross-Validation	Very detailed Each observation used for validation exactly once	Computationally intensive Not suitable for large datasets	Small but rich datasets Highly sensitive models	Python
k-Fold Cross-Validation	Balances computational cost and validation accuracy Suitable for various data sizes	Variability in results depending on how data is divided Choice of ‘k’ can impact results	Medium-sized datasets Model selection	Python
The Bootstrap Method	Good for estimating model accuracy Effective for small datasets	Results can be sensitive to outliers May overestimate accuracy for small datasets	Small or medium-sized datasets Uncertainty estimation	Python