# Import all required libraries
# Data handling and manipulation
import pandas as pd
import numpy as np
# Machine learning models
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.decomposition import PCA
import mord as m
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
# Model evaluation and validation methods
from sklearn.model_selection import train_test_split
from sklearn.model_selection import KFold
from sklearn.model_selection import LeaveOneOut
from sklearn.model_selection import cross_val_score
from sklearn.utils import resample
# Metrics for model evaluation
from sklearn.metrics import accuracy_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
from sklearn.metrics import f1_score
from sklearn.metrics import roc_curve, auc
from sklearn.metrics import roc_auc_score
from sklearn.metrics import confusion_matrix
# Utility for data preprocessing
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import label_binarize
# For advanced visualizations
import matplotlib.pyplot as plt
import seaborn as sns
# Increase font size of all Seaborn plot elements
sns.set(font_scale = 1.25)
# Set Seaborn theme
sns.set_theme(style = "white")Classification II + Model Evaluation
Lecture 6
Looking back
Can we predict high-risk credit individuals by their financial traits?
What models will we reuse?
Ordinal Logistic Regression Accuracy: 0.549
Evaluating performance
Back to Confusion Matrices
Accuracy
Error Rate \(= \frac{FP + FN}{TP + TN + FP + FN}\)
Definition: The proportion of true results (both true positives and true negatives) among the total number of cases examined.
Use Case: Best for balanced datasets where each class is approximately equally represented.
Pros:
Intuitive and easy to understand.
Useful when the costs of false positives and false negatives are similar.
Cons:
- Can be misleading in imbalanced datasets (where one class significantly outnumbers the other).
- Doesn’t consider the type of errors (false positives vs. false negatives).
Precision
Error Rate \(= \frac{TP}{TP + FP}\)
Definition: The ratio of true positives to the sum of true and false positives. It’s a measure of a classifier’s exactness.
Use Case: Important when the cost of false positives is high (e.g., in spam detection).
Pros:
Focuses on the positive class’s predictive power.
Useful in imbalanced datasets to evaluate the performance of the minority class.
Cons:
- Does not consider false negatives (i.e., how many positive cases were missed).
Recall (Sensitivity)
Error Rate \(= \frac{TP}{TP + FN}\)
Definition: The ratio of true positives to the sum of true positives and false negatives. It’s a measure of a classifier’s completeness.
Use Case: Crucial when the cost of false negatives is high (e.g., in disease screening).
Pros:
Ensures that most positive examples are correctly recognized.
Very useful in imbalanced datasets for evaluating how well the minority class is being predicted
Cons:
- Does not consider false positives (i.e., can lead to many false alarms).
F1 Score
Error Rate \(= 2\times\frac{precision \times recall}{precision + recall} = \frac{TP}{TP + \frac{1}{2}(FP + FN)}\)
Definition: The harmonic mean of precision and recall.
Use Case: When you need a balance between precision and recall.
Pros:
Combines precision and recall into a single metric.
Useful in imbalanced datasets or when both types of errors are equally costly.
Cons:
- Not as intuitive as accuracy.
- Harmonic mean can be influenced heavily by lower values of either precision or recall.
AUC-ROC
Left side = more “confident” thresholds: lower recall and fewer false positive errors
Right side = “less strict” scenarios when the threshold is low: both recall and False Positive rates are higher, ultimately reaching 100%
Single metric to summarize the performance of models
Common calculation - trapezoid rule
Definition: Measures the ability of a classifier to distinguish between classes and is used as a summary of the ROC curve.
Use Case: Effective for balanced and imbalanced datasets, especially for binary classification problems.
Pros:
Performance measurement for the classification model at various thresholds settings.
AUC-ROC near 1 indicates a good ability to distinguish between positive and negative classes.
Cons:
Can be overly optimistic in imbalanced datasets.
Does not distinguish between types of errors.
Leave-One-Out Cross-Validation (LOOCV)
Overall score:
\(\frac{score_1 + score_2 + score_3 + score_4 + score_5 + score_6}{6}\)
For classification the score is:
- Accuracy
- Precision, Recall, F1-Score
- AUC-ROC
Definition: The number of folds equals the number of instances in the dataset. Each model is trained on all data points except one, which is used as the test set.
Use Case: Useful for small datasets where maximizing the training data is important.
Pros:
Utilizes the data to its maximum extent.
Reduces bias as each data point gets to be in the test set exactly once.
Cons:
- Highly computationally expensive with large datasets.
- High variance in the estimate of model performance as the evaluation can be highly dependent on the data points chosen as the test set.
k-Fold Cross-Validation (k-Fold)
Overall score:
\(\frac{score_1 + score_2 + score_3}{3}\)
For classification the score is:
- Accuracy
- Precision, Recall, F1-Score
- AUC-ROC
Definition: The dataset is divided into k subsets, and the holdout method is repeated k times. Each time, one of the k subsets is used as the test set and the other k-1 subsets are put together to form a training set.
Use Case: Ideal for both small and medium-sized datasets and when the balance between bias and variance is crucial.
Pros:
Reduces the variance of a single trial of train/test split.
More reliable estimate of out-of-sample performance than LOOCV.
Cons:
- Still computationally intensive, especially with large k.
- Results can be dependent on the random division of the data into folds.
Definition: Refers to managing the trade-off between the bias of the model (error due to overly simplistic assumptions) and its variance (error due to sensitivity to small fluctuations in the training set).
The bootstrap method
Sample creation
Model training + evaluation
Performance aggregation
Definition: A statistical technique using repeated sampling with replacement from a dataset to train and evaluate model performance.
Use Case: Ideal for small datasets or when assessing model performance variability. Commonly used for accuracy estimation and model validation.
Pros:
Variability estimation: Offers insights into the model’s performance variability.
Flexible: Non-parametric and adaptable to various data distributions.
Cons:
Computationally demanding: High computational cost due to repeated resampling and training.
Overfitting bias: Can be overly optimistic for overfitting models.
In-class Exercise
Go to ex-06 and perform the tasks
