Example: Multiclass classification¶
This example shows how to compare the performance of three models on a multiclass classification task.
Import the wine dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict wines into three groups (which cultivator it's from) using features based on the results of chemical analysis.
Load the data¶
In [1]:
Copied!
# Import packages
from sklearn.datasets import load_wine
from atom import ATOMClassifier
# Import packages
from sklearn.datasets import load_wine
from atom import ATOMClassifier
In [2]:
Copied!
# Load data
X, y = load_wine(return_X_y=True, as_frame=True)
# Let's have a look
X.head()
# Load data
X, y = load_wine(return_X_y=True, as_frame=True)
# Let's have a look
X.head()
Out[2]:
alcohol | malic_acid | ash | alcalinity_of_ash | magnesium | total_phenols | flavanoids | nonflavanoid_phenols | proanthocyanins | color_intensity | hue | od280/od315_of_diluted_wines | proline | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 14.23 | 1.71 | 2.43 | 15.6 | 127.0 | 2.80 | 3.06 | 0.28 | 2.29 | 5.64 | 1.04 | 3.92 | 1065.0 |
1 | 13.20 | 1.78 | 2.14 | 11.2 | 100.0 | 2.65 | 2.76 | 0.26 | 1.28 | 4.38 | 1.05 | 3.40 | 1050.0 |
2 | 13.16 | 2.36 | 2.67 | 18.6 | 101.0 | 2.80 | 3.24 | 0.30 | 2.81 | 5.68 | 1.03 | 3.17 | 1185.0 |
3 | 14.37 | 1.95 | 2.50 | 16.8 | 113.0 | 3.85 | 3.49 | 0.24 | 2.18 | 7.80 | 0.86 | 3.45 | 1480.0 |
4 | 13.24 | 2.59 | 2.87 | 21.0 | 118.0 | 2.80 | 2.69 | 0.39 | 1.82 | 4.32 | 1.04 | 2.93 | 735.0 |
Run the pipeline¶
In [3]:
Copied!
atom = ATOMClassifier(X, y, n_jobs=1, verbose=2, random_state=1)
# Fit the pipeline with the selected models
atom.run(
models=["LR","LDA", "RF"],
metric="roc_auc_ovr",
n_trials=14,
n_bootstrap=5,
errors="raise",
)
atom = ATOMClassifier(X, y, n_jobs=1, verbose=2, random_state=1)
# Fit the pipeline with the selected models
atom.run(
models=["LR","LDA", "RF"],
metric="roc_auc_ovr",
n_trials=14,
n_bootstrap=5,
errors="raise",
)
<< ================== ATOM ================== >> Configuration ==================== >> Algorithm task: Multiclass classification. Dataset stats ==================== >> Shape: (178, 14) Train set size: 143 Test set size: 35 ------------------------------------- Memory: 19.36 kB Scaled: False Outlier values: 12 (0.6%) Training ========================= >> Models: LR, LDA, RF Metric: roc_auc_ovr Running hyperparameter tuning for LogisticRegression... | trial | penalty | C | solver | max_iter | l1_ratio | roc_auc_ovr | best_roc_auc_ovr | time_trial | time_ht | state | | ----- | ------- | ------- | ------- | -------- | -------- | ----------- | ---------------- | ---------- | ------- | -------- | | 0 | l1 | 0.0054 | saga | 480 | 0.7 | 0.5 | 0.5 | 0.027s | 0.027s | COMPLETE | | 1 | l1 | 0.122 | saga | 380 | 0.7 | 0.9951 | 0.9951 | 0.042s | 0.069s | COMPLETE | | 2 | l2 | 0.0071 | sag | 720 | 0.3 | 1.0 | 1.0 | 0.033s | 0.101s | COMPLETE | | 3 | l1 | 87.9641 | libli.. | 920 | 0.3 | 1.0 | 1.0 | 0.030s | 0.131s | COMPLETE | | 4 | l2 | 0.0114 | sag | 630 | 0.7 | 1.0 | 1.0 | 0.031s | 0.162s | COMPLETE | | 5 | l2 | 0.0018 | sag | 920 | 0.1 | 1.0 | 1.0 | 0.029s | 0.191s | COMPLETE | | 6 | l2 | 43.4053 | sag | 780 | 0.3 | 1.0 | 1.0 | 0.050s | 0.241s | COMPLETE | | 7 | l2 | 2.0759 | libli.. | 470 | 0.2 | 1.0 | 1.0 | 0.033s | 0.274s | COMPLETE | | 8 | None | 0.043 | sag | 110 | 1.0 | 1.0 | 1.0 | 0.028s | 0.301s | COMPLETE | | 9 | l1 | 46.0233 | saga | 740 | 0.1 | 1.0 | 1.0 | 0.059s | 0.360s | COMPLETE | | 10 | l2 | 1.2173 | lbfgs | 280 | 0.5 | 1.0 | 1.0 | 0.050s | 0.410s | COMPLETE | | 11 | l2 | 6.8102 | libli.. | 940 | 0.4 | 1.0 | 1.0 | 0.041s | 0.451s | COMPLETE | | 12 | l2 | 0.3324 | newto.. | 780 | 0.3 | 1.0 | 1.0 | 0.052s | 0.503s | COMPLETE | | 13 | l2 | 0.0012 | libli.. | 1000 | 0.0 | 1.0 | 1.0 | 0.033s | 0.536s | COMPLETE | Hyperparameter tuning --------------------------- Best trial --> 2 Best parameters: --> penalty: l2 --> C: 0.0071 --> solver: sag --> max_iter: 720 --> l1_ratio: 0.3 Best evaluation --> roc_auc_ovr: 1.0 Time elapsed: 0.536s Fit --------------------------------------------- Train evaluation --> roc_auc_ovr: 0.9991 Test evaluation --> roc_auc_ovr: 0.9977 Time elapsed: 0.106s Bootstrap --------------------------------------- Evaluation --> roc_auc_ovr: 0.9984 ± 0.001 Time elapsed: 0.111s ------------------------------------------------- Time: 0.753s Running hyperparameter tuning for LinearDiscriminantAnalysis... | trial | solver | shrinkage | roc_auc_ovr | best_roc_auc_ovr | time_trial | time_ht | state | | ----- | ------- | --------- | ----------- | ---------------- | ---------- | ------- | -------- | | 0 | lsqr | 0.9 | 0.9221 | 0.9221 | 0.018s | 0.018s | COMPLETE | | 1 | eigen | 1.0 | 0.9121 | 0.9221 | 0.020s | 0.038s | COMPLETE | | 2 | eigen | 1.0 | 0.9121 | 0.9221 | 0.000s | 0.038s | COMPLETE | | 3 | lsqr | 0.7 | 0.8638 | 0.9221 | 0.011s | 0.049s | COMPLETE | | 4 | eigen | 0.7 | 0.9019 | 0.9221 | 0.016s | 0.065s | COMPLETE | | 5 | lsqr | auto | 1.0 | 1.0 | 0.022s | 0.087s | COMPLETE | | 6 | eigen | 1.0 | 0.9121 | 1.0 | 0.000s | 0.087s | COMPLETE | | 7 | lsqr | 1.0 | 0.9445 | 1.0 | 0.012s | 0.099s | COMPLETE | | 8 | svd | None | 1.0 | 1.0 | 0.014s | 0.113s | COMPLETE | | 9 | svd | None | 1.0 | 1.0 | 0.001s | 0.114s | COMPLETE | | 10 | lsqr | auto | 1.0 | 1.0 | 0.002s | 0.116s | COMPLETE | | 11 | svd | None | 1.0 | 1.0 | 0.003s | 0.119s | COMPLETE | | 12 | svd | None | 1.0 | 1.0 | 0.002s | 0.121s | COMPLETE | | 13 | svd | None | 1.0 | 1.0 | 0.002s | 0.123s | COMPLETE | Hyperparameter tuning --------------------------- Best trial --> 5 Best parameters: --> solver: lsqr --> shrinkage: auto Best evaluation --> roc_auc_ovr: 1.0 Time elapsed: 0.123s Fit --------------------------------------------- Train evaluation --> roc_auc_ovr: 1.0 Test evaluation --> roc_auc_ovr: 1.0 Time elapsed: 0.038s Bootstrap --------------------------------------- Evaluation --> roc_auc_ovr: 0.9998 ± 0.0005 Time elapsed: 0.091s ------------------------------------------------- Time: 0.252s Running hyperparameter tuning for RandomForest... | trial | n_estimators | criterion | max_depth | min_samples_split | min_samples_leaf | max_features | bootstrap | max_samples | ccp_alpha | roc_auc_ovr | best_roc_auc_ovr | time_trial | time_ht | state | | ----- | ------------ | --------- | --------- | ----------------- | ---------------- | ------------ | --------- | ----------- | --------- | ----------- | ---------------- | ---------- | ------- | -------- | | 0 | 210 | gini | 10 | 17 | 20 | 0.5 | False | None | 0.0 | 0.9803 | 0.9803 | 0.218s | 0.218s | COMPLETE | | 1 | 380 | gini | 4 | 15 | 3 | 0.9 | False | None | 0.01 | 0.9816 | 0.9816 | 0.472s | 0.690s | COMPLETE | | 2 | 380 | entropy | 6 | 2 | 13 | 0.9 | False | None | 0.03 | 0.9944 | 0.9944 | 0.502s | 1.192s | COMPLETE | | 3 | 470 | gini | 11 | 9 | 18 | nan | True | 0.6 | 0.025 | 0.9569 | 0.9944 | 0.500s | 1.692s | COMPLETE | | 4 | 100 | entropy | 12 | 14 | 6 | 0.9 | False | nan | 0.035 | 1.0 | 1.0 | 0.146s | 1.838s | COMPLETE | | 5 | 470 | entropy | 13 | 11 | 1 | nan | True | 0.6 | 0.01 | 1.0 | 1.0 | 0.750s | 2.588s | COMPLETE | | 6 | 250 | gini | 14 | 13 | 17 | 0.7 | True | nan | 0.02 | 1.0 | 1.0 | 0.309s | 2.897s | COMPLETE | | 7 | 220 | gini | 5 | 10 | 7 | 0.5 | True | 0.9 | 0.035 | 0.9981 | 1.0 | 0.292s | 3.189s | COMPLETE | | 8 | 130 | entropy | 4 | 6 | 11 | 0.9 | False | nan | 0.03 | 1.0 | 1.0 | 0.199s | 3.387s | COMPLETE | | 9 | 370 | gini | 12 | 2 | 4 | 0.5 | False | nan | 0.02 | 0.9916 | 1.0 | 0.392s | 3.780s | COMPLETE | | 10 | 10 | entropy | 12 | 20 | 7 | log2 | False | nan | 0.035 | 1.0 | 1.0 | 0.044s | 3.824s | COMPLETE | | 11 | 70 | entropy | 13 | 12 | 1 | None | True | 0.5 | 0.01 | 0.9928 | 1.0 | 0.124s | 3.948s | COMPLETE | | 12 | 500 | entropy | 9 | 7 | 7 | 0.6 | True | 0.6 | 0.01 | 1.0 | 1.0 | 0.624s | 4.572s | COMPLETE | | 13 | 140 | entropy | 16 | 16 | 1 | 0.8 | True | 0.7 | 0.0 | 1.0 | 1.0 | 0.192s | 4.764s | COMPLETE | Hyperparameter tuning --------------------------- Best trial --> 4 Best parameters: --> n_estimators: 100 --> criterion: entropy --> max_depth: 12 --> min_samples_split: 14 --> min_samples_leaf: 6 --> max_features: 0.9 --> bootstrap: False --> max_samples: None --> ccp_alpha: 0.035 Best evaluation --> roc_auc_ovr: 1.0 Time elapsed: 4.764s Fit --------------------------------------------- Train evaluation --> roc_auc_ovr: 0.9993 Test evaluation --> roc_auc_ovr: 1.0 Time elapsed: 0.207s Bootstrap --------------------------------------- Evaluation --> roc_auc_ovr: 0.9936 ± 0.0067 Time elapsed: 0.602s ------------------------------------------------- Time: 5.573s Final results ==================== >> Total time: 9.866s ------------------------------------- LogisticRegression --> roc_auc_ovr: 0.9984 ± 0.001 LinearDiscriminantAnalysis --> roc_auc_ovr: 0.9998 ± 0.0005 ! RandomForest --> roc_auc_ovr: 0.9936 ± 0.0067
Analyze the results¶
In [4]:
Copied!
atom.results
atom.results
Out[4]:
roc_auc_ovr_ht | time_ht | roc_auc_ovr_train | roc_auc_ovr_test | time_fit | roc_auc_ovr_bootstrap | time_bootstrap | time | |
---|---|---|---|---|---|---|---|---|
LR | 1.0 | 0.536130 | 0.9979 | 0.9977 | 0.105813 | 0.998413 | 0.111023 | 0.752966 |
LDA | 1.0 | 0.122947 | 1.0000 | 0.9989 | 0.038458 | 0.999773 | 0.090809 | 0.252214 |
RF | 1.0 | 4.764301 | 0.9993 | 1.0000 | 0.206899 | 0.993613 | 0.601960 | 5.573160 |
In [5]:
Copied!
# Show the score for some different metrics
atom.evaluate(["precision_macro", "recall_macro", "jaccard_weighted"])
# Show the score for some different metrics
atom.evaluate(["precision_macro", "recall_macro", "jaccard_weighted"])
Out[5]:
precision_macro | recall_macro | jaccard_weighted | |
---|---|---|---|
LR | 0.942900 | 0.948400 | 0.892400 |
LDA | 0.966700 | 0.976200 | 0.945700 |
RF | 0.879900 | 0.891500 | 0.796800 |
In [6]:
Copied!
# Some plots allow you to choose the target class to look at
atom.rf.plot_probabilities(rows="train", target=0)
# Some plots allow you to choose the target class to look at
atom.rf.plot_probabilities(rows="train", target=0)
In [7]:
Copied!
atom.lda.plot_shap_heatmap(target=2, show=7)
atom.lda.plot_shap_heatmap(target=2, show=7)