Example: Multiclass classification¶
This example shows how to compare the performance of three models on a multiclass classification task.
Import the wine dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict wines into three groups (which cultivator it's from) using features based on the results of chemical analysis.
Load the data¶
In [1]:
Copied!
# Import packages
from sklearn.datasets import load_wine
from atom import ATOMClassifier
# Import packages
from sklearn.datasets import load_wine
from atom import ATOMClassifier
UserWarning: The pandas version installed (1.5.3) does not match the supported pandas version in Modin (1.5.2). This may cause undesired side effects!
In [2]:
Copied!
# Load data
X, y = load_wine(return_X_y=True, as_frame=True)
# Let's have a look
X.head()
# Load data
X, y = load_wine(return_X_y=True, as_frame=True)
# Let's have a look
X.head()
Out[2]:
| alcohol | malic_acid | ash | alcalinity_of_ash | magnesium | total_phenols | flavanoids | nonflavanoid_phenols | proanthocyanins | color_intensity | hue | od280/od315_of_diluted_wines | proline | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 14.23 | 1.71 | 2.43 | 15.6 | 127.0 | 2.80 | 3.06 | 0.28 | 2.29 | 5.64 | 1.04 | 3.92 | 1065.0 |
| 1 | 13.20 | 1.78 | 2.14 | 11.2 | 100.0 | 2.65 | 2.76 | 0.26 | 1.28 | 4.38 | 1.05 | 3.40 | 1050.0 |
| 2 | 13.16 | 2.36 | 2.67 | 18.6 | 101.0 | 2.80 | 3.24 | 0.30 | 2.81 | 5.68 | 1.03 | 3.17 | 1185.0 |
| 3 | 14.37 | 1.95 | 2.50 | 16.8 | 113.0 | 3.85 | 3.49 | 0.24 | 2.18 | 7.80 | 0.86 | 3.45 | 1480.0 |
| 4 | 13.24 | 2.59 | 2.87 | 21.0 | 118.0 | 2.80 | 2.69 | 0.39 | 1.82 | 4.32 | 1.04 | 2.93 | 735.0 |
Run the pipeline¶
In [3]:
Copied!
atom = ATOMClassifier(X, y, n_jobs=-1, verbose=2, random_state=1)
# Fit the pipeline with the selected models
atom.run(
models=["LR","LDA", "RF"],
metric="roc_auc_ovr",
n_trials=14,
n_bootstrap=5,
errors="raise",
)
atom = ATOMClassifier(X, y, n_jobs=-1, verbose=2, random_state=1)
# Fit the pipeline with the selected models
atom.run(
models=["LR","LDA", "RF"],
metric="roc_auc_ovr",
n_trials=14,
n_bootstrap=5,
errors="raise",
)
<< ================== ATOM ================== >> Algorithm task: multiclass classification. Parallel processing with 16 cores. Parallelization backend: loky Dataset stats ==================== >> Shape: (178, 14) Train set size: 143 Test set size: 35 ------------------------------------- Memory: 19.35 kB Scaled: False Outlier values: 12 (0.6%) Training ========================= >> Models: LR, LDA, RF Metric: roc_auc_ovr Running hyperparameter tuning for LogisticRegression... | trial | penalty | C | solver | max_iter | l1_ratio | roc_auc_ovr | best_roc_auc_ovr | time_trial | time_ht | state | | ----- | ------- | ------- | ------- | -------- | -------- | ----------- | ---------------- | ---------- | ------- | -------- | | 0 | l1 | 0.0054 | saga | 480 | --- | 0.5 | 0.5 | 0.870s | 0.870s | COMPLETE | | 1 | l1 | 0.122 | saga | 380 | --- | 0.9951 | 0.9951 | 0.509s | 1.379s | COMPLETE | | 2 | l2 | 0.0071 | sag | 720 | --- | 1.0 | 1.0 | 0.391s | 1.770s | COMPLETE | | 3 | l1 | 87.9641 | libli.. | 920 | --- | 1.0 | 1.0 | 0.020s | 1.790s | COMPLETE | | 4 | l2 | 0.0114 | sag | 630 | --- | 1.0 | 1.0 | 0.371s | 2.161s | COMPLETE | | 5 | l2 | 0.0018 | sag | 920 | --- | 1.0 | 1.0 | 0.366s | 2.528s | COMPLETE | | 6 | l2 | 43.4053 | sag | 780 | --- | 1.0 | 1.0 | 0.379s | 2.907s | COMPLETE | | 7 | l2 | 2.0759 | libli.. | 470 | --- | 1.0 | 1.0 | 0.019s | 2.926s | COMPLETE | | 8 | None | --- | sag | 110 | --- | 1.0 | 1.0 | 0.391s | 3.317s | COMPLETE | | 9 | l1 | 46.0233 | saga | 740 | --- | 1.0 | 1.0 | 0.557s | 3.874s | COMPLETE | | 10 | None | --- | lbfgs | 280 | --- | 1.0 | 1.0 | 0.525s | 4.399s | COMPLETE | | 11 | l1 | 6.8102 | libli.. | 940 | --- | 1.0 | 1.0 | 0.020s | 4.419s | COMPLETE | | 12 | l2 | 0.3324 | newto.. | 780 | --- | 1.0 | 1.0 | 0.368s | 4.788s | COMPLETE | | 13 | l1 | 0.0012 | libli.. | 1000 | --- | 0.5 | 1.0 | 0.020s | 4.808s | COMPLETE | Hyperparameter tuning --------------------------- Best trial --> 2 Best parameters: --> penalty: l2 --> C: 0.0071 --> solver: sag --> max_iter: 720 Best evaluation --> roc_auc_ovr: 1.0 Time elapsed: 4.808s Fit --------------------------------------------- Train evaluation --> roc_auc_ovr: 0.9991 Test evaluation --> roc_auc_ovr: 0.9977 Time elapsed: 0.362s Bootstrap --------------------------------------- Evaluation --> roc_auc_ovr: 0.9984 ± 0.001 Time elapsed: 1.770s ------------------------------------------------- Total time: 6.940s Running hyperparameter tuning for LinearDiscriminantAnalysis... | trial | solver | shrinkage | roc_auc_ovr | best_roc_auc_ovr | time_trial | time_ht | state | | ----- | ------- | --------- | ----------- | ---------------- | ---------- | ------- | -------- | | 0 | lsqr | 0.9 | 0.9221 | 0.9221 | 0.017s | 0.017s | COMPLETE | | 1 | eigen | 1.0 | 0.9121 | 0.9221 | 0.011s | 0.028s | COMPLETE | | 2 | eigen | 1.0 | 0.9121 | 0.9221 | 0.001s | 0.029s | COMPLETE | | 3 | lsqr | 0.7 | 0.8638 | 0.9221 | 0.009s | 0.038s | COMPLETE | | 4 | eigen | 0.7 | 0.9019 | 0.9221 | 0.010s | 0.048s | COMPLETE | | 5 | lsqr | auto | 1.0 | 1.0 | 0.010s | 0.058s | COMPLETE | | 6 | eigen | 1.0 | 0.9121 | 1.0 | 0.001s | 0.059s | COMPLETE | | 7 | lsqr | 1.0 | 0.9445 | 1.0 | 0.009s | 0.068s | COMPLETE | | 8 | svd | --- | 1.0 | 1.0 | 0.008s | 0.076s | COMPLETE | | 9 | svd | --- | 1.0 | 1.0 | 0.001s | 0.077s | COMPLETE | | 10 | lsqr | auto | 1.0 | 1.0 | 0.002s | 0.079s | COMPLETE | | 11 | svd | --- | 1.0 | 1.0 | 0.002s | 0.081s | COMPLETE | | 12 | svd | --- | 1.0 | 1.0 | 0.002s | 0.083s | COMPLETE | | 13 | svd | --- | 1.0 | 1.0 | 0.002s | 0.085s | COMPLETE | Hyperparameter tuning --------------------------- Best trial --> 5 Best parameters: --> solver: lsqr --> shrinkage: auto Best evaluation --> roc_auc_ovr: 1.0 Time elapsed: 0.085s Fit --------------------------------------------- Train evaluation --> roc_auc_ovr: 1.0 Test evaluation --> roc_auc_ovr: 1.0 Time elapsed: 0.014s Bootstrap --------------------------------------- Evaluation --> roc_auc_ovr: 0.9998 ± 0.0005 Time elapsed: 0.037s ------------------------------------------------- Total time: 0.136s Running hyperparameter tuning for RandomForest... | trial | n_estimators | criterion | max_depth | min_samples_split | min_samples_leaf | max_features | bootstrap | max_samples | ccp_alpha | roc_auc_ovr | best_roc_auc_ovr | time_trial | time_ht | state | | ----- | ------------ | --------- | --------- | ----------------- | ---------------- | ------------ | --------- | ----------- | --------- | ----------- | ---------------- | ---------- | ------- | -------- | | 0 | 210 | gini | 10 | 17 | 20 | 0.5 | False | --- | 0.0 | 0.9803 | 0.9803 | 0.318s | 0.318s | COMPLETE | | 1 | 380 | gini | 4 | 15 | 3 | 0.9 | False | --- | 0.01 | 0.9816 | 0.9816 | 0.350s | 0.668s | COMPLETE | | 2 | 380 | entropy | 6 | 2 | 13 | 0.9 | False | --- | 0.03 | 0.9944 | 0.9944 | 0.357s | 1.025s | COMPLETE | | 3 | 470 | gini | 11 | 9 | 18 | None | True | 0.6 | 0.025 | 0.9569 | 0.9944 | 0.501s | 1.527s | COMPLETE | | 4 | 100 | entropy | 12 | 14 | 6 | 0.9 | False | --- | 0.035 | 1.0 | 1.0 | 0.143s | 1.670s | COMPLETE | | 5 | 470 | entropy | 13 | 11 | 1 | None | True | 0.6 | 0.01 | 1.0 | 1.0 | 0.484s | 2.154s | COMPLETE | | 6 | 250 | gini | 14 | 13 | 17 | 0.7 | True | None | 0.02 | 1.0 | 1.0 | 0.268s | 2.422s | COMPLETE | | 7 | 220 | gini | 5 | 10 | 7 | 0.5 | True | 0.9 | 0.035 | 0.9981 | 1.0 | 0.273s | 2.695s | COMPLETE | | 8 | 130 | entropy | 4 | 6 | 11 | 0.9 | False | --- | 0.03 | 1.0 | 1.0 | 0.193s | 2.889s | COMPLETE | | 9 | 370 | gini | 12 | 2 | 4 | 0.5 | False | --- | 0.02 | 0.9916 | 1.0 | 0.317s | 3.206s | COMPLETE | | 10 | 10 | entropy | None | 20 | 7 | log2 | False | --- | 0.035 | 1.0 | 1.0 | 0.041s | 3.247s | COMPLETE | | 11 | 70 | entropy | 13 | 12 | 1 | None | True | 0.5 | 0.01 | 0.9928 | 1.0 | 0.117s | 3.364s | COMPLETE | | 12 | 500 | entropy | 9 | 7 | 7 | 0.6 | True | 0.6 | 0.01 | 1.0 | 1.0 | 0.392s | 3.756s | COMPLETE | | 13 | 140 | entropy | 16 | 16 | 1 | 0.8 | True | 0.5 | 0.0 | 1.0 | 1.0 | 0.204s | 3.961s | COMPLETE | Hyperparameter tuning --------------------------- Best trial --> 4 Best parameters: --> n_estimators: 100 --> criterion: entropy --> max_depth: 12 --> min_samples_split: 14 --> min_samples_leaf: 6 --> max_features: 0.9 --> bootstrap: False --> ccp_alpha: 0.035 Best evaluation --> roc_auc_ovr: 1.0 Time elapsed: 3.961s Fit --------------------------------------------- Train evaluation --> roc_auc_ovr: 0.9993 Test evaluation --> roc_auc_ovr: 1.0 Time elapsed: 0.149s Bootstrap --------------------------------------- Evaluation --> roc_auc_ovr: 0.9936 ± 0.0067 Time elapsed: 0.639s ------------------------------------------------- Total time: 4.749s Final results ==================== >> Total time: 12.149s ------------------------------------- LogisticRegression --> roc_auc_ovr: 0.9984 ± 0.001 LinearDiscriminantAnalysis --> roc_auc_ovr: 0.9998 ± 0.0005 ! RandomForest --> roc_auc_ovr: 0.9936 ± 0.0067
Analyze the results¶
In [4]:
Copied!
atom.results
atom.results
Out[4]:
| score_ht | time_ht | score_train | score_test | time_fit | score_bootstrap | time_bootstrap | time | |
|---|---|---|---|---|---|---|---|---|
| LR | 1.0 | 4.807741 | 0.9991 | 0.9977 | 0.362331 | 0.998413 | 1.770084 | 6.940156 |
| LDA | 1.0 | 0.084991 | 1.0000 | 1.0000 | 0.014012 | 0.999773 | 0.037034 | 0.136037 |
| RF | 1.0 | 3.960678 | 0.9993 | 1.0000 | 0.149179 | 0.993613 | 0.639135 | 4.748992 |
In [5]:
Copied!
# Show the score for some different metrics
atom.evaluate(["precision_macro", "recall_macro", "jaccard_weighted"])
# Show the score for some different metrics
atom.evaluate(["precision_macro", "recall_macro", "jaccard_weighted"])
Out[5]:
| precision_macro | recall_macro | jaccard_weighted | |
|---|---|---|---|
| LR | 0.9429 | 0.9484 | 0.8924 |
| LDA | 0.9667 | 0.9762 | 0.9457 |
| RF | 0.8799 | 0.8915 | 0.7968 |
In [6]:
Copied!
# Some plots allow you to choose the target class to look at
atom.rf.plot_probabilities(dataset="train", target=2)
# Some plots allow you to choose the target class to look at
atom.rf.plot_probabilities(dataset="train", target=2)
In [7]:
Copied!
atom.lda.plot_shap_heatmap(target=2, show=7)
atom.lda.plot_shap_heatmap(target=2, show=7)
