Example: Multiclass classification¶

This example shows how to compare the performance of three models on a multiclass classification task.

Import the wine dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict wines into three groups (which cultivator it's from) using features based on the results of chemical analysis.

Load the data¶

In [1]:

            
                Copied!
                
# Import packages
from sklearn.datasets import load_wine
from atom import ATOMClassifier
# Import packages
from sklearn.datasets import load_wine
from atom import ATOMClassifier

In [2]:

            
                Copied!
                
# Load data
X, y = load_wine(return_X_y=True, as_frame=True)

# Let's have a look
X.head()
# Load data
X, y = load_wine(return_X_y=True, as_frame=True)

# Let's have a look
X.head()

Out[2]:

	alcohol	malic_acid	ash	alcalinity_of_ash	magnesium	total_phenols	flavanoids	nonflavanoid_phenols	proanthocyanins	color_intensity	hue	od280/od315_of_diluted_wines	proline
0	14.23	1.71	2.43	15.6	127.0	2.80	3.06	0.28	2.29	5.64	1.04	3.92	1065.0
1	13.20	1.78	2.14	11.2	100.0	2.65	2.76	0.26	1.28	4.38	1.05	3.40	1050.0
2	13.16	2.36	2.67	18.6	101.0	2.80	3.24	0.30	2.81	5.68	1.03	3.17	1185.0
3	14.37	1.95	2.50	16.8	113.0	3.85	3.49	0.24	2.18	7.80	0.86	3.45	1480.0
4	13.24	2.59	2.87	21.0	118.0	2.80	2.69	0.39	1.82	4.32	1.04	2.93	735.0

Run the pipeline¶

In [3]:

            
                Copied!
                
                    
                    
                
                

        
atom = ATOMClassifier(X, y, n_jobs=-1, verbose=2, random_state=1)

# Fit the pipeline with the selected models
atom.run(
    models=["LR","LDA", "RF"],
    metric="roc_auc_ovr",
    n_trials=14,
    n_bootstrap=5,
)
atom = ATOMClassifier(X, y, n_jobs=-1, verbose=2, random_state=1)

# Fit the pipeline with the selected models
atom.run(
    models=["LR","LDA", "RF"],
    metric="roc_auc_ovr",
    n_trials=14,
    n_bootstrap=5,
)

<< ================== ATOM ================== >>
Algorithm task: multiclass classification.
Parallel processing with 16 cores.

Dataset stats ==================== >>
Shape: (178, 14)
Memory: 19.35 kB
Scaled: False
Outlier values: 12 (0.6%)
-------------------------------------
Train set size: 143
Test set size: 35
-------------------------------------
|   |    dataset |      train |       test |
| - | ---------- | ---------- | ---------- |
| 0 |   59 (1.2) |   47 (1.2) |   12 (1.3) |
| 1 |   71 (1.5) |   57 (1.5) |   14 (1.6) |
| 2 |   48 (1.0) |   39 (1.0) |    9 (1.0) |


Training ========================= >>
Models: LR, LDA, RF
Metric: roc_auc_ovr


Running hyperparameter tuning for LogisticRegression...
| trial | penalty |       C |  solver | max_iter | l1_ratio | roc_auc_ovr | best_roc_auc_ovr | time_trial | time_ht |    state |
| ----- | ------- | ------- | ------- | -------- | -------- | ----------- | ---------------- | ---------- | ------- | -------- |
| 0     |      l2 |  0.0054 |    saga |      480 |      --- |         1.0 |              1.0 |     0.023s |  0.023s | COMPLETE |
| 1     |      l2 |   0.122 |    saga |      380 |      --- |         1.0 |              1.0 |     0.022s |  0.045s | COMPLETE |
| 2     |      l2 |  0.0071 |     sag |      720 |      --- |         1.0 |              1.0 |     0.019s |  0.064s | COMPLETE |
| 3     |      l2 | 87.9641 | libli.. |      920 |      --- |      0.9984 |              1.0 |     0.014s |  0.078s | COMPLETE |
| 4     |    none |     --- |     sag |      630 |      --- |         1.0 |              1.0 |     0.038s |  0.116s | COMPLETE |
| 5     |      l2 |  0.0018 |     sag |      920 |      --- |         1.0 |              1.0 |     0.023s |  0.139s | COMPLETE |
| 6     |      l2 | 43.4053 |     sag |      780 |      --- |         1.0 |              1.0 |     0.040s |  0.179s | COMPLETE |
| 7     |      l2 |  2.0759 | libli.. |      470 |      --- |         1.0 |              1.0 |     0.015s |  0.194s | COMPLETE |
| 8     |      l2 |   0.043 |     sag |      110 |      --- |         1.0 |              1.0 |     0.021s |  0.215s | COMPLETE |
| 9     |      l2 | 46.0233 |    saga |      740 |      --- |         1.0 |              1.0 |     0.035s |  0.250s | COMPLETE |
| 10    |      l2 |  1.2173 |   lbfgs |      280 |      --- |         1.0 |              1.0 |     0.815s |  1.065s | COMPLETE |
| 11    |      l2 |  0.0983 |    saga |      450 |      --- |         1.0 |              1.0 |     0.030s |  1.095s | COMPLETE |
| 12    | elast.. |   0.001 |    saga |      330 |      0.6 |         0.5 |              1.0 |     0.027s |  1.122s | COMPLETE |
| 13    |      l2 |  0.4352 | newto.. |      300 |      --- |         1.0 |              1.0 |     0.365s |  1.487s | COMPLETE |
Hyperparameter tuning ---------------------------
Best trial --> 0
Best parameters:
 --> penalty: l2
 --> C: 0.0054
 --> solver: saga
 --> max_iter: 480
Best evaluation --> roc_auc_ovr: 1.0
Time elapsed: 1.487s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 0.999
Test evaluation --> roc_auc_ovr: 0.9977
Time elapsed: 0.029s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.9982 ± 0.0006
Time elapsed: 0.081s
-------------------------------------------------
Total time: 1.597s


Running hyperparameter tuning for LinearDiscriminantAnalysis...
| trial |  solver | shrinkage | roc_auc_ovr | best_roc_auc_ovr | time_trial | time_ht |    state |
| ----- | ------- | --------- | ----------- | ---------------- | ---------- | ------- | -------- |
| 0     |    lsqr |       0.9 |      0.9221 |           0.9221 |     0.010s |  0.010s | COMPLETE |
| 1     |   eigen |       1.0 |      0.9121 |           0.9221 |     0.006s |  0.016s | COMPLETE |
| 2     |   eigen |       1.0 |      0.9121 |           0.9221 |     0.001s |  0.017s | COMPLETE |
| 3     |    lsqr |       0.7 |      0.8638 |           0.9221 |     0.006s |  0.023s | COMPLETE |
| 4     |   eigen |       0.7 |      0.9019 |           0.9221 |     0.009s |  0.032s | COMPLETE |
| 5     |    lsqr |      auto |         1.0 |              1.0 |     0.008s |  0.040s | COMPLETE |
| 6     |   eigen |       1.0 |      0.9121 |              1.0 |     0.001s |  0.041s | COMPLETE |
| 7     |    lsqr |       1.0 |      0.9445 |              1.0 |     0.006s |  0.047s | COMPLETE |
| 8     |     svd |       --- |         1.0 |              1.0 |     0.007s |  0.054s | COMPLETE |
| 9     |     svd |       --- |         1.0 |              1.0 |     0.001s |  0.055s | COMPLETE |
| 10    |    lsqr |      auto |         1.0 |              1.0 |     0.002s |  0.057s | COMPLETE |
| 11    |     svd |       --- |         1.0 |              1.0 |     0.002s |  0.059s | COMPLETE |
| 12    |     svd |       --- |         1.0 |              1.0 |     0.002s |  0.061s | COMPLETE |
| 13    |     svd |       --- |         1.0 |              1.0 |     0.002s |  0.063s | COMPLETE |
Hyperparameter tuning ---------------------------
Best trial --> 5
Best parameters:
 --> solver: lsqr
 --> shrinkage: auto
Best evaluation --> roc_auc_ovr: 1.0
Time elapsed: 0.063s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 1.0
Test evaluation --> roc_auc_ovr: 1.0
Time elapsed: 0.015s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.9998 ± 0.0005
Time elapsed: 0.037s
-------------------------------------------------
Total time: 0.115s


Running hyperparameter tuning for RandomForest...
| trial | n_estimators | criterion | max_depth | min_samples_split | min_samples_leaf | max_features | bootstrap | max_samples | ccp_alpha | roc_auc_ovr | best_roc_auc_ovr | time_trial | time_ht |    state |
| ----- | ------------ | --------- | --------- | ----------------- | ---------------- | ------------ | --------- | ----------- | --------- | ----------- | ---------------- | ---------- | ------- | -------- |
| 0     |          210 |      gini |        10 |                17 |               20 |          0.5 |     False |         --- |       0.0 |      0.9803 |           0.9803 |     0.214s |  0.214s | COMPLETE |
| 1     |          380 |      gini |         4 |                15 |                3 |          0.9 |     False |         --- |      0.01 |      0.9816 |           0.9816 |     0.390s |  0.605s | COMPLETE |
| 2     |          380 |   entropy |         6 |                 2 |               13 |          0.9 |     False |         --- |      0.03 |      0.9944 |           0.9944 |     0.388s |  0.993s | COMPLETE |
| 3     |          470 |      gini |        11 |                 9 |               18 |         None |      True |         0.6 |     0.025 |      0.9569 |           0.9944 |     0.583s |  1.575s | COMPLETE |
| 4     |          100 |   entropy |        12 |                14 |                6 |          0.9 |     False |         --- |     0.035 |         1.0 |              1.0 |     0.116s |  1.692s | COMPLETE |
| 5     |          470 |   entropy |        13 |                11 |                1 |         None |      True |         0.6 |      0.01 |         1.0 |              1.0 |     0.594s |  2.285s | COMPLETE |
| 6     |          250 |      gini |        14 |                13 |               17 |          0.7 |      True |        None |      0.02 |         1.0 |              1.0 |     0.325s |  2.610s | COMPLETE |
| 7     |          220 |      gini |         5 |                10 |                7 |          0.5 |      True |         0.9 |     0.035 |      0.9981 |              1.0 |     0.295s |  2.906s | COMPLETE |
| 8     |          130 |   entropy |         4 |                 6 |               11 |          0.9 |     False |         --- |      0.03 |         1.0 |              1.0 |     0.148s |  3.054s | COMPLETE |
| 9     |          370 |      gini |        12 |                 2 |                4 |          0.5 |     False |         --- |      0.02 |      0.9916 |              1.0 |     0.377s |  3.431s | COMPLETE |
| 10    |           10 |   entropy |      None |                20 |                7 |         log2 |     False |         --- |     0.035 |         1.0 |              1.0 |     0.037s |  3.468s | COMPLETE |
| 11    |           70 |   entropy |        13 |                12 |                1 |         None |      True |         0.5 |      0.01 |      0.9928 |              1.0 |     0.116s |  3.584s | COMPLETE |
| 12    |          500 |   entropy |         9 |                 7 |                7 |          0.6 |      True |         0.6 |      0.01 |         1.0 |              1.0 |     0.642s |  4.226s | COMPLETE |
| 13    |          140 |   entropy |        16 |                16 |                1 |          0.8 |      True |         0.5 |       0.0 |         1.0 |              1.0 |     0.190s |  4.416s | COMPLETE |
Hyperparameter tuning ---------------------------
Best trial --> 4
Best parameters:
 --> n_estimators: 100
 --> criterion: entropy
 --> max_depth: 12
 --> min_samples_split: 14
 --> min_samples_leaf: 6
 --> max_features: 0.9
 --> bootstrap: False
 --> ccp_alpha: 0.035
Best evaluation --> roc_auc_ovr: 1.0
Time elapsed: 4.416s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 0.9993
Test evaluation --> roc_auc_ovr: 1.0
Time elapsed: 0.132s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.9936 ± 0.0067
Time elapsed: 0.555s
-------------------------------------------------
Total time: 5.103s


Final results ==================== >>
Total time: 7.139s
-------------------------------------
LogisticRegression         --> roc_auc_ovr: 0.9982 ± 0.0006
LinearDiscriminantAnalysis --> roc_auc_ovr: 0.9998 ± 0.0005 !
RandomForest               --> roc_auc_ovr: 0.9936 ± 0.0067

Analyze the results¶

In [4]:

            
                Copied!
                
atom.results
atom.results

Out[4]:

	score_ht	time_ht	score_train	score_test	time_fit	score_bootstrap	time_bootstrap	time
LR	1.0	1.487354	0.9990	0.9977	0.029027	0.998186	0.081073	1.597454
LDA	1.0	0.063054	1.0000	1.0000	0.015013	0.999773	0.037034	0.115101
RF	1.0	4.416013	0.9993	1.0000	0.132121	0.993613	0.554504	5.102638

In [5]:

            
                Copied!
                
# Show the score for some different metrics
atom.evaluate(["precision_macro", "recall_macro", "jaccard_weighted"])
# Show the score for some different metrics
atom.evaluate(["precision_macro", "recall_macro", "jaccard_weighted"])

Out[5]:

	precision_macro	recall_macro	jaccard_weighted
LR	0.9429	0.9484	0.8924
LDA	0.9667	0.9762	0.9457
RF	0.8799	0.8915	0.7968

In [6]:

            
                Copied!
                
# Some plots allow you to choose the target class to look at
atom.rf.plot_probabilities(dataset="train", target=2)
# Some plots allow you to choose the target class to look at
atom.rf.plot_probabilities(dataset="train", target=2)

In [9]:

            
                Copied!
                
atom.lda.plot_shap_heatmap(target=2, show=7)
atom.lda.plot_shap_heatmap(target=2, show=7)