Multiclass classification¶

This example shows how to compare the performance of three models on a multiclass classification task.

Import the wine dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict wines into three groups (which cultivator it's from) using features based on the results of chemical analysis.

Load the data¶

In [1]:

            
                Copied!
                
# Import packages
from sklearn.datasets import load_wine
from atom import ATOMClassifier
# Import packages
from sklearn.datasets import load_wine
from atom import ATOMClassifier

In [2]:

            
                Copied!
                
# Load data
X, y = load_wine(return_X_y=True, as_frame=True)

# Let's have a look
X.head()
# Load data
X, y = load_wine(return_X_y=True, as_frame=True)

# Let's have a look
X.head()

Out[2]:

	alcohol	malic_acid	ash	alcalinity_of_ash	magnesium	total_phenols	flavanoids	nonflavanoid_phenols	proanthocyanins	color_intensity	hue	od280/od315_of_diluted_wines	proline
0	14.23	1.71	2.43	15.6	127.0	2.80	3.06	0.28	2.29	5.64	1.04	3.92	1065.0
1	13.20	1.78	2.14	11.2	100.0	2.65	2.76	0.26	1.28	4.38	1.05	3.40	1050.0
2	13.16	2.36	2.67	18.6	101.0	2.80	3.24	0.30	2.81	5.68	1.03	3.17	1185.0
3	14.37	1.95	2.50	16.8	113.0	3.85	3.49	0.24	2.18	7.80	0.86	3.45	1480.0
4	13.24	2.59	2.87	21.0	118.0	2.80	2.69	0.39	1.82	4.32	1.04	2.93	735.0

Run the pipeline¶

In [3]:

            
                Copied!
                
                    
                    
                
                

        
atom = ATOMClassifier(X, y, n_jobs=-1, warnings=False, verbose=2, random_state=1)

# Fit the pipeline with the selected models
atom.run(
    models=["LR","LDA", "RF"],
    metric="roc_auc_ovr",
    n_calls=4,
    n_initial_points=3,
    bo_params={"base_estimator": "rf", "max_time": 100},
    n_bootstrap=5,
)
atom = ATOMClassifier(X, y, n_jobs=-1, warnings=False, verbose=2, random_state=1)

# Fit the pipeline with the selected models
atom.run(
    models=["LR","LDA", "RF"],
    metric="roc_auc_ovr",
    n_calls=4,
    n_initial_points=3,
    bo_params={"base_estimator": "rf", "max_time": 100},
    n_bootstrap=5,
)

<< ================== ATOM ================== >>
Algorithm task: multiclass classification.
Parallel processing with 16 cores.

Dataset stats ==================== >>
Shape: (178, 14)
Scaled: False
Outlier values: 10 (0.5%)
-------------------------------------
Train set size: 143
Test set size: 35
-------------------------------------
|    |    dataset |      train |       test |
| -- | ---------- | ---------- | ---------- |
| 0  |   59 (1.2) |   50 (1.4) |    9 (1.0) |
| 1  |   71 (1.5) |   58 (1.7) |   13 (1.4) |
| 2  |   48 (1.0) |   35 (1.0) |   13 (1.4) |


Training ========================= >>
Models: LR, LDA, RF
Metric: roc_auc_ovr


Running BO for Logistic Regression...
| call             | penalty |       C |  solver | max_iter | l1_ratio | roc_auc_ovr | best_roc_auc_ovr |    time | total_time |
| ---------------- | ------- | ------- | ------- | -------- | -------- | ----------- | ---------------- | ------- | ---------- |
| Initial point 1  |      l2 |  46.003 |   lbfgs |      745 |      --- |         1.0 |              1.0 |  1.085s |     1.092s |
| Initial point 2  |    none |     --- | newto.. |      490 |      --- |         1.0 |              1.0 |  0.496s |     1.777s |
| Initial point 3  |      l2 |   0.037 | libli.. |      352 |      --- |         1.0 |              1.0 |  0.017s |     1.858s |
| Iteration 4      |    none |     --- | newto.. |      378 |      --- |         1.0 |              1.0 |  0.486s |     2.688s |

Results for Logistic Regression:         
Bayesian Optimization ---------------------------
Best call --> Initial point 1
Best parameters --> {'penalty': 'l2', 'C': 46.003, 'solver': 'lbfgs', 'max_iter': 745}
Best evaluation --> roc_auc_ovr: 1.0
Time elapsed: 3.021s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 1.0
Test evaluation --> roc_auc_ovr: 0.9965
Time elapsed: 0.418s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.9953 ± 0.0007
Time elapsed: 2.324s
-------------------------------------------------
Total time: 5.765s


Running BO for Linear Discriminant Analysis...
| call             |  solver | shrinkage | roc_auc_ovr | best_roc_auc_ovr |    time | total_time |
| ---------------- | ------- | --------- | ----------- | ---------------- | ------- | ---------- |
| Initial point 1  |   eigen |       1.0 |      0.9125 |           0.9125 |  0.010s |     0.013s |
| Initial point 2  |     svd |       --- |         1.0 |              1.0 |  0.009s |     0.085s |
| Initial point 3  |     svd |       --- |         1.0 |              1.0 |  0.000s |     0.151s |
| Iteration 4      |    lsqr |       0.7 |      0.8761 |              1.0 |  0.009s |     0.457s |

Results for Linear Discriminant Analysis:         
Bayesian Optimization ---------------------------
Best call --> Initial point 2
Best parameters --> {'solver': 'svd'}
Best evaluation --> roc_auc_ovr: 1.0
Time elapsed: 0.737s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 1.0
Test evaluation --> roc_auc_ovr: 1.0
Time elapsed: 0.009s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.9966 ± 0.0024
Time elapsed: 0.028s
-------------------------------------------------
Total time: 0.775s


Running BO for Random Forest...
| call             | n_estimators | criterion | max_depth | min_samples_split | min_samples_leaf | max_features | bootstrap | ccp_alpha | max_samples | roc_auc_ovr | best_roc_auc_ovr |    time | total_time |
| ---------------- | ------------ | --------- | --------- | ----------------- | ---------------- | ------------ | --------- | --------- | ----------- | ----------- | ---------------- | ------- | ---------- |
| Initial point 1  |          245 |   entropy |      None |                13 |                6 |         None |      True |     0.007 |         0.6 |      0.9928 |           0.9928 |  0.332s |     0.339s |
| Initial point 2  |          400 |   entropy |         8 |                 7 |               19 |          0.7 |      True |     0.008 |         0.7 |      0.9886 |           0.9928 |  0.536s |     0.943s |
| Initial point 3  |           78 |      gini |         5 |                 2 |               14 |          0.8 |     False |     0.003 |         --- |      0.9916 |           0.9928 |  0.105s |     1.118s |
| Iteration 4      |          394 |   entropy |         3 |                19 |               14 |          0.8 |     False |     0.015 |         --- |      0.9841 |           0.9928 |  0.446s |     2.039s |

Results for Random Forest:         
Bayesian Optimization ---------------------------
Best call --> Initial point 1
Best parameters --> {'n_estimators': 245, 'criterion': 'entropy', 'max_depth': None, 'min_samples_split': 13, 'min_samples_leaf': 6, 'max_features': None, 'bootstrap': True, 'ccp_alpha': 0.007, 'max_samples': 0.6}
Best evaluation --> roc_auc_ovr: 0.9928
Time elapsed: 2.452s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 0.9997
Test evaluation --> roc_auc_ovr: 0.9709
Time elapsed: 0.369s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.9462 ± 0.025
Time elapsed: 1.778s
-------------------------------------------------
Total time: 4.601s


Final results ==================== >>
Duration: 11.141s
-------------------------------------
Logistic Regression          --> roc_auc_ovr: 0.9953 ± 0.0007
Linear Discriminant Analysis --> roc_auc_ovr: 0.9966 ± 0.0024 !
Random Forest                --> roc_auc_ovr: 0.9462 ± 0.025

Analyze the results¶

In [4]:

            
                Copied!
                
atom.results
atom.results

Out[4]:

	metric_bo	time_bo	metric_train	metric_test	time_fit	mean_bootstrap	std_bootstrap	time_bootstrap	time
LR	1.000000	3.021s	1.000000	0.996503	0.418s	0.995338	0.000737	2.324s	5.765s
LDA	1.000000	0.737s	1.000000	1.000000	0.009s	0.996633	0.002441	0.028s	0.775s
RF	0.992813	2.452s	0.999658	0.970862	0.369s	0.946180	0.024957	1.778s	4.601s

In [5]:

            
                Copied!
                
# Show the score for some different metrics
atom.evaluate(["precision_macro", "recall_macro", "jaccard_weighted"])
# Show the score for some different metrics
atom.evaluate(["precision_macro", "recall_macro", "jaccard_weighted"])

Out[5]:

	precision_macro	recall_macro	jaccard_weighted
LR	0.948718	0.948718	0.893878
LDA	1.000000	1.000000	1.000000
RF	0.834188	0.846154	0.711190

In [6]:

            
                Copied!
                
# Some plots allow you to choose the target class to look at
atom.rf.plot_probabilities(dataset="train", target=2)
# Some plots allow you to choose the target class to look at
atom.rf.plot_probabilities(dataset="train", target=2)

In [7]:

            
                Copied!
                
atom.lda.heatmap_plot(target=2, show=8, figsize=(16, 6))
atom.lda.heatmap_plot(target=2, show=8, figsize=(16, 6))