Multiclass classification¶

---

This example shows how to compare the performance of three models on a multiclass classification task.

Import the wine dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict wines into three groups (which cultivator it's from) using features based on the results of chemical analysis.

Load the data¶

In [1]:

# Import packages
from sklearn.datasets import load_wine
from atom import ATOMClassifier

# Import packages from sklearn.datasets import load_wine from atom import ATOMClassifier

In [2]:

# Load data
X, y = load_wine(return_X_y=True, as_frame=True)

# Let's have a look
X.head()

# Load data X, y = load_wine(return_X_y=True, as_frame=True) # Let's have a look X.head()

Out[2]:

	alcohol	malic_acid	ash	alcalinity_of_ash	magnesium	total_phenols	flavanoids	nonflavanoid_phenols	proanthocyanins	color_intensity	hue	od280/od315_of_diluted_wines	proline
0	14.23	1.71	2.43	15.6	127.0	2.80	3.06	0.28	2.29	5.64	1.04	3.92	1065.0
1	13.20	1.78	2.14	11.2	100.0	2.65	2.76	0.26	1.28	4.38	1.05	3.40	1050.0
2	13.16	2.36	2.67	18.6	101.0	2.80	3.24	0.30	2.81	5.68	1.03	3.17	1185.0
3	14.37	1.95	2.50	16.8	113.0	3.85	3.49	0.24	2.18	7.80	0.86	3.45	1480.0
4	13.24	2.59	2.87	21.0	118.0	2.80	2.69	0.39	1.82	4.32	1.04	2.93	735.0

Run the pipeline¶

In [3]:

atom = ATOMClassifier(X, y, n_jobs=-1, warnings=False, verbose=2, random_state=1)

# Fit the pipeline with the selected models
atom.run(
    models=["LR","LDA", "RF"],
    metric="roc_auc_ovr",
    n_calls=4,
    n_initial_points=3,
    bo_params={"base_estimator": "rf", "max_time": 100},
    n_bootstrap=5,
)

atom = ATOMClassifier(X, y, n_jobs=-1, warnings=False, verbose=2, random_state=1) # Fit the pipeline with the selected models atom.run( models=["LR","LDA", "RF"], metric="roc_auc_ovr", n_calls=4, n_initial_points=3, bo_params={"base_estimator": "rf", "max_time": 100}, n_bootstrap=5, )

<< ================== ATOM ================== >>
Algorithm task: multiclass classification.
Parallel processing with 16 cores.

Dataset stats ====================== >>
Shape: (178, 14)
Scaled: False
Outlier values: 10 (0.5%)
---------------------------------------
Train set size: 143
Test set size: 35
---------------------------------------
|    | dataset   | train    | test     |
|---:|:----------|:---------|:---------|
|  0 | 59 (1.2)  | 50 (1.4) | 9 (1.0)  |
|  1 | 71 (1.5)  | 58 (1.7) | 13 (1.4) |
|  2 | 48 (1.0)  | 35 (1.0) | 13 (1.4) |


Training ===================================== >>
Models: LR, LDA, RF
Metric: roc_auc_ovr


Running BO for Logistic Regression...
Initial point 1 ---------------------------------
Parameters --> {'penalty': 'l2', 'C': 46.003, 'solver': 'lbfgs', 'max_iter': 745}
Evaluation --> roc_auc_ovr: 1.0000  Best roc_auc_ovr: 1.0000
Time iteration: 6.715s   Total time: 6.720s
Initial point 2 ---------------------------------
Parameters --> {'penalty': 'none', 'solver': 'newton-cg', 'max_iter': 490}
Evaluation --> roc_auc_ovr: 1.0000  Best roc_auc_ovr: 1.0000
Time iteration: 6.565s   Total time: 13.517s
Initial point 3 ---------------------------------
Parameters --> {'penalty': 'l2', 'C': 0.037, 'solver': 'liblinear', 'max_iter': 352}
Evaluation --> roc_auc_ovr: 0.9993  Best roc_auc_ovr: 1.0000
Time iteration: 6.620s   Total time: 20.163s
Iteration 4 -------------------------------------
Parameters --> {'penalty': 'none', 'solver': 'newton-cg', 'max_iter': 378}
Evaluation --> roc_auc_ovr: 1.0000  Best roc_auc_ovr: 1.0000
Time iteration: 5.306s   Total time: 25.703s

Results for Logistic Regression:         
Bayesian Optimization ---------------------------
Best parameters --> {'penalty': 'l2', 'C': 46.003, 'solver': 'lbfgs', 'max_iter': 745}
Best evaluation --> roc_auc_ovr: 1.0
Time elapsed: 25.962s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 1.0
Test evaluation --> roc_auc_ovr: 0.9965
Time elapsed: 0.037s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.9942 ± 0.0026
Time elapsed: 0.100s
-------------------------------------------------
Total time: 26.099s


Running BO for Linear Discriminant Analysis...
Initial point 1 ---------------------------------
Parameters --> {'solver': 'eigen', 'shrinkage': 1.0}
Evaluation --> roc_auc_ovr: 0.8975  Best roc_auc_ovr: 0.8975
Time iteration: 0.016s   Total time: 0.031s
Initial point 2 ---------------------------------
Parameters --> {'solver': 'svd'}
Evaluation --> roc_auc_ovr: 1.0000  Best roc_auc_ovr: 1.0000
Time iteration: 0.030s   Total time: 0.092s
Initial point 3 ---------------------------------
Parameters --> {'solver': 'svd'}
Evaluation --> roc_auc_ovr: 1.0000  Best roc_auc_ovr: 1.0000
Time iteration: 0.031s   Total time: 0.140s
Iteration 4 -------------------------------------
Parameters --> {'solver': 'lsqr', 'shrinkage': 0.7}
Evaluation --> roc_auc_ovr: 0.8996  Best roc_auc_ovr: 1.0000
Time iteration: 0.034s   Total time: 0.383s

Results for Linear Discriminant Analysis:         
Bayesian Optimization ---------------------------
Best parameters --> {'solver': 'svd'}
Best evaluation --> roc_auc_ovr: 1.0
Time elapsed: 0.604s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 1.0
Test evaluation --> roc_auc_ovr: 1.0
Time elapsed: 0.001s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.9998 ± 0.0005
Time elapsed: 0.031s
-------------------------------------------------
Total time: 0.636s


Running BO for Random Forest...
Initial point 1 ---------------------------------
Parameters --> {'n_estimators': 245, 'criterion': 'entropy', 'max_depth': None, 'min_samples_split': 13, 'min_samples_leaf': 6, 'max_features': None, 'bootstrap': True, 'ccp_alpha': 0.007, 'max_samples': 0.6}
Evaluation --> roc_auc_ovr: 0.9921  Best roc_auc_ovr: 0.9921
Time iteration: 0.398s   Total time: 0.398s
Initial point 2 ---------------------------------
Parameters --> {'n_estimators': 400, 'criterion': 'entropy', 'max_depth': 8, 'min_samples_split': 7, 'min_samples_leaf': 19, 'max_features': 0.7, 'bootstrap': True, 'ccp_alpha': 0.008, 'max_samples': 0.7}
Evaluation --> roc_auc_ovr: 0.9927  Best roc_auc_ovr: 0.9927
Time iteration: 0.569s   Total time: 1.004s
Initial point 3 ---------------------------------
Parameters --> {'n_estimators': 78, 'criterion': 'gini', 'max_depth': 5, 'min_samples_split': 2, 'min_samples_leaf': 14, 'max_features': 0.8, 'bootstrap': False, 'ccp_alpha': 0.003}
Evaluation --> roc_auc_ovr: 0.9851  Best roc_auc_ovr: 0.9927
Time iteration: 0.125s   Total time: 1.160s
Iteration 4 -------------------------------------
Parameters --> {'n_estimators': 394, 'criterion': 'entropy', 'max_depth': 3, 'min_samples_split': 19, 'min_samples_leaf': 14, 'max_features': 0.8, 'bootstrap': False, 'ccp_alpha': 0.015}
Evaluation --> roc_auc_ovr: 0.9897  Best roc_auc_ovr: 0.9927
Time iteration: 0.476s   Total time: 1.965s

Results for Random Forest:         
Bayesian Optimization ---------------------------
Best parameters --> {'n_estimators': 400, 'criterion': 'entropy', 'max_depth': 8, 'min_samples_split': 7, 'min_samples_leaf': 19, 'max_features': 0.7, 'bootstrap': True, 'ccp_alpha': 0.008, 'max_samples': 0.7}
Best evaluation --> roc_auc_ovr: 0.9927
Time elapsed: 2.290s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 0.9997
Test evaluation --> roc_auc_ovr: 0.9802
Time elapsed: 0.575s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.974 ± 0.0074
Time elapsed: 2.658s
-------------------------------------------------
Total time: 5.524s


Final results ========================= >>
Duration: 32.259s
------------------------------------------
Logistic Regression          --> roc_auc_ovr: 0.9942 ± 0.0026
Linear Discriminant Analysis --> roc_auc_ovr: 0.9998 ± 0.0005 !
Random Forest                --> roc_auc_ovr: 0.974 ± 0.0074

Analyze the results¶

In [4]:

atom.results

atom.results

Out[4]:

	metric_bo	time_bo	metric_train	metric_test	time_fit	mean_bootstrap	std_bootstrap	time_bootstrap	time
LR	1.000000	25.962s	1.000000	0.996503	0.037s	0.994172	0.002553	0.100s	26.099s
LDA	1.000000	0.604s	1.000000	1.000000	0.001s	0.999767	0.000466	0.031s	0.636s
RF	0.992716	2.290s	0.999654	0.980186	0.575s	0.974022	0.007351	2.658s	5.524s

In [5]:

# Show the score for some different metrics
atom.evaluate(["precision_macro", "recall_macro", "jaccard_weighted"])

# Show the score for some different metrics atom.evaluate(["precision_macro", "recall_macro", "jaccard_weighted"])

Out[5]:

	jaccard_weighted	precision_macro	recall_macro
LR	0.893878	0.948718	0.948718
LDA	1.000000	1.000000	1.000000
RF	0.842857	0.919048	0.923077

In [6]:

# Some plots allow you to choose the target class to look at
atom.rf.plot_probabilities(dataset="train", target=2)

# Some plots allow you to choose the target class to look at atom.rf.plot_probabilities(dataset="train", target=2)

In [7]:

atom.lda.heatmap_plot(target=2, show=8, figsize=(16, 6))

atom.lda.heatmap_plot(target=2, show=8, figsize=(16, 6))