Multiclass classification¶

---

This example shows how to compare the performance of three models on a multiclass classification task.

Import the wine dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict wines into three groups (which cultivator it's from) using features based on the results of chemical analysis.

Load the data¶

In [1]:

# Import packages
from sklearn.datasets import load_wine
from atom import ATOMClassifier

# Import packages from sklearn.datasets import load_wine from atom import ATOMClassifier

In [2]:

# Load data
X, y = load_wine(return_X_y=True, as_frame=True)

# Let's have a look
X.head()

# Load data X, y = load_wine(return_X_y=True, as_frame=True) # Let's have a look X.head()

Out[2]:

	alcohol	malic_acid	ash	alcalinity_of_ash	magnesium	total_phenols	flavanoids	nonflavanoid_phenols	proanthocyanins	color_intensity	hue	od280/od315_of_diluted_wines	proline
0	14.23	1.71	2.43	15.6	127.0	2.80	3.06	0.28	2.29	5.64	1.04	3.92	1065.0
1	13.20	1.78	2.14	11.2	100.0	2.65	2.76	0.26	1.28	4.38	1.05	3.40	1050.0
2	13.16	2.36	2.67	18.6	101.0	2.80	3.24	0.30	2.81	5.68	1.03	3.17	1185.0
3	14.37	1.95	2.50	16.8	113.0	3.85	3.49	0.24	2.18	7.80	0.86	3.45	1480.0
4	13.24	2.59	2.87	21.0	118.0	2.80	2.69	0.39	1.82	4.32	1.04	2.93	735.0

Run the pipeline¶

In [3]:

atom = ATOMClassifier(X, y, n_jobs=-1, warnings=False, verbose=2, random_state=1)

# Fit the pipeline with the selected models
atom.run(
    models=["LR","LDA", "RF"],
    metric="roc_auc_ovr",
    n_calls=4,
    n_initial_points=3,
    bo_params={"base_estimator": "rf", "max_time": 100},
    n_bootstrap=5,
)

atom = ATOMClassifier(X, y, n_jobs=-1, warnings=False, verbose=2, random_state=1) # Fit the pipeline with the selected models atom.run( models=["LR","LDA", "RF"], metric="roc_auc_ovr", n_calls=4, n_initial_points=3, bo_params={"base_estimator": "rf", "max_time": 100}, n_bootstrap=5, )

<< ================== ATOM ================== >>
Algorithm task: multiclass classification.
Parallel processing with 16 cores.

Dataset stats ====================== >>
Shape: (178, 14)
Scaled: False
Outlier values: 10 (0.5%)
---------------------------------------
Train set size: 143
Test set size: 35
---------------------------------------
|    | dataset   | train    | test     |
|---:|:----------|:---------|:---------|
|  0 | 59 (1.2)  | 50 (1.4) | 9 (1.0)  |
|  1 | 71 (1.5)  | 58 (1.7) | 13 (1.4) |
|  2 | 48 (1.0)  | 35 (1.0) | 13 (1.4) |


Training ===================================== >>
Models: LR, LDA, RF
Metric: roc_auc_ovr


Running BO for Logistic Regression...
Initial point 1 ---------------------------------
Parameters --> {'penalty': 'l2', 'C': 46.003, 'solver': 'lbfgs', 'max_iter': 745}
Evaluation --> roc_auc_ovr: 1.0000  Best roc_auc_ovr: 1.0000
Time iteration: 6.863s   Total time: 6.868s
Initial point 2 ---------------------------------
Parameters --> {'penalty': 'none', 'solver': 'newton-cg', 'max_iter': 490}
Evaluation --> roc_auc_ovr: 1.0000  Best roc_auc_ovr: 1.0000
Time iteration: 6.336s   Total time: 13.391s
Initial point 3 ---------------------------------
Parameters --> {'penalty': 'l2', 'C': 0.037, 'solver': 'liblinear', 'max_iter': 352}
Evaluation --> roc_auc_ovr: 0.9993  Best roc_auc_ovr: 1.0000
Time iteration: 6.379s   Total time: 19.803s
Iteration 4 -------------------------------------
Parameters --> {'penalty': 'none', 'solver': 'newton-cg', 'max_iter': 378}
Evaluation --> roc_auc_ovr: 1.0000  Best roc_auc_ovr: 1.0000
Time iteration: 5.243s   Total time: 25.296s

Results for Logistic Regression:         
Bayesian Optimization ---------------------------
Best parameters --> {'penalty': 'l2', 'C': 46.003, 'solver': 'lbfgs', 'max_iter': 745}
Best evaluation --> roc_auc_ovr: 1.0
Time elapsed: 25.535s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 1.0
Test evaluation --> roc_auc_ovr: 0.9965
Time elapsed: 0.029s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.9942 ± 0.0026
Time elapsed: 0.103s
-------------------------------------------------
Total time: 25.668s


Running BO for Linear Discriminant Analysis...
Initial point 1 ---------------------------------
Parameters --> {'solver': 'eigen', 'shrinkage': 1.0}
Evaluation --> roc_auc_ovr: 0.8975  Best roc_auc_ovr: 0.8975
Time iteration: 0.023s   Total time: 0.025s
Initial point 2 ---------------------------------
Parameters --> {'solver': 'svd'}
Evaluation --> roc_auc_ovr: 1.0000  Best roc_auc_ovr: 1.0000
Time iteration: 0.024s   Total time: 0.072s
Initial point 3 ---------------------------------
Parameters --> {'solver': 'svd'}
Evaluation --> roc_auc_ovr: 1.0000  Best roc_auc_ovr: 1.0000
Time iteration: 0.024s   Total time: 0.119s
Iteration 4 -------------------------------------
Parameters --> {'solver': 'lsqr', 'shrinkage': 0.7}
Evaluation --> roc_auc_ovr: 0.8996  Best roc_auc_ovr: 1.0000
Time iteration: 0.024s   Total time: 0.349s

Results for Linear Discriminant Analysis:         
Bayesian Optimization ---------------------------
Best parameters --> {'solver': 'svd'}
Best evaluation --> roc_auc_ovr: 1.0
Time elapsed: 0.551s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 1.0
Test evaluation --> roc_auc_ovr: 1.0
Time elapsed: 0.010s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.9998 ± 0.0005
Time elapsed: 0.029s
-------------------------------------------------
Total time: 0.591s


Running BO for Random Forest...
Initial point 1 ---------------------------------
Parameters --> {'n_estimators': 245, 'criterion': 'entropy', 'max_depth': None, 'min_samples_split': 13, 'min_samples_leaf': 6, 'max_features': None, 'bootstrap': True, 'ccp_alpha': 0.007, 'max_samples': 0.6}
Evaluation --> roc_auc_ovr: 0.9921  Best roc_auc_ovr: 0.9921
Time iteration: 0.381s   Total time: 0.387s
Initial point 2 ---------------------------------
Parameters --> {'n_estimators': 400, 'criterion': 'entropy', 'max_depth': 8, 'min_samples_split': 7, 'min_samples_leaf': 19, 'max_features': 0.7, 'bootstrap': True, 'ccp_alpha': 0.008, 'max_samples': 0.7}
Evaluation --> roc_auc_ovr: 0.9927  Best roc_auc_ovr: 0.9927
Time iteration: 0.622s   Total time: 1.044s
Initial point 3 ---------------------------------
Parameters --> {'n_estimators': 78, 'criterion': 'gini', 'max_depth': 5, 'min_samples_split': 2, 'min_samples_leaf': 14, 'max_features': 0.8, 'bootstrap': False, 'ccp_alpha': 0.003}
Evaluation --> roc_auc_ovr: 0.9851  Best roc_auc_ovr: 0.9927
Time iteration: 0.133s   Total time: 1.207s
Iteration 4 -------------------------------------
Parameters --> {'n_estimators': 394, 'criterion': 'entropy', 'max_depth': 3, 'min_samples_split': 19, 'min_samples_leaf': 14, 'max_features': 0.8, 'bootstrap': False, 'ccp_alpha': 0.015}
Evaluation --> roc_auc_ovr: 0.9897  Best roc_auc_ovr: 0.9927
Time iteration: 0.502s   Total time: 2.014s

Results for Random Forest:         
Bayesian Optimization ---------------------------
Best parameters --> {'n_estimators': 400, 'criterion': 'entropy', 'max_depth': 8, 'min_samples_split': 7, 'min_samples_leaf': 19, 'max_features': 0.7, 'bootstrap': True, 'ccp_alpha': 0.008, 'max_samples': 0.7}
Best evaluation --> roc_auc_ovr: 0.9927
Time elapsed: 2.318s
Fit ---------------------------------------------
Train evaluation --> roc_auc_ovr: 0.9997
Test evaluation --> roc_auc_ovr: 0.9802
Time elapsed: 0.594s
Bootstrap ---------------------------------------
Evaluation --> roc_auc_ovr: 0.974 ± 0.0074
Time elapsed: 2.648s
-------------------------------------------------
Total time: 5.561s


Final results ========================= >>
Duration: 31.821s
------------------------------------------
Logistic Regression          --> roc_auc_ovr: 0.9942 ± 0.0026
Linear Discriminant Analysis --> roc_auc_ovr: 0.9998 ± 0.0005 !
Random Forest                --> roc_auc_ovr: 0.974 ± 0.0074

Analyze the results¶

In [4]:

atom.results

atom.results

Out[4]:

	metric_bo	time_bo	metric_train	metric_test	time_fit	mean_bootstrap	std_bootstrap	time_bootstrap	time
LR	1.000000	25.535s	1.000000	0.996503	0.029s	0.994172	0.002553	0.103s	25.668s
LDA	1.000000	0.551s	1.000000	1.000000	0.010s	0.999767	0.000466	0.029s	0.591s
RF	0.992716	2.318s	0.999654	0.980186	0.594s	0.974022	0.007351	2.648s	5.561s

In [10]:

# Show the score for some different metrics
atom.scoring(["precision_macro", "recall_macro", "jaccard_weighted"])

# Show the score for some different metrics atom.scoring(["precision_macro", "recall_macro", "jaccard_weighted"])

Out[10]:

	jaccard_weighted	precision_macro	recall_macro
LR	0.893878	0.948718	0.948718
LDA	1.000000	1.000000	1.000000
RF	0.842857	0.919048	0.923077

In [24]:

# Some plots allow you to choose the target class to look at
atom.rf.plot_probabilities(dataset="train", target=2)

# Some plots allow you to choose the target class to look at atom.rf.plot_probabilities(dataset="train", target=2)

In [17]:

atom.lda.heatmap_plot(target=2, show=8, figsize=(16, 6))

atom.lda.heatmap_plot(target=2, show=8, figsize=(16, 6))