Successive halving¶
This example shows how to compare multiple tree-based models using successive halving.
Import the california housing dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict house prices.
Load the data¶
In [1]:
Copied!
from sklearn.datasets import fetch_california_housing
from atom import ATOMRegressor
from sklearn.datasets import fetch_california_housing
from atom import ATOMRegressor
In [2]:
Copied!
# Load the data
X, y = fetch_california_housing(return_X_y=True)
# Load the data
X, y = fetch_california_housing(return_X_y=True)
Run the pipeline¶
In [3]:
Copied!
atom = ATOMRegressor(X, y, verbose=2, random_state=1)
atom = ATOMRegressor(X, y, verbose=2, random_state=1)
<< ================== ATOM ================== >> Algorithm task: regression. Dataset stats ==================== >> Shape: (20640, 9) Memory: 1.49 MB Scaled: False Outlier values: 799 (0.5%) ------------------------------------- Train set size: 16512 Test set size: 4128
In [4]:
Copied!
# Compare tree-based models via successive halving
atom.successive_halving(
models=["Tree", "Bag", "ET", "RF", "LGB", "CatB"],
metric="mae",
n_bootstrap=5,
)
# Compare tree-based models via successive halving
atom.successive_halving(
models=["Tree", "Bag", "ET", "RF", "LGB", "CatB"],
metric="mae",
n_bootstrap=5,
)
Run: 0 ================================ >> Models: Tree6, Bag6, ET6, RF6, LGB6, CatB6 Size of training set: 16512 (17%) Size of test set: 4128 Training ========================= >> Metric: neg_mean_absolute_error Results for Decision Tree: Fit --------------------------------------------- Train evaluation --> neg_mean_absolute_error: -0.0 Test evaluation --> neg_mean_absolute_error: -0.5598 Time elapsed: 0.031s Bootstrap --------------------------------------- Evaluation --> neg_mean_absolute_error: -0.546 ± 0.0081 Time elapsed: 0.250s ------------------------------------------------- Total time: 0.281s Results for Bagging: Fit --------------------------------------------- Train evaluation --> neg_mean_absolute_error: -0.1649 Test evaluation --> neg_mean_absolute_error: -0.4074 Time elapsed: 0.125s Bootstrap --------------------------------------- Evaluation --> neg_mean_absolute_error: -0.4145 ± 0.0081 Time elapsed: 0.844s ------------------------------------------------- Total time: 0.969s Results for Extra-Trees: Fit --------------------------------------------- Train evaluation --> neg_mean_absolute_error: -0.0 Test evaluation --> neg_mean_absolute_error: -0.3749 Time elapsed: 0.688s Bootstrap --------------------------------------- Evaluation --> neg_mean_absolute_error: -0.3834 ± 0.0008 Time elapsed: 2.688s ------------------------------------------------- Total time: 3.375s Results for Random Forest: Fit --------------------------------------------- Train evaluation --> neg_mean_absolute_error: -0.1457 Test evaluation --> neg_mean_absolute_error: -0.3905 Time elapsed: 1.094s Bootstrap --------------------------------------- Evaluation --> neg_mean_absolute_error: -0.3969 ± 0.0046 Time elapsed: 5.314s ------------------------------------------------- Total time: 6.408s Results for LightGBM: Fit --------------------------------------------- Train evaluation --> neg_mean_absolute_error: -0.1945 Test evaluation --> neg_mean_absolute_error: -0.3457 Time elapsed: 0.281s Bootstrap --------------------------------------- Evaluation --> neg_mean_absolute_error: -0.3549 ± 0.0024 Time elapsed: 0.719s ------------------------------------------------- Total time: 1.000s Results for CatBoost: Fit --------------------------------------------- Train evaluation --> neg_mean_absolute_error: -0.1637 Test evaluation --> neg_mean_absolute_error: -0.3319 Time elapsed: 3.532s Bootstrap --------------------------------------- Evaluation --> neg_mean_absolute_error: -0.341 ± 0.0008 Time elapsed: 15.904s ------------------------------------------------- Total time: 19.436s Final results ==================== >> Duration: 31.469s ------------------------------------- Decision Tree --> neg_mean_absolute_error: -0.546 ± 0.0081 ~ Bagging --> neg_mean_absolute_error: -0.4145 ± 0.0081 ~ Extra-Trees --> neg_mean_absolute_error: -0.3834 ± 0.0008 ~ Random Forest --> neg_mean_absolute_error: -0.3969 ± 0.0046 ~ LightGBM --> neg_mean_absolute_error: -0.3549 ± 0.0024 ~ CatBoost --> neg_mean_absolute_error: -0.341 ± 0.0008 ~ ! Run: 1 ================================ >> Models: ET3, LGB3, CatB3 Size of training set: 16512 (33%) Size of test set: 4128 Training ========================= >> Metric: neg_mean_absolute_error Results for Extra-Trees: Fit --------------------------------------------- Train evaluation --> neg_mean_absolute_error: -0.0 Test evaluation --> neg_mean_absolute_error: -0.3575 Time elapsed: 1.125s Bootstrap --------------------------------------- Evaluation --> neg_mean_absolute_error: -0.3685 ± 0.0033 Time elapsed: 4.813s ------------------------------------------------- Total time: 5.938s Results for LightGBM: Fit --------------------------------------------- Train evaluation --> neg_mean_absolute_error: -0.231 Test evaluation --> neg_mean_absolute_error: -0.3278 Time elapsed: 0.188s Bootstrap --------------------------------------- Evaluation --> neg_mean_absolute_error: -0.3359 ± 0.0032 Time elapsed: 0.859s ------------------------------------------------- Total time: 1.047s Results for CatBoost: Fit --------------------------------------------- Train evaluation --> neg_mean_absolute_error: -0.1954 Test evaluation --> neg_mean_absolute_error: -0.3138 Time elapsed: 3.661s Bootstrap --------------------------------------- Evaluation --> neg_mean_absolute_error: -0.3221 ± 0.0019 Time elapsed: 17.507s ------------------------------------------------- Total time: 21.168s Final results ==================== >> Duration: 28.169s ------------------------------------- Extra-Trees --> neg_mean_absolute_error: -0.3685 ± 0.0033 ~ LightGBM --> neg_mean_absolute_error: -0.3359 ± 0.0032 ~ CatBoost --> neg_mean_absolute_error: -0.3221 ± 0.0019 ~ ! Run: 2 ================================ >> Models: CatB1 Size of training set: 16512 (100%) Size of test set: 4128 Training ========================= >> Metric: neg_mean_absolute_error Results for CatBoost: Fit --------------------------------------------- Train evaluation --> neg_mean_absolute_error: -0.232 Test evaluation --> neg_mean_absolute_error: -0.2932 Time elapsed: 5.192s Bootstrap --------------------------------------- Evaluation --> neg_mean_absolute_error: -0.3028 ± 0.0016 Time elapsed: 24.953s ------------------------------------------------- Total time: 30.145s Final results ==================== >> Duration: 30.146s ------------------------------------- CatBoost --> neg_mean_absolute_error: -0.3028 ± 0.0016 ~
Analyze results¶
In [5]:
Copied!
# The results is now multi-index, where frac is the fraction
# of the training set used to fit the model. The model names
# end with the number of models fitted during that run
atom.results
# The results is now multi-index, where frac is the fraction
# of the training set used to fit the model. The model names
# end with the number of models fitted during that run
atom.results
Out[5]:
| metric_train | metric_test | time_fit | mean_bootstrap | std_bootstrap | time_bootstrap | time | ||
|---|---|---|---|---|---|---|---|---|
| frac | model | |||||||
| 0.17 | Bag6 | -1.649188e-01 | -0.407445 | 0.125s | -0.414546 | 0.008140 | 0.844s | 0.969s |
| CatB6 | -2.320437e-01 | -0.293153 | 3.532s | -0.340974 | 0.000769 | 15.904s | 19.436s | |
| ET6 | -2.544643e-15 | -0.357537 | 0.688s | -0.383449 | 0.000824 | 2.688s | 3.375s | |
| LGB6 | -2.310124e-01 | -0.327832 | 0.281s | -0.354878 | 0.002439 | 0.719s | 1.000s | |
| RF6 | -1.457109e-01 | -0.390478 | 1.094s | -0.396854 | 0.004553 | 5.314s | 6.408s | |
| Tree6 | -3.324214e-17 | -0.559810 | 0.031s | -0.545994 | 0.008073 | 0.250s | 0.281s | |
| 0.33 | CatB3 | -2.320437e-01 | -0.293153 | 3.661s | -0.322100 | 0.001936 | 17.507s | 21.168s |
| ET3 | -2.544643e-15 | -0.357537 | 1.125s | -0.368461 | 0.003350 | 4.813s | 5.938s | |
| LGB3 | -2.310124e-01 | -0.327832 | 0.188s | -0.335865 | 0.003234 | 0.859s | 1.047s | |
| 1.00 | CatB1 | -2.320437e-01 | -0.293153 | 5.192s | -0.302825 | 0.001575 | 24.953s | 30.145s |
In [6]:
Copied!
# Plot the successive halving's results
atom.plot_successive_halving()
# Plot the successive halving's results
atom.plot_successive_halving()
In [7]:
Copied!
# Use an acronym to call all the models with the same estimator
atom.plot_errors(models=["CatB"])
# Use an acronym to call all the models with the same estimator
atom.plot_errors(models=["CatB"])
In [8]:
Copied!
# Use the number to call the models from the same run
atom.plot_errors(models="3")
# Use the number to call the models from the same run
atom.plot_errors(models="3")
