Early stopping¶

---

This example shows how to use early stopping to reduce the time it takes to run a pipeline. This option is only available for models that allow in-training evaluation (XGBoost, LightGBM and CatBoost).

Import the breast cancer dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict whether a patient has breast cancer or not.

Load the data¶

In [1]:

# Import packages
from sklearn.datasets import load_breast_cancer
from atom import ATOMClassifier

# Import packages from sklearn.datasets import load_breast_cancer from atom import ATOMClassifier

In [2]:

# Load the data
X, y = load_breast_cancer(return_X_y=True)

# Load the data X, y = load_breast_cancer(return_X_y=True)

Run the pipeline¶

In [3]:

# Initialize atom
atom = ATOMClassifier(X, y, n_jobs=2, verbose=2, warnings=False, random_state=1)

# Initialize atom atom = ATOMClassifier(X, y, n_jobs=2, verbose=2, warnings=False, random_state=1)

<< ================== ATOM ================== >>
Algorithm task: binary classification.
Parallel processing with 2 cores.

Dataset stats ==================== >>
Shape: (569, 31)
Scaled: False
Outlier values: 174 (1.2%)
-------------------------------------
Train set size: 456
Test set size: 113
-------------------------------------
|    |     dataset |       train |        test |
| -- | ----------- | ----------- | ----------- |
| 0  |   212 (1.0) |   167 (1.0) |    45 (1.0) |
| 1  |   357 (1.7) |   289 (1.7) |    68 (1.5) |

In [4]:

# Train the models using early stopping. An early stopping value of 0.1 means
# that the model will stop if it didn't improve in the last 10% of it's iterations
atom.run(
    models="LGB",
    metric="ap",
    n_calls=7,
    n_initial_points=3,
    bo_params={"early_stopping": 0.1},
)

# Train the models using early stopping. An early stopping value of 0.1 means # that the model will stop if it didn't improve in the last 10% of it's iterations atom.run( models="LGB", metric="ap", n_calls=7, n_initial_points=3, bo_params={"early_stopping": 0.1}, )

Training ========================= >>
Models: LGB
Metric: average_precision



Running BO for LightGBM...
| call             | n_estimators | learning_rate | max_depth | num_leaves | min_child_weight | min_child_samples | subsample | colsample_bytree | reg_alpha | reg_lambda | average_precision | best_average_precision | early_stopping |    time | total_time |
| ---------------- | ------------ | ------------- | --------- | ---------- | ---------------- | ----------------- | --------- | ---------------- | --------- | ---------- | ----------------- | ---------------------- | -------------- | ------- | ---------- |
| Initial point 1  |          499 |          0.73 |         1 |         40 |                5 |                18 |       0.7 |              0.8 |     100.0 |       10.0 |            0.6374 |                 0.6374 |         50/499 |  0.043s |     0.066s |
| Initial point 2  |          170 |          0.11 |         4 |         25 |               11 |                28 |       0.7 |              0.6 |     100.0 |       10.0 |            0.6374 |                 0.6374 |         18/170 |  0.040s |     0.294s |
| Initial point 3  |          364 |           0.4 |         1 |         30 |               17 |                27 |       0.9 |              0.5 |       0.0 |        1.0 |            0.9777 |                 0.9777 |         42/364 |  0.040s |     0.394s |
| Iteration 4      |          238 |          0.49 |         2 |         29 |               18 |                25 |       0.9 |              0.4 |       0.0 |       10.0 |            0.9897 |                 0.9897 |         31/238 |  0.041s |     1.430s |
| Iteration 5      |          144 |          0.03 |         9 |         21 |                1 |                23 |       0.8 |              0.4 |       0.0 |      100.0 |             0.971 |                 0.9897 |        144/144 |  0.054s |     1.897s |
| Iteration 6      |           20 |           1.0 |        -1 |         40 |               20 |                10 |       1.0 |              0.3 |       0.0 |      100.0 |             0.994 |                  0.994 |          11/20 |  0.037s |     2.437s |
| Iteration 7      |           31 |          0.04 |         5 |         39 |               20 |                19 |       1.0 |              0.5 |      0.01 |      100.0 |            0.9918 |                  0.994 |          31/31 |  0.039s |     2.928s |

Results for LightGBM:         
Bayesian Optimization ---------------------------
Best call --> Iteration 6
Best parameters --> {'n_estimators': 20, 'learning_rate': 1.0, 'max_depth': -1, 'num_leaves': 40, 'min_child_weight': 20, 'min_child_samples': 10, 'subsample': 1.0, 'colsample_bytree': 0.3, 'reg_alpha': 0.0, 'reg_lambda': 100.0}
Best evaluation --> average_precision: 0.994
Time elapsed: 3.409s
Fit ---------------------------------------------
Early stop at iteration 15 of 20.
Train evaluation --> average_precision: 0.995
Test evaluation --> average_precision: 0.991
Time elapsed: 0.026s
-------------------------------------------------
Total time: 3.436s


Final results ==================== >>
Duration: 3.436s
-------------------------------------
LightGBM --> average_precision: 0.991

Analyze the results¶

In [5]:

# Plot the evaluation on the train and test set during training
# Note that the metric is provided by the estimator's package, not ATOM!
atom.lgb.plot_evals(title="LightGBM's evaluation curve", figsize=(11, 9))

# Plot the evaluation on the train and test set during training # Note that the metric is provided by the estimator's package, not ATOM! atom.lgb.plot_evals(title="LightGBM's evaluation curve", figsize=(11, 9))