Early stopping¶

---

This example shows how to use early stopping to reduce the time it takes to run a pipeline. This option is only available for models that allow in-training evaluation (XGBoost, LightGBM and CatBoost).

Import the breast cancer dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict whether a patient has breast cancer or not.

Load the data¶

In [1]:

# Import packages
from sklearn.datasets import load_breast_cancer
from atom import ATOMClassifier

# Import packages from sklearn.datasets import load_breast_cancer from atom import ATOMClassifier

In [2]:

# Load the data
X, y = load_breast_cancer(return_X_y=True)

# Load the data X, y = load_breast_cancer(return_X_y=True)

Run the pipeline¶

In [3]:

# Initialize atom
atom = ATOMClassifier(X, y, n_jobs=2, verbose=2, warnings=False, random_state=1)

# Initialize atom atom = ATOMClassifier(X, y, n_jobs=2, verbose=2, warnings=False, random_state=1)

<< ================== ATOM ================== >>
Algorithm task: binary classification.
Parallel processing with 2 cores.

Dataset stats ==================== >>
Shape: (569, 31)
Memory: 138.96 kB
Scaled: False
Outlier values: 169 (1.2%)
-------------------------------------
Train set size: 456
Test set size: 113
-------------------------------------
|   |     dataset |       train |        test |
| - | ----------- | ----------- | ----------- |
| 0 |   212 (1.0) |   170 (1.0) |    42 (1.0) |
| 1 |   357 (1.7) |   286 (1.7) |    71 (1.7) |

In [4]:

# Train the models using early stopping. An early stopping value of 0.1 means
# that the model will stop if it didn't improve in the last 10% of it's iterations
atom.run(
    models="LGB",
    metric="ap",
    n_calls=7,
    n_initial_points=3,
    bo_params={"early_stopping": 0.1},
)

# Train the models using early stopping. An early stopping value of 0.1 means # that the model will stop if it didn't improve in the last 10% of it's iterations atom.run( models="LGB", metric="ap", n_calls=7, n_initial_points=3, bo_params={"early_stopping": 0.1}, )

Training ========================= >>
Models: LGB
Metric: average_precision


Running BO for LightGBM...
| call             | n_estimators | learning_rate | max_depth | num_leaves | min_child_weight | min_child_samples | subsample | colsample_bytree | reg_alpha | reg_lambda | average_precision | best_average_precision | early_stopping |    time | total_time |
| ---------------- | ------------ | ------------- | --------- | ---------- | ---------------- | ----------------- | --------- | ---------------- | --------- | ---------- | ----------------- | ---------------------- | -------------- | ------- | ---------- |
| Initial point 1  |          499 |         0.733 |         2 |         40 |            0.001 |                13 |       0.7 |              0.8 |       100 |         10 |            0.6264 |                 0.6264 |         50/499 |  0.040s |     0.075s |
| Initial point 2  |          170 |         0.112 |         7 |         25 |              0.1 |                28 |       0.7 |              0.7 |       100 |         10 |            0.6264 |                 0.6264 |         18/170 |  0.028s |     0.592s |
| Initial point 3  |          364 |        0.4032 |         1 |         30 |               10 |                25 |       0.9 |              0.6 |         0 |          1 |            0.9923 |                 0.9923 |         49/364 |  0.025s |     0.782s |
| Iteration 4      |          280 |        0.4959 |         3 |         30 |              0.1 |                14 |       0.9 |              0.5 |         0 |        0.1 |            0.9991 |                 0.9991 |         45/280 |  0.013s |     3.047s |
| Iteration 5      |           72 |        0.8273 |         8 |         30 |           0.0001 |                 1 |       0.9 |              0.4 |         0 |          0 |            0.9923 |                 0.9991 |          13/72 |  0.029s |     3.700s |
| Iteration 6      |          500 |        0.0483 |        -1 |         30 |              100 |                16 |       0.9 |              0.6 |      0.01 |        100 |            0.6264 |                 0.9991 |         51/500 |  0.324s |     4.801s |
| Iteration 7      |          500 |          0.01 |        -1 |         37 |              100 |                30 |       0.8 |              0.4 |         0 |        100 |            0.6264 |                 0.9991 |         51/500 |  0.028s |     5.580s |
Bayesian Optimization ---------------------------
Best call --> Iteration 4
Best parameters --> {'n_estimators': 280, 'learning_rate': 0.4959, 'max_depth': 3, 'num_leaves': 30, 'min_child_weight': 0.1, 'min_child_samples': 14, 'subsample': 0.9, 'colsample_bytree': 0.5, 'reg_alpha': 0, 'reg_lambda': 0.1}
Best evaluation --> average_precision: 0.9991
Time elapsed: 6.269s
Fit ---------------------------------------------
Early stop at iteration 106 of 280.
Train evaluation --> average_precision: 1.0
Test evaluation --> average_precision: 0.9976
Time elapsed: 0.096s
-------------------------------------------------
Total time: 6.366s


Final results ==================== >>
Duration: 6.368s
-------------------------------------
LightGBM --> average_precision: 0.9976

Analyze the results¶

In [5]:

# Plot the evaluation on the train and test set during training
# Note that the metric is provided by the estimator's package, not ATOM!
atom.lgb.plot_evals(title="LightGBM's evaluation curve", figsize=(11, 9))

# Plot the evaluation on the train and test set during training # Note that the metric is provided by the estimator's package, not ATOM! atom.lgb.plot_evals(title="LightGBM's evaluation curve", figsize=(11, 9))