Early stopping¶

---

This example shows how to use early stopping to reduce the time it takes to run a pipeline. This option is only available for models that allow in-training evaluation (XGBoost, LightGBM and CatBoost).

Import the breast cancer dataset from sklearn.datasets. This is a small and easy to train dataset whose goal is to predict whether a patient has breast cancer or not.

Load the data¶

In [1]:

# Import packages
from sklearn.datasets import load_breast_cancer
from atom import ATOMClassifier

# Import packages from sklearn.datasets import load_breast_cancer from atom import ATOMClassifier

In [2]:

# Load the data
X, y = load_breast_cancer(return_X_y=True)

# Load the data X, y = load_breast_cancer(return_X_y=True)

Run the pipeline¶

In [3]:

# Initialize atom
atom = ATOMClassifier(X, y, n_jobs=2, verbose=2, warnings=False, random_state=1)

# Initialize atom atom = ATOMClassifier(X, y, n_jobs=2, verbose=2, warnings=False, random_state=1)

<< ================== ATOM ================== >>
Algorithm task: binary classification.
Parallel processing with 2 cores.

Dataset stats ==================== >>
Shape: (569, 31)
Scaled: False
Outlier values: 174 (1.2%)
-------------------------------------
Train set size: 456
Test set size: 113
-------------------------------------
|    |     dataset |       train |        test |
| -- | ----------- | ----------- | ----------- |
| 0  |   212 (1.0) |   167 (1.0) |    45 (1.0) |
| 1  |   357 (1.7) |   289 (1.7) |    68 (1.5) |

In [4]:

# Train the models using early stopping. An early stopping value of 0.1 means
# that the model will stop if it didn't improve in the last 10% of it's iterations
atom.run(
    models="LGB",
    metric="ap",
    n_calls=7,
    n_initial_points=3,
    bo_params={"early_stopping": 0.1},
)

# Train the models using early stopping. An early stopping value of 0.1 means # that the model will stop if it didn't improve in the last 10% of it's iterations atom.run( models="LGB", metric="ap", n_calls=7, n_initial_points=3, bo_params={"early_stopping": 0.1}, )

Training ========================= >>
Models: LGB
Metric: average_precision


Running BO for LightGBM...
| call             | n_estimators | learning_rate | max_depth | num_leaves | min_child_weight | min_child_samples | subsample | colsample_bytree | reg_alpha | reg_lambda | average_precision | best_average_precision | early_stopping |    time | total_time |
| ---------------- | ------------ | ------------- | --------- | ---------- | ---------------- | ----------------- | --------- | ---------------- | --------- | ---------- | ----------------- | ---------------------- | -------------- | ------- | ---------- |
| Initial point 1  |          499 |          0.73 |         1 |         40 |                5 |                18 |       0.7 |              0.8 |     100.0 |       10.0 |            0.6304 |                 0.6304 |         50/499 |  0.034s |     0.057s |
| Initial point 2  |          170 |          0.11 |         4 |         25 |               11 |                28 |       0.7 |              0.6 |     100.0 |       10.0 |            0.6304 |                 0.6304 |         18/170 |  0.025s |     0.236s |
| Initial point 3  |          364 |           0.4 |         1 |         30 |               17 |                27 |       0.9 |              0.5 |       0.0 |        1.0 |            0.9774 |                 0.9774 |         42/364 |  0.022s |     0.311s |
| Iteration 4      |          238 |          0.49 |         2 |         29 |               18 |                25 |       0.9 |              0.4 |       0.0 |       10.0 |            0.9911 |                 0.9911 |         30/238 |  0.059s |     3.278s |
| Iteration 5      |           31 |          0.07 |         5 |         21 |               18 |                28 |       0.8 |              0.5 |       0.0 |      100.0 |             0.992 |                  0.992 |          31/31 |  0.063s |     4.270s |
| Iteration 6      |           42 |          0.55 |         3 |         39 |               11 |                12 |       0.8 |              0.4 |      0.01 |      100.0 |            0.9991 |                 0.9991 |          42/42 |  0.077s |     5.281s |
| Iteration 7      |          238 |           1.0 |         2 |         40 |                1 |                10 |       0.8 |              0.3 |     100.0 |      100.0 |            0.6304 |                 0.9991 |         24/238 |  0.059s |     6.491s |

Results for LightGBM:         
Bayesian Optimization ---------------------------
Best call --> Iteration 6
Best parameters --> {'n_estimators': 42, 'learning_rate': 0.55, 'max_depth': 3, 'num_leaves': 39, 'min_child_weight': 11, 'min_child_samples': 12, 'subsample': 0.8, 'colsample_bytree': 0.4, 'reg_alpha': 0.01, 'reg_lambda': 100.0}
Best evaluation --> average_precision: 0.9991
Time elapsed: 7.616s
Fit ---------------------------------------------
Train evaluation --> average_precision: 0.9975
Test evaluation --> average_precision: 0.9885
Time elapsed: 0.067s
-------------------------------------------------
Total time: 7.690s


Final results ==================== >>
Duration: 7.690s
-------------------------------------
LightGBM --> average_precision: 0.9885

Analyze the results¶

In [5]:

# Plot the evaluation on the train and test set during training
# Note that the metric is provided by the estimator's package, not ATOM!
atom.lgb.plot_evals(title="LightGBM's evaluation curve", figsize=(11, 9))

# Plot the evaluation on the train and test set during training # Note that the metric is provided by the estimator's package, not ATOM! atom.lgb.plot_evals(title="LightGBM's evaluation curve", figsize=(11, 9))