Holdout set¶
This example shows when and how to use ATOM's holdout set in an exploration pipeline.
The data used is a variation on the Australian weather dataset from Kaggle. You can download it from here. The goal of this dataset is to predict whether or not it will rain tomorrow training a binary classifier on target RainTomorrow
.
Load the data¶
# Import packages
import pandas as pd
from atom import ATOMClassifier
# Load data
X = pd.read_csv("./datasets/weatherAUS.csv")
# Let's have a look
X.head()
Location | MinTemp | MaxTemp | Rainfall | Evaporation | Sunshine | WindGustDir | WindGustSpeed | WindDir9am | WindDir3pm | ... | Humidity9am | Humidity3pm | Pressure9am | Pressure3pm | Cloud9am | Cloud3pm | Temp9am | Temp3pm | RainToday | RainTomorrow | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | MelbourneAirport | 18.0 | 26.9 | 21.4 | 7.0 | 8.9 | SSE | 41.0 | W | SSE | ... | 95.0 | 54.0 | 1019.5 | 1017.0 | 8.0 | 5.0 | 18.5 | 26.0 | Yes | 0 |
1 | Adelaide | 17.2 | 23.4 | 0.0 | NaN | NaN | S | 41.0 | S | WSW | ... | 59.0 | 36.0 | 1015.7 | 1015.7 | NaN | NaN | 17.7 | 21.9 | No | 0 |
2 | Cairns | 18.6 | 24.6 | 7.4 | 3.0 | 6.1 | SSE | 54.0 | SSE | SE | ... | 78.0 | 57.0 | 1018.7 | 1016.6 | 3.0 | 3.0 | 20.8 | 24.1 | Yes | 0 |
3 | Portland | 13.6 | 16.8 | 4.2 | 1.2 | 0.0 | ESE | 39.0 | ESE | ESE | ... | 76.0 | 74.0 | 1021.4 | 1020.5 | 7.0 | 8.0 | 15.6 | 16.0 | Yes | 1 |
4 | Walpole | 16.4 | 19.9 | 0.0 | NaN | NaN | SE | 44.0 | SE | SE | ... | 78.0 | 70.0 | 1019.4 | 1018.9 | NaN | NaN | 17.4 | 18.1 | No | 0 |
5 rows × 22 columns
Run the pipeline¶
# Initialize atom specifying a fraction of the dataset for holdout
atom = ATOMClassifier(X, n_rows=0.5, holdout_size=0.2, verbose=2, warnings=False)
<< ================== ATOM ================== >> Algorithm task: binary classification. Dataset stats ==================== >> Shape: (56877, 22) Memory: 24.67 MB Scaled: False Missing values: 126549 (10.1%) Categorical features: 5 (23.8%) Duplicate samples: 16 (0.0%) ------------------------------------- Train set size: 42658 Test set size: 14219 Holdout set size: 14219 ------------------------------------- | | dataset | train | test | | - | ------------- | ------------- | ------------- | | 0 | 44113 (3.5) | 33085 (3.5) | 11028 (3.5) | | 1 | 12764 (1.0) | 9573 (1.0) | 3191 (1.0) |
# The test and holdout fractions are split after subsampling the dataset
# Also note that the holdout data set is not a part of atom's dataset
print("Length loaded data:", len(X))
print("Length dataset + holdout:", len(atom.dataset) + len(atom.holdout))
Length loaded data: 142193 Length dataset + holdout: 71096
atom.impute()
atom.encode()
Fitting Imputer... Imputing missing values... --> Dropping 255 samples due to missing values in feature MinTemp. --> Dropping 124 samples due to missing values in feature MaxTemp. --> Dropping 607 samples due to missing values in feature Rainfall. --> Dropping 24204 samples due to missing values in feature Evaporation. --> Dropping 27058 samples due to missing values in feature Sunshine. --> Dropping 3828 samples due to missing values in feature WindGustDir. --> Dropping 3801 samples due to missing values in feature WindGustSpeed. --> Dropping 4078 samples due to missing values in feature WindDir9am. --> Dropping 1562 samples due to missing values in feature WindDir3pm. --> Dropping 553 samples due to missing values in feature WindSpeed9am. --> Dropping 1093 samples due to missing values in feature WindSpeed3pm. --> Dropping 705 samples due to missing values in feature Humidity9am. --> Dropping 1455 samples due to missing values in feature Humidity3pm. --> Dropping 5568 samples due to missing values in feature Pressure9am. --> Dropping 5565 samples due to missing values in feature Pressure3pm. --> Dropping 21344 samples due to missing values in feature Cloud9am. --> Dropping 22688 samples due to missing values in feature Cloud3pm. --> Dropping 359 samples due to missing values in feature Temp9am. --> Dropping 1095 samples due to missing values in feature Temp3pm. --> Dropping 607 samples due to missing values in feature RainToday. Fitting Encoder... Encoding categorical columns... --> LeaveOneOut-encoding feature Location. Contains 26 classes. --> LeaveOneOut-encoding feature WindGustDir. Contains 16 classes. --> LeaveOneOut-encoding feature WindDir9am. Contains 16 classes. --> LeaveOneOut-encoding feature WindDir3pm. Contains 16 classes. --> Ordinal-encoding feature RainToday. Contains 2 classes.
# Unlike train and test, the holdout data set is not transformed until used for predictions
atom.holdout
Location | MinTemp | MaxTemp | Rainfall | Evaporation | Sunshine | WindGustDir | WindGustSpeed | WindDir9am | WindDir3pm | ... | Humidity9am | Humidity3pm | Pressure9am | Pressure3pm | Cloud9am | Cloud3pm | Temp9am | Temp3pm | RainToday | RainTomorrow | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | Albany | 10.8 | 19.8 | 0.4 | 1.4 | 9.4 | NaN | NaN | N | NNW | ... | 79.0 | 53.0 | 1019.0 | 1014.9 | 2.0 | 2.0 | 14.2 | 19.5 | No | 0 |
1 | NorahHead | 14.2 | 21.3 | 52.0 | NaN | NaN | SSE | 56.0 | S | SSE | ... | 69.0 | 70.0 | 1015.5 | 1013.6 | NaN | NaN | 18.5 | 19.6 | Yes | 1 |
2 | SalmonGums | 10.7 | 16.5 | 0.4 | NaN | NaN | WNW | 72.0 | WNW | WNW | ... | 77.0 | 61.0 | NaN | NaN | NaN | NaN | 12.3 | 11.9 | No | 0 |
3 | Tuggeranong | 11.4 | 26.9 | 12.8 | NaN | NaN | ENE | 37.0 | WSW | NE | ... | 85.0 | 54.0 | 1016.8 | 1014.6 | NaN | NaN | 15.5 | 26.0 | Yes | 1 |
4 | Richmond | 8.5 | 18.2 | 0.0 | NaN | NaN | WSW | 31.0 | NNE | E | ... | 98.0 | 55.0 | 1027.4 | 1024.3 | 8.0 | 8.0 | 10.8 | 16.6 | No | 0 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
14214 | Nuriootpa | 10.1 | 20.1 | 0.0 | 4.0 | 7.0 | SE | 35.0 | E | ESE | ... | 49.0 | 35.0 | 1027.7 | 1024.0 | 2.0 | NaN | 14.7 | 19.0 | No | 0 |
14215 | Sale | 9.3 | 17.5 | 0.0 | 4.4 | 4.9 | W | 78.0 | NW | WNW | ... | 52.0 | 46.0 | 1010.7 | 1011.2 | 6.0 | 5.0 | 14.1 | 15.9 | No | 0 |
14216 | Bendigo | 8.3 | 25.1 | 0.0 | NaN | NaN | SSE | 35.0 | SSE | SSW | ... | 51.0 | 23.0 | 1017.7 | 1014.8 | NaN | NaN | 13.1 | 23.6 | No | 0 |
14217 | Tuggeranong | 9.0 | 27.0 | 0.0 | NaN | NaN | N | 31.0 | NW | NNW | ... | 74.0 | 30.0 | 1017.9 | 1013.5 | NaN | NaN | 14.7 | 25.3 | No | 0 |
14218 | BadgerysCreek | 13.2 | 26.5 | 0.0 | NaN | NaN | WSW | 37.0 | WSW | SW | ... | 49.0 | 28.0 | 1014.2 | 1013.8 | NaN | NaN | 19.8 | 25.6 | No | 0 |
14219 rows × 22 columns
atom.run(models=["GNB", "LR", "RF"])
Training ========================= >> Models: GNB, LR, RF Metric: f1 Results for Gaussian Naive Bayes: Fit --------------------------------------------- Train evaluation --> f1: 0.6126 Test evaluation --> f1: 0.5915 Time elapsed: 0.035s ------------------------------------------------- Total time: 0.035s Results for Logistic Regression: Fit --------------------------------------------- Train evaluation --> f1: 0.6325 Test evaluation --> f1: 0.6014 Time elapsed: 0.131s ------------------------------------------------- Total time: 0.131s Results for Random Forest: Fit --------------------------------------------- Train evaluation --> f1: 1.0 Test evaluation --> f1: 0.6149 Time elapsed: 2.839s ------------------------------------------------- Total time: 2.839s Final results ==================== >> Duration: 3.005s ------------------------------------- Gaussian Naive Bayes --> f1: 0.5915 Logistic Regression --> f1: 0.6014 Random Forest --> f1: 0.6149 ~ !
atom.plot_prc()
# Based on the results on the test set, we select the best model for further tuning
atom.run("lr_tuned", n_calls=10, n_initial_points=5)
Training ========================= >> Models: LR_tuned Metric: f1 Running BO for Logistic Regression... | call | penalty | C | solver | max_iter | l1_ratio | f1 | best_f1 | time | total_time | | ---------------- | ------- | ------- | ------- | -------- | -------- | ------- | ------- | ------- | ---------- | | Initial point 1 | l2 | 0.063 | libli.. | 498 | --- | 0.6373 | 0.6373 | 0.633s | 0.646s | | Initial point 2 | none | --- | saga | 625 | --- | 0.6142 | 0.6373 | 0.746s | 1.628s | | Initial point 3 | l2 | 0.6394 | newto.. | 174 | --- | 0.6226 | 0.6373 | 0.568s | 2.269s | | Initial point 4 | l2 | 0.1383 | libli.. | 659 | --- | 0.6191 | 0.6373 | 0.530s | 2.874s | | Initial point 5 | l2 | 11.7507 | sag | 603 | --- | 0.6343 | 0.6373 | 0.589s | 3.536s | | Iteration 6 | l2 | 0.9379 | lbfgs | 575 | --- | 0.6218 | 0.6373 | 0.542s | 4.372s | | Iteration 7 | l1 | 0.0276 | libli.. | 375 | --- | 0.6337 | 0.6373 | 0.566s | 5.224s | | Iteration 8 | l1 | 7.9949 | saga | 856 | --- | 0.6306 | 0.6373 | 0.721s | 6.277s | | Iteration 9 | l2 | 0.0011 | lbfgs | 179 | --- | 0.5654 | 0.6373 | 0.571s | 7.152s | | Iteration 10 | l2 | 100.0 | lbfgs | 100 | --- | 0.633 | 0.6373 | 0.542s | 8.097s | Bayesian Optimization --------------------------- Best call --> Initial point 1 Best parameters --> {'penalty': 'l2', 'C': 0.063, 'solver': 'liblinear', 'max_iter': 498} Best evaluation --> f1: 0.6373 Time elapsed: 8.612s Fit --------------------------------------------- Train evaluation --> f1: 0.6348 Test evaluation --> f1: 0.5979 Time elapsed: 0.126s ------------------------------------------------- Total time: 8.739s Final results ==================== >> Duration: 8.739s ------------------------------------- Logistic Regression --> f1: 0.5979
Analyze the results¶
We already used the test set to choose the best model for futher tuning, so this set is no longer truly independent. Although it may not be directly visible in the results, using the test set now to evaluate the tuned LR model would be a mistake, since it carries a bias. For this reason, we have set apart an extra, indepedent set to validate the final model: the holdout set. If we are not going to use the test set for validation, we might as well use it to train the model and so optimize the use of the available data. Use the full_train method for this.
# Re-train the model on the full dataset (train + test)
atom.lr_tuned.full_train()
Model LR_tuned successfully retrained.
# Evaluate on the holdout set
atom.lr_tuned.evaluate(dataset="holdout")
accuracy 0.848277 average_precision 0.711065 balanced_accuracy 0.730813 f1 0.604979 jaccard 0.433670 matthews_corrcoef 0.525790 precision 0.727684 recall 0.517685 roc_auc 0.882586 Name: LR_tuned, dtype: float64
atom.lr_tuned.plot_prc(dataset="holdout")