LightGBM

---

LGBneeds scalingaccept sparseallows validationsupports acceleration

LightGBM is a gradient boosting model that uses tree based learning algorithms. It is designed to be distributed and efficient with the following advantages:

• Faster training speed and higher efficiency.
• Lower memory usage.
• Better accuracy.
• Capable of handling large-scale data.

Corresponding estimators are:

• LGBMClassifier for classification tasks.
• LGBMRegressor for regression tasks.

Read more in LightGBM's documentation.

Info

Using LightGBM's GPU acceleration requires additional software dependencies.

See Also

CatBoost

Categorical Boosting Machine.

GradientBoosting

Gradient Boosting Machine.

XGBoost

Extreme Gradient Boosting.

Example

>>> from atom import ATOMClassifier
>>> from sklearn.datasets import load_breast_cancer

>>> X, y = load_breast_cancer(return_X_y=True, as_frame=True)

>>> atom = ATOMClassifier(X, y)
>>> atom.run(models="LGB", metric="f1", verbose=2)

Training ========================= >>
Models: LGB
Metric: f1


Results for LightGBM:
Fit ---------------------------------------------
Train evaluation --> f1: 1.0
Test evaluation --> f1: 0.979
Time elapsed: 0.465s
-------------------------------------------------
Total time: 0.465s
Final results ==================== >>


Total time: 0.466s
-------------------------------------
LightGBM --> f1: 0.979

Hyperparameters

classificationregression

Parameters

n_estimators IntDistribution(high=500, log=False, low=20, step=10) learning_rate FloatDistribution(high=1.0, log=True, low=0.01, step=None) max_depth IntDistribution(high=17, log=False, low=-1, step=2) num_leaves IntDistribution(high=40, log=False, low=20, step=1) min_child_weight FloatDistribution(high=100.0, log=True, low=0.0001, step=None) min_child_samples IntDistribution(high=30, log=False, low=1, step=1) subsample FloatDistribution(high=1.0, log=False, low=0.5, step=0.1) colsample_bytree FloatDistribution(high=1.0, log=False, low=0.4, step=0.1) reg_alpha FloatDistribution(high=100.0, log=True, low=0.0001, step=None) reg_lambda FloatDistribution(high=100.0, log=True, low=0.0001, step=None)

Parameters

n_estimators IntDistribution(high=500, log=False, low=20, step=10) learning_rate FloatDistribution(high=1.0, log=True, low=0.01, step=None) max_depth IntDistribution(high=17, log=False, low=-1, step=2) num_leaves IntDistribution(high=40, log=False, low=20, step=1) min_child_weight FloatDistribution(high=100.0, log=True, low=0.0001, step=None) min_child_samples IntDistribution(high=30, log=False, low=1, step=1) subsample FloatDistribution(high=1.0, log=False, low=0.5, step=0.1) colsample_bytree FloatDistribution(high=1.0, log=False, low=0.4, step=0.1) reg_alpha FloatDistribution(high=100.0, log=True, low=0.0001, step=None) reg_lambda FloatDistribution(high=100.0, log=True, low=0.0001, step=None)

Attributes

Data attributes

Attributes

pipeline: pd.Series Transformers fitted on the data. Models that used automated feature scaling have the scaler added. Use this attribute only to access the individual instances. To visualize the pipeline, use the plot_pipeline method. mapping: dict Encoded values and their respective mapped values. The column name is the key to its mapping dictionary. Only for columns mapped to a single column (e.g. Ordinal, Leave-one-out, etc...). dataset: dataframe Complete data set. train: dataframe Training set. test: dataframe Test set. X: dataframe Feature set. y: series | dataframe Target column(s). X_train: dataframe Features of the training set. y_train: series | dataframe Target column(s) of the training set. X_test: dataframe Features of the test set. y_test: series | dataframe Target column(s) of the test set. shape: tuple[int, int] Shape of the dataset (n_rows, n_columns). columns: series Name of all the columns. n_columns: int Number of columns. features: series Name of the features. n_features: int Number of features. target: str | list[str] Name of the target column(s).

Utility attributes

Attributes

name: str Name of the model. Use the property's @setter to change the model's name. The acronym always stays at the beginning of the model's name. If the model is being tracked by mlflow, the name of the corresponding run also changes. study: Study | None Optuna study used for hyperparameter tuning. trials: pd.DataFrame | None Overview of the trials' results. All durations are in seconds. Columns include: params: Parameters used for this trial. estimator: Estimator used for this trial. score: Objective score(s) of the trial. time_trial: Duration of the trial. time_ht: Duration of the hyperparameter tuning. state: Trial's state (COMPLETE, PRUNED, FAIL). best_trial: Trial | None Trial that returned the highest score. For multi-metric runs, the best trial is the trial that performed best on the main metric. Use the property's @setter to change the best trial. See here an example. best_params: dict Hyperparameters used by the best trial. score_ht: float | list[float] | None Metric score obtained by the best trial. time_ht: int | None Duration of the hyperparameter tuning (in seconds). estimator: Predictor Estimator fitted on the training set. evals: dict Scores obtained per iteration of the training. Only the scores of the main metric are tracked. Included keys are: train and test. Read more in the user guide. score_train: float | list[float] Metric score on the training set. score_test: float | list[float] Metric score on the test set. score_holdout: float | list[float] Metric score on the holdout set. time_fit: int Duration of the model fitting on the train set (in seconds). bootstrap: pd.DataFrame | None Overview of the bootstrapping scores. The dataframe has shape=(n_bootstrap, metric) and shows the score obtained by every bootstrapped sample for every metric. Using atom.bootstrap.mean() yields the same values as score_bootstrap. score_bootstrap: float | list[float] | None Mean metric score on the bootstrapped samples. time_bootstrap: int | None Duration of the bootstrapping (in seconds). time: int Total duration of the run (in seconds). feature_importance: pd.Series | None Normalized feature importance scores. The sum of importances for all features is 1. The scores are extracted from the estimator's scores_, coef_ or feature_importances_ attribute, checked in that order. Returns None for estimators without any of those attributes. results: pd.Series Overview of the training results. All durations are in seconds. Values include: score_ht: Score obtained by the hyperparameter tuning. time_ht: Duration of the hyperparameter tuning. score_train: Metric score on the train set. score_test: Metric score on the test set. time_fit: Duration of the model fitting on the train set. score_bootstrap: Mean score on the bootstrapped samples. time_bootstrap: Duration of the bootstrapping. time: Total duration of the run.

Prediction attributes

The prediction attributes are not calculated until the attribute is called for the first time. This mechanism avoids having to calculate attributes that are never used, saving time and memory.

Attributes

decision_function_train: series | dataframe Predicted confidence scores on the training set. The shape of the output depends on the task: (n_samples,) for binary classification. (n_samples, n_classes) for multiclass classification. (n_samples, n_targets) for multilabel classification. decision_function_test: series | dataframe Predicted confidence scores on the test set. The shape of the output depends on the task: (n_samples,) for binary classification. (n_samples, n_classes) for multiclass classification. (n_samples, n_targets) for multilabel classification. decision_function_holdout: series | dataframe | None Predicted confidence scores on the holdout set. The shape of the output depends on the task: (n_samples,) for binary classification. (n_samples, n_classes) for multiclass classification. (n_samples, n_targets) for multilabel classification. predict_train: series | dataframe Class predictions on the training set. The shape of the output depends on the task: (n_samples,) for non-multioutput tasks. (n_samples, n_targets) for multioutput tasks. predict_test: series | dataframe Class predictions on the test set. The shape of the output depends on the task: (n_samples,) for non-multioutput tasks. (n_samples, n_targets) for multioutput tasks. predict_holdout: series | dataframe | None Class predictions on the holdout set. The shape of the output depends on the task: (n_samples,) for non-multioutput tasks. (n_samples, n_targets) for multioutput tasks. predict_log_proba_train: dataframe Class log-probability predictions on the training set. The shape of the output depends on the task: (n_samples, n_classes) for binary and multiclass. (n_samples, n_targets) for multilabel. (n_samples * n_classes, n_targets) for multiclass-multioutput. predict_log_proba_test: dataframe Class log-probability predictions on the test set. The shape of the output depends on the task: (n_samples, n_classes) for binary and multiclass. (n_samples, n_targets) for multilabel. (n_samples * n_classes, n_targets) for multiclass-multioutput. predict_log_proba_holdout: dataframe | None Class log-probability predictions on the holdout set. The shape of the output depends on the task: (n_samples, n_classes) for binary and multiclass. (n_samples, n_targets) for multilabel. (n_samples * n_classes, n_targets) for multiclass-multioutput. predict_proba_train: dataframe Class probability predictions on the training set. The shape of the output depends on the task: (n_samples, n_classes) for binary and multiclass. (n_samples, n_targets) for multilabel. (n_samples * n_classes, n_targets) for multiclass-multioutput. predict_proba_test: dataframe Class probability predictions on the test set. The shape of the output depends on the task: (n_samples, n_classes) for binary and multiclass. (n_samples, n_targets) for multilabel. (n_samples * n_classes, n_targets) for multiclass-multioutput. predict_proba_holdout: dataframe | None Class probability predictions on the holdout set. The shape of the output depends on the task: (n_samples, n_classes) for binary and multiclass. (n_samples, n_targets) for multilabel. (n_samples * n_classes, n_targets) for multiclass-multioutput.

Methods

The plots and prediction methods can be called directly from the model. The remaining utility methods can be found hereunder.

bootstrapping	Apply a bootstrap algorithm.
calibrate	Calibrate the model.
clear	Reset attributes and clear cache from the model.
create_app	Create an interactive app to test model predictions.
create_dashboard	Create an interactive dashboard to analyze the model.
cross_validate	Evaluate the model using cross-validation.
evaluate	Get the model's scores for the provided metrics.
export_pipeline	Export the model's pipeline to a sklearn-like object.
fit	Fit and validate the model.
full_train	Train the estimator on the complete dataset.
get_best_threshold	Get the threshold that maximizes the ROC curve.
hyperparameter_tuning	Run the hyperparameter tuning algorithm.
inverse_transform	Inversely transform new data through the pipeline.
save_estimator	Save the estimator to a pickle file.
serve	Serve the model as rest API endpoint for inference.
register	Register the model in mlflow's model registry.
transform	Transform new data through the pipeline.

method bootstrapping(n_bootstrap, reset=False)[source]

Apply a bootstrap algorithm.

Take bootstrapped samples from the training set and test them on the test set to get a distribution of the model's results.

Parameters

n_bootstrap: int umber of bootstrapped samples to fit on. reset: bool, default=False Whether to start a new run or continue the existing one.

method calibrate(**kwargs)[source]

Calibrate the model.

Applies probability calibration on the model. The estimator is trained via cross-validation on a subset of the training data, using the rest to fit the calibrator. The new classifier will replace the estimator attribute. If there is an active mlflow experiment, a new run is started using the name [model_name]_calibrate. Since the estimator changed, the model is cleared. Only for classifiers.

Parameters

**kwargs Additional keyword arguments for sklearn's CCV. Using cv="prefit" will use the trained model and fit the calibrator on the test set. Use this only if you have another, independent set for testing.

method clear()[source]

Reset attributes and clear cache from the model.

Reset certain model attributes to their initial state, deleting potentially large data arrays. Use this method to free some memory before saving the instance. The affected attributes are:

• In-training validation scores
• Shap values
• App instance
• Dashboard instance
• Cached prediction attributes
• Cached metric scores
• Cached holdout data sets

method create_app(**kwargs)[source]

Create an interactive app to test model predictions.

Demo your machine learning model with a friendly web interface. This app launches directly in the notebook or on an external browser page. The created Interface instance can be accessed through the app attribute.

Parameters

**kwargs Additional keyword arguments for the Interface instance or the Interface.launch method.

method create_dashboard(dataset="test", filename=None, **kwargs)[source]

Create an interactive dashboard to analyze the model.

ATOM uses the explainerdashboard package to provide a quick and easy way to analyze and explain the predictions and workings of the model. The dashboard allows you to investigate SHAP values, permutation importances, interaction effects, partial dependence plots, all kinds of performance plots, and even individual decision trees.

By default, the dashboard renders in a new tab in your default browser, but if preferable, you can render it inside the notebook using the mode="inline" parameter. The created ExplainerDashboard instance can be accessed through the dashboard attribute. This method is not available for multioutput tasks.

Note

Plots displayed by the dashboard are not created by ATOM and can differ from those retrieved through this package.

Parameters

dataset: str, default="test" Data set to get the report from. Choose from: "train", "test", "both" (train and test) or "holdout". filename: str or None, default=None Name to save the file with (as .html). None to not save anything. **kwargs Additional keyword arguments for the ExplainerDashboard instance.

method cross_validate(**kwargs)[source]

Evaluate the model using cross-validation.

This method cross-validates the whole pipeline on the complete dataset. Use it to assess the robustness of the solution's performance.

Parameters	**kwargs Additional keyword arguments for sklearn's cross_validate function. If the scoring method is not specified, it uses atom's metric.
Returns	pd.DataFrame Overview of the results.

method evaluate(metric=None, dataset="test", threshold=0.5, sample_weight=None)[source]

Get the model's scores for the provided metrics.

Tip

Use the self-get_best_threshold or plot_threshold method to determine a suitable value for the threshold parameter.

Parameters

metric: str, func, scorer, sequence or None, default=None Metrics to calculate. If None, a selection of the most common metrics per task are used. dataset: str, default="test" Data set on which to calculate the metric. Choose from: "train", "test" or "holdout". threshold: float or sequence, default=0.5 Threshold between 0 and 1 to convert predicted probabilities to class labels. Only used when: The task is binary or multilabel classification. The model has a predict_proba method. The metric evaluates predicted probabilities. For multilabel classification tasks, it's possible to provide a sequence of thresholds (one per target column, as returned by the get_best_threshold method). If float, the same threshold is applied to all target columns. sample_weight: sequence or None, default=None Sample weights corresponding to y in dataset.

Returns

pd.Series Scores of the model.

method export_pipeline(memory=None, verbose=None)[source]

Export the model's pipeline to a sklearn-like object.

The returned pipeline is already fitted on the training set. Note that, if the model used automated feature scaling, the Scaler is added to the pipeline.

Info

The returned pipeline behaves similarly to sklearn's Pipeline, and additionally:

• Accepts transformers that change the target column.
• Accepts transformers that drop rows.
• Accepts transformers that only are fitted on a subset of the provided dataset.
• Always returns pandas objects.
• Uses transformers that are only applied on the training set to fit the pipeline, not to make predictions.

Parameters

memory: bool, str, Memory or None, default=None Used to cache the fitted transformers of the pipeline. - If None or False: No caching is performed. - If True: A default temp directory is used. - If str: Path to the caching directory. - If Memory: Object with the joblib.Memory interface. verbose: int or None, default=None Verbosity level of the transformers in the pipeline. If None, it leaves them to their original verbosity. Note that this is not the pipeline's own verbose parameter. To change that, use the set_params method.

Returns

Pipeline Current branch as a sklearn-like Pipeline object.

method fit(X=None, y=None)[source]

Fit and validate the model.

The estimator is fitted using the best hyperparameters found during hyperparameter tuning. Afterwards, the estimator is evaluated on the test set. Only use this method to re-fit the model after having continued the study.

Parameters

X: dataframe or None Feature set with shape=(n_samples, n_features). If None, self.X_train is used. y: series or None Target column corresponding to X. If None, self.y_train is used.

method full_train(include_holdout=False)[source]

Train the estimator on the complete dataset.

In some cases it might be desirable to use all available data to train a final model. Note that doing this means that the estimator can no longer be evaluated on the test set. The newly retrained estimator will replace the estimator attribute. If there is an active mlflow experiment, a new run is started with the name [model_name]_full_train. Since the estimator changed, the model is cleared.

Warning

Although the model is trained on the complete dataset, the pipeline is not. To get a fully trained pipeline, use: pipeline = atom.export_pipeline().fit(atom.X, atom.y).

Parameters

include_holdout: bool, default=False Whether to include the holdout set (if available) in the training of the estimator. It's discouraged to use this option since it means the model can no longer be evaluated on any set.

method get_best_threshold(dataset="train")[source]

Get the threshold that maximizes the ROC curve.

Only available for models with a predict_proba method in a binary or multilabel classification task.

Parameters	dataset: str, default="train" Data set on which to calculate the threshold. Choose from: train, test, dataset.
Returns	float or list Best threshold or list of thresholds for multilabel tasks.

method hyperparameter_tuning(n_trials, reset=False)[source]

Run the hyperparameter tuning algorithm.

Search for the best combination of hyperparameters. The function to optimize is evaluated either with a K-fold cross-validation on the training set or using a random train and validation split every trial. Use this method to continue the optimization.

Parameters

n_trials: int Number of trials for the hyperparameter tuning. reset: bool, default=False Whether to start a new study or continue the existing one.

method inverse_transform(X=None, y=None, verbose=None)[source]

Inversely transform new data through the pipeline.

Transformers that are only applied on the training set are skipped. The rest should all implement a inverse_transform method. If only X or only y is provided, it ignores transformers that require the other parameter. This can be of use to, for example, inversely transform only the target column. If called from a model that used automated feature scaling, the scaling is inverted as well.

Parameters

X: dataframe-like or None, default=None Transformed feature set with shape=(n_samples, n_features). If None, X is ignored in the transformers. y: int, str, dict, sequence, dataframe or None, default=None Target column corresponding to X. If None: y is ignored. If int: Position of the target column in X. If str: Name of the target column in X. If sequence: Target array with shape=(n_samples,) or sequence of column names or positions for multioutput tasks. If dataframe: Target columns for multioutput tasks. verbose: int or None, default=None Verbosity level for the transformers. If None, it uses the transformer's own verbosity.

Returns

dataframe Original feature set. Only returned if provided. series Original target column. Only returned if provided.

method save_estimator(filename="auto")[source]

Save the estimator to a pickle file.

Parameters

filename: str, default="auto" Name of the file. Use "auto" for automatic naming.

method serve(method="predict", host="127.0.0.1", port=8000)[source]

Serve the model as rest API endpoint for inference.

The complete pipeline is served with the model. The inference data must be supplied as json to the HTTP request, e.g. requests.get("http://127.0.0.1:8000/", json=X.to_json()). The deployment is done on a ray cluster. The default host and port parameters deploy to localhost.

Tip

Use import ray; ray.serve.shutdown() to close the endpoint after finishing.

Parameters

method: str, default="predict" Estimator's method to do inference on. host: str, default="127.0.0.1" Host for HTTP servers to listen on. To expose serve publicly, you probably want to set this to "0.0.0.0". port: int, default=8000 Port for HTTP server.

method register(name=None, stage="Staging")[source]

Register the model in mlflow's model registry.

This method is only available when model tracking is enabled using one of the following URI schemes: databricks, http, https, postgresql, mysql, sqlite, mssql.

Parameters

name: str or None, default=None Name for the registered model. If None, the model's full name is used. stage: str, default="Staging" New desired stage for the model.

method transform(X=None, y=None, verbose=None)[source]

Transform new data through the pipeline.

Transformers that are only applied on the training set are skipped. If only X or only y is provided, it ignores transformers that require the other parameter. This can be of use to, for example, transform only the target column. If called from a model that used automated feature scaling, the data is scaled as well.

Parameters

X: dataframe-like or None, default=None Feature set with shape=(n_samples, n_features). If None, X is ignored. If None, X is ignored in the transformers. y: int, str, dict, sequence, dataframe or None, default=None Target column corresponding to X. If None: y is ignored. If int: Position of the target column in X. If str: Name of the target column in X. If sequence: Target array with shape=(n_samples,) or sequence of column names or positions for multioutput tasks. If dataframe: Target columns for multioutput tasks. verbose: int or None, default=None Verbosity level for the transformers. If None, it uses the transformer's own verbosity.

Returns

dataframe Transformed feature set. Only returned if provided. series Transformed target column. Only returned if provided.