Pruner

class atom.data_cleaning.Pruner(strategy="zscore", method="drop", max_sigma=3, include_target=False, device="cpu", engine="sklearn", verbose=0, logger=None, **kwargs)[source]

Prune outliers from the data.

Replace or remove outliers. The definition of outlier depends on the selected strategy and can greatly differ from one another. Ignores categorical columns.

This class can be accessed from atom through the prune method. Read more in the user guide.

Info

The "sklearnex" and "cuml" engines are only supported for strategy="dbscan".

Parameters

strategy: str or sequence, default="zscore"

Strategy with which to select the outliers. If sequence of strategies, only samples marked as outliers by all chosen strategies are dropped. Choose from:

"zscore": Z-score of each data value.
"iforest": Isolation Forest.
"ee": Elliptic Envelope.
"lof": Local Outlier Factor.
"svm": One-class SVM.
"dbscan": Density-Based Spatial Clustering.
"optics": DBSCAN-like clustering approach.

method: int, float or str, default="drop"

Method to apply on the outliers. Only the zscore strategy accepts another method than "drop". Choose from:

"drop": Drop any sample with outlier values.
"min_max": Replace outlier with the min/max of the column.
Any numerical value with which to replace the outliers.

max_sigma: int or float, default=3

Maximum allowed standard deviations from the mean of the column. If more, it is considered an outlier. Only if strategy="zscore".

include_target: bool, default=False

Whether to include the target column in the search for outliers. This can be useful for regression tasks. Only if strategy="zscore".

device: str, default="cpu"

Device on which to train the estimators. Use any string that follows the SYCL_DEVICE_FILTER filter selector, e.g. device="gpu" to use the GPU. Read more in the user guide.

engine: str, default="sklearn"

Execution engine to use for the estimators. Refer to the user guide for an explanation regarding every choice. Choose from:

"sklearn" (only if device="cpu")
"sklearnex"
"cuml" (only if device="gpu")

verbose: int, default=0

Verbosity level of the class. Choose from:

0 to not print anything.
1 to print basic information.
2 to print detailed information.

logger: str, Logger or None, default=None

If None: Doesn't save a logging file.
If str: Name of the log file. Use "auto" for automatic naming.
Else: Python logging.Logger instance.

**kwargs

Additional keyword arguments for the strategy estimator. If sequence of strategies, the params should be provided in a dict with the strategy's name as key.

Attributes

[strategy]: sklearn estimator

Object used to prune the data, e.g. pruner.iforest for the isolation forest strategy.

Example

atomstand-alone

>>> from atom import ATOMClassifier
>>> from sklearn.datasets import load_breast_cancer

>>> X, y = load_breast_cancer(return_X_y=True, as_frame=True)

>>> atom = ATOMClassifier(X, y)
>>> print(atom.dataset)

     mean radius  mean texture  ...  worst fractal dimension  target
0          11.04         14.93  ...                  0.07287       1
1          12.46         24.04  ...                  0.20750       0
2          13.47         14.06  ...                  0.09326       1
3          13.44         21.58  ...                  0.07146       0
4          11.93         21.53  ...                  0.08541       1
..           ...           ...  ...                      ...     ...
564        14.54         27.54  ...                  0.13410       0
565        18.66         17.12  ...                  0.08456       0
566        10.95         21.35  ...                  0.09606       0
567        17.01         20.26  ...                  0.06469       0
568        12.40         17.68  ...                  0.09359       1

[569 rows x 31 columns]

>>> atom.prune(stratgey="iforest", verbose=2)

Pruning outliers...
 --> Dropping 46 outliers.

>>> # Note the reduced number of rows
>>> print(atom.dataset)

     mean radius  mean texture  ...  worst fractal dimension  target
0          11.04         14.93  ...                  0.07287       1
1          13.47         14.06  ...                  0.09326       1
2          13.44         21.58  ...                  0.07146       0
3          11.93         21.53  ...                  0.08541       1
4          13.21         25.25  ...                  0.06788       1
..           ...           ...  ...                      ...     ...
518        14.54         27.54  ...                  0.13410       0
519        18.66         17.12  ...                  0.08456       0
520        10.95         21.35  ...                  0.09606       0
521        17.01         20.26  ...                  0.06469       0
522        12.40         17.68  ...                  0.09359       1

[523 rows x 31 columns]


>>> atom.plot_distribution(columns=0)

>>> from atom.data_cleaning import Normalizer
>>> from sklearn.datasets import load_breast_cancer

>>> X, y = load_breast_cancer(return_X_y=True, as_frame=True)

     mean radius  mean texture  ...  worst symmetry  worst fractal dimension
0          17.99         10.38  ...          0.4601                  0.11890
1          20.57         17.77  ...          0.2750                  0.08902
2          19.69         21.25  ...          0.3613                  0.08758
3          11.42         20.38  ...          0.6638                  0.17300
4          20.29         14.34  ...          0.2364                  0.07678
..           ...           ...  ...             ...                      ...
564        21.56         22.39  ...          0.2060                  0.07115
565        20.13         28.25  ...          0.2572                  0.06637
566        16.60         28.08  ...          0.2218                  0.07820
567        20.60         29.33  ...          0.4087                  0.12400
568         7.76         24.54  ...          0.2871                  0.07039

[569 rows x 30 columns]

>>> normalizer = Normalizer(verbose=2)
>>> X = normalizer.fit_transform(X)

Fitting Pruner...
Pruning outliers...
 --> Dropping 74 outliers.

>>> # Note the reduced number of rows
>>> print(X)

     mean radius  mean texture  ...  worst symmetry  worst fractal dimension
1          20.57         17.77  ...          0.2750                  0.08902
2          19.69         21.25  ...          0.3613                  0.08758
4          20.29         14.34  ...          0.2364                  0.07678
5          12.45         15.70  ...          0.3985                  0.12440
6          18.25         19.98  ...          0.3063                  0.08368
..           ...           ...  ...             ...                      ...
560        14.05         27.15  ...          0.2250                  0.08321
563        20.92         25.09  ...          0.2929                  0.09873
564        21.56         22.39  ...          0.2060                  0.07115
565        20.13         28.25  ...          0.2572                  0.06637
566        16.60         28.08  ...          0.2218                  0.07820

[495 rows x 30 columns]

Methods

fit	Does nothing.
fit_transform	Fit to data, then transform it.
get_params	Get parameters for this estimator.
inverse_transform	Does nothing.
log	Print message and save to log file.
save	Save the instance to a pickle file.
set_params	Set the parameters of this estimator.
transform	Apply the outlier strategy on the data.

method fit(X=None, y=None, **fit_params)[source]

Does nothing.

Implemented for continuity of the API.

Parameters

X: dataframe-like or None, default=None

Feature set with shape=(n_samples, n_features). If None, X is ignored.

y: int, str, sequence, dataframe-like or None, default=None

Target column corresponding to X.

If None: y is ignored.
If int: Position of the target column in X.
If str: Name of the target column in X.
If sequence: Target column with shape=(n_samples,) or sequence of column names or positions for multioutput tasks.
If dataframe-like: Target columns with shape=(n_samples, n_targets) for multioutput tasks.

**fit_params

Additional keyword arguments for the fit method.

Returns

self

Estimator instance.

method fit_transform(X=None, y=None, **fit_params)[source]

Fit to data, then transform it.

Parameters

X: dataframe-like or None, default=None

Feature set with shape=(n_samples, n_features). If None, X is ignored.

y: int, str, sequence, dataframe-like or None, default=None

Target column corresponding to X.

If None: y is ignored.
If int: Position of the target column in X.
If str: Name of the target column in X.
If sequence: Target column with shape=(n_samples,) or sequence of column names or positions for multioutput tasks.
If dataframe-like: Target columns with shape=(n_samples, n_targets) for multioutput tasks.

**fit_params

Additional keyword arguments for the fit method.

Returns

dataframe

Transformed feature set. Only returned if provided.

series

Transformed target column. Only returned if provided.

method get_params(deep=True)[source]

Get parameters for this estimator.

Parameters	deep : bool, default=True If True, will return the parameters for this estimator and contained subobjects that are estimators.
Returns	params : dict Parameter names mapped to their values.

method inverse_transform(X=None, y=None)[source]

Does nothing.

Parameters

X: dataframe-like or None, default=None

Feature set with shape=(n_samples, n_features). If None, X is ignored.

y: int, str, sequence, dataframe-like or None, default=None

Target column corresponding to X.

If None: y is ignored.
If int: Position of the target column in X.
If str: Name of the target column in X.
If sequence: Target column with shape=(n_samples,) or sequence of column names or positions for multioutput tasks.
If dataframe-like: Target columns with shape=(n_samples, n_targets) for multioutput tasks.

Returns

dataframe

Transformed feature set. Only returned if provided.

series

Transformed target column. Only returned if provided.

method log(msg, level=0, severity="info")[source]

Print message and save to log file.

Parameters

msg: int, float or str

Message to save to the logger and print to stdout.

level: int, default=0

Minimum verbosity level to print the message.

severity: str, default="info"

Severity level of the message. Choose from: debug, info, warning, error, critical.

method save(filename="auto", save_data=True)[source]

Save the instance to a pickle file.

Parameters

filename: str, default="auto"

Name of the file. Use "auto" for automatic naming.

save_data: bool, default=True

Whether to save the dataset with the instance. This parameter is ignored if the method is not called from atom. If False, add the data to the load method.

method set_params(**params)[source]

Set the parameters of this estimator.

Parameters	**params : dict Estimator parameters.
Returns	self : estimator instance Estimator instance.

method transform(X, y=None)[source]

Apply the outlier strategy on the data.

Parameters

X: dataframe-like

Feature set with shape=(n_samples, n_features).

y: int, str, dict, sequence, dataframe-like or None, default=None

Target column corresponding to X.

If None: y is ignored.
If int: Position of the target column in X.
If str: Name of the target column in X.
If sequence: Target array with shape=(n_samples,) or sequence of column names or positions for multioutput tasks.
If dataframe: Target columns for multioutput tasks.

Returns

dataframe

Transformed feature set.

series

Transformed target column. Only returned if provided.