Skip to content

Normalizer

class atom.nlp.Normalizer(stopwords=True, custom_stopwords=None, stem=False, lemmatize=True, verbose=0, logger=None) [source]

Convert words to a more uniform standard. The transformations are applied on the column named Corpus, in the same order the parameters are presented. If there is no column with that name, an exception is raised. If the provided documents are strings, words are separated by spaces. This class can be accessed from atom through the normalize method. Read more in the user guide.

Parameters: stopwords: bool or str, optional (default=True)
Whether to remove a predefined dictionary of stopwords.
  • If False: Don't remove any predefined stopwords.
  • If True: Drop predefined english stopwords from the text.
  • If str: Language from nltk.corpus.stopwords.words.

custom_stopwords: sequence or None, optional (default=None)
Custom stopwords to remove from the text.

stem: bool or str, optional (default=False)
Whether to apply stemming using SnowballStemmer.
  • If False: Don't apply stemming.
  • If True: Apply stemmer based on the english language.
  • If str: Language from SnowballStemmer.languages.

lemmatize: bool, optional (default=True)
Whether to apply lemmatization using WordNetLemmatizer.

verbose: int, optional (default=0)
Verbosity level of the class. Possible values are:
  • 0 to not print anything.
  • 1 to print basic information.
  • 2 to print detailed information.
logger: str, Logger or None, optional (default=None)
  • If None: Doesn't save a logging file.
  • If str: Name of the log file. Use "auto" for automatic naming.
  • Else: Python logging.Logger instance.

Tip

Use the tokenize method to convert the documents from a string to a sequence of words.


Methods

fit_transform Same as transform.
get_params Get parameters for this estimator.
log Write information to the logger and print to stdout.
save Save the instance to a pickle file.
set_params Set the parameters of this estimator.
transform Transform the text.


method fit_transform(X, y=None) [source]

Normalize the text.

Parameters:

X: dict, list, tuple, np.array, sps.matrix or pd.DataFrame
Feature set with shape=(n_samples, n_features). If X is not a pd.DataFrame, it should be composed of a single feature containing the text documents.

y: int, str, sequence or None, optional (default=None)
Does nothing. Implemented for continuity of the API.
Returns:

X: pd.DataFrame
Transformed corpus.


method get_params(deep=True) [source]

Get parameters for this estimator.

Parameters:

deep: bool, optional (default=True)
If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns: params: dict
Dictionary of the parameter names mapped to their values.


method log(msg, level=0) [source]

Write a message to the logger and print it to stdout.

Parameters:

msg: str
Message to write to the logger and print to stdout.

level: int, optional (default=0)
Minimum verbosity level to print the message.


method save(filename="auto") [source]

Save the instance to a pickle file.

Parameters: filename: str, optional (default="auto")
Name of the file. Use "auto" for automatic naming.


method set_params(**params) [source]

Set the parameters of this estimator.

Parameters: **params: dict
Estimator parameters.
Returns: self: Normalizer
Estimator instance.


method transform(X, y=None) [source]

Normalize the text.

Parameters:

X: dict, list, tuple, np.array, sps.matrix or pd.DataFrame
Feature set with shape=(n_samples, n_features). If X is not a pd.DataFrame, it should be composed of a single feature containing the text documents.

y: int, str, sequence or None, optional (default=None)
Does nothing. Implemented for continuity of the API.
Returns:

X: pd.DataFrame
Transformed corpus.


Example

from atom import ATOMClassifier

atom = ATOMClassifier(X, y)
atom.normalize()
or 
from atom.nlp import Normalizer

normalizer = Normalizer()
X = normalizer.transform(X)

Back to top