Skip to content

Vectorizer

class atom.nlp.Vectorizer(strategy="bow", return_sparse=True, verbose=0, logger=None, *kwargs) [source]

Transform the corpus into meaningful vectors of numbers. The transformation is applied on the column named corpus. If there is no column with that name, an exception is raised. The transformed columns are named after the word they are embedding (if the column is already present in the provided dataset, _[strategy] is added behind the name). This class can be accessed from atom through the vectorize method. Read more in the user guide.

Parameters: strategy: str, optional (default="bow")
Strategy with which to vectorize the text. Choose from:
  • "bow": Bag of Words.
  • "tfidf": Term Frequency - Inverse Document Frequency.
  • "hashing": Vectorize to a matrix of token occurrences.

return_sparse: bool, optional (default=True)
Whether to return the transformation output as a dataframe of sparse arrays. Must be False when there are other columns in X (besides corpus) that are non-sparse.

verbose: int, optional (default=0)
Verbosity level of the class. Possible values are:
  • 0 to not print anything.
  • 1 to print basic information.
  • 2 to print detailed information.
logger: str, Logger or None, optional (default=None)
  • If None: Doesn't save a logging file.
  • If str: Name of the log file. Use "auto" for automatic naming.
  • Else: Python logging.Logger instance.

**kwargs
Additional keyword arguments for the strategy estimator.

Warning

Using return_sparse=True can turn the transformation very slow and occupy large chunks of memory when the corpus contains many tokens.


Attributes

Attributes: <strategy>: sklearn estimator
Object used to prune the data, e.g.vectorizer.bow for the Bag of Words strategy.


Methods

fit Fit to data.
fit_transform Fit to text, then vectorize it.
get_params Get parameters for this estimator.
log Write information to the logger and print to stdout.
save Save the instance to a pickle file.
set_params Set the parameters of this estimator.
transform Transform the text.


method fit(X, y=None) [source]

Fit to text.

Parameters:

X: dataframe-like
Feature set with shape=(n_samples, n_features). If X is not a pd.DataFrame, it should be composed of a single feature containing the text documents.

y: int, str, sequence or None, optional (default=None)
Does nothing. Implemented for continuity of the API.

Returns: Vectorizer
Fitted instance of self.


method fit_transform(X, y=None) [source]

Fit to text, then vectorize it.

Parameters:

X: dataframe-like
Feature set with shape=(n_samples, n_features). If X is not a pd.DataFrame, it should be composed of a single feature containing the text documents.

y: int, str, sequence or None, optional (default=None)
Does nothing. Implemented for continuity of the API.
Returns:

X: pd.DataFrame
Transformed corpus.


method get_params(deep=True) [source]

Get parameters for this estimator.

Parameters:

deep: bool, optional (default=True)
If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns: dict
Parameter names mapped to their values.


method log(msg, level=0) [source]

Write a message to the logger and print it to stdout.

Parameters:

msg: str
Message to write to the logger and print to stdout.

level: int, optional (default=0)
Minimum verbosity level to print the message.


method save(filename="auto") [source]

Save the instance to a pickle file.

Parameters: filename: str, optional (default="auto")
Name of the file. Use "auto" for automatic naming.


method set_params(**params) [source]

Set the parameters of this estimator.

Parameters: **params: dict
Estimator parameters.
Returns: Vectorizer
Estimator instance.


method transform(X, y=None) [source]

Normalize the text.

Parameters:

X: dataframe-like
Feature set with shape=(n_samples, n_features). If X is not a pd.DataFrame, it should be composed of a single feature containing the text documents.

y: int, str, sequence or None, optional (default=None)
Does nothing. Implemented for continuity of the API.
Returns:

X: pd.DataFrame
Transformed corpus.


Example

from atom import ATOMClassifier

atom = ATOMClassifier(X, y)
atom.vectorize(strategy="tfidf")

from atom.nlp import Vectorizer

vectorizer = Vectorizer("tfidf")
X = vectorizer.transform(X)
Back to top