Tokenizer
Tokenize the corpus.
Convert documents into sequences of words. Additionally,
create n-grams (represented by words united with underscores,
e.g., "New_York") based on their frequency in the corpus. The
transformations are applied on the column named corpus
. If
there is no column with that name, an exception is raised.
This class can be accessed from atom through the tokenize method. Read more in the user guide.
See Also
Example
>>> from atom import ATOMClassifier
>>> X = [
... ["I àm in ne'w york"],
... ["New york is nice"],
... ["new york"],
... ["hi there this is a test!"],
... ["another line..."],
... ["new york is larger than washington"],
... ["running the test"],
... ["this is a test"],
... ]
>>> y = [1, 0, 0, 1, 1, 1, 0, 0]
>>> atom = ATOMClassifier(X, y, test_size=2, random_state=1)
>>> print(atom.dataset)
corpus target
0 new york 0
1 another line... 1
2 New york is nice 0
3 new york is larger than washington 1
4 running the test 0
5 I àm in ne'w york 1
6 this is a test 0
7 hi there this is a test! 1
>>> atom.tokenize(verbose=2)
Fitting Tokenizer...
Tokenizing the corpus...
>>> print(atom.dataset)
corpus target
0 [new, york] 0
1 [another, line, ...] 1
2 [New, york, is, nice] 0
3 [new, york, is, larger, than, washington] 1
4 [running, the, test] 0
5 [I, àm, in, ne, ', w, york] 1
6 [this, is, a, test] 0
7 [hi, there, this, is, a, test, !] 1
>>> from atom.nlp import Tokenizer
>>> X = [
... ["I àm in ne'w york"],
... ["New york is nice"],
... ["new york"],
... ["hi there this is a test!"],
... ["another line..."],
... ["new york is larger than washington"],
... ["running the test"],
... ["this is a test"],
... ]
>>> tokenizer = Tokenizer(bigram_freq=2, verbose=2)
>>> X = tokenizer.transform(X)
Tokenizing the corpus...
--> Creating 5 bigrams on 10 locations.
>>> print(X)
corpus
0 [I, àm, in, ne, ', w, york]
1 [New, york_is, nice]
2 [new_york]
3 [hi, there, this_is, a_test, !]
4 [another, line, ...]
5 [new, york_is, larger, than, washington]
6 [running, the, test]
7 [this_is, a_test]
Methods
fit | Do nothing. |
fit_transform | Fit to data, then transform it. |
get_feature_names_out | Get output feature names for transformation. |
get_params | Get parameters for this estimator. |
inverse_transform | Do nothing. |
set_output | Set output container. |
set_params | Set the parameters of this estimator. |
transform | Tokenize the text. |
Do nothing.
Implemented for continuity of the API.
Fit to data, then transform it.
Get output feature names for transformation.
Get parameters for this estimator.
Parameters |
deep : bool, default=True
If True, will return the parameters for this estimator and
contained subobjects that are estimators.
|
Returns |
params : dict
Parameter names mapped to their values.
|
Do nothing.
Returns the input unchanged. Implemented for continuity of the API.
Set output container.
See sklearn's user guide on how to use the
set_output
API. See here a description
of the choices.
Set the parameters of this estimator.
Parameters |
**params : dict
Estimator parameters.
|
Returns |
self : estimator instance
Estimator instance.
|
Tokenize the text.