class sklearn.feature_extraction.text.TfidfTransformer(norm=’l2’, use_idf=True, smooth_idf=True, sublinear_tf=False)
[source]
Transform a count matrix to a normalized tf or tfidf representation
Tf means termfrequency while tfidf means termfrequency times inverse documentfrequency. This is a common term weighting scheme in information retrieval, that has also found good use in document classification.
The goal of using tfidf instead of the raw frequencies of occurrence of a token in a given document is to scale down the impact of tokens that occur very frequently in a given corpus and that are hence empirically less informative than features that occur in a small fraction of the training corpus.
The formula that is used to compute the tfidf of term t is tfidf(d, t) = tf(t) * idf(d, t), and the idf is computed as idf(d, t) = log [ n / df(d, t) ] + 1 (if smooth_idf=False
), where n is the total number of documents and df(d, t) is the document frequency; the document frequency is the number of documents d that contain term t. The effect of adding “1” to the idf in the equation above is that terms with zero idf, i.e., terms that occur in all documents in a training set, will not be entirely ignored. (Note that the idf formula above differs from the standard textbook notation that defines the idf as idf(d, t) = log [ n / (df(d, t) + 1) ]).
If smooth_idf=True
(the default), the constant “1” is added to the numerator and denominator of the idf as if an extra document was seen containing every term in the collection exactly once, which prevents zero divisions: idf(d, t) = log [ (1 + n) / (1 + df(d, t)) ] + 1.
Furthermore, the formulas used to compute tf and idf depend on parameter settings that correspond to the SMART notation used in IR as follows:
Tf is “n” (natural) by default, “l” (logarithmic) when sublinear_tf=True
. Idf is “t” when use_idf is given, “n” (none) otherwise. Normalization is “c” (cosine) when norm='l2'
, “n” (none) when norm=None
.
Read more in the User Guide.
Parameters: 


Attributes: 

[Yates2011]  R. BaezaYates and B. RibeiroNeto (2011). Modern Information Retrieval. Addison Wesley, pp. 6874. 
[MRS2008]  C.D. Manning, P. Raghavan and H. Schütze (2008). Introduction to Information Retrieval. Cambridge University Press, pp. 118120. 
fit (X[, y])  Learn the idf vector (global term weights) 
fit_transform (X[, y])  Fit to data, then transform it. 
get_params ([deep])  Get parameters for this estimator. 
set_params (**params)  Set the parameters of this estimator. 
transform (X[, copy])  Transform a count matrix to a tf or tfidf representation 
__init__(norm=’l2’, use_idf=True, smooth_idf=True, sublinear_tf=False)
[source]
fit(X, y=None)
[source]
Learn the idf vector (global term weights)
Parameters: 


fit_transform(X, y=None, **fit_params)
[source]
Fit to data, then transform it.
Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.
Parameters: 


Returns: 

get_params(deep=True)
[source]
Get parameters for this estimator.
Parameters: 


Returns: 

set_params(**params)
[source]
Set the parameters of this estimator.
The method works on simple estimators as well as on nested objects (such as pipelines). The latter have parameters of the form <component>__<parameter>
so that it’s possible to update each component of a nested object.
Returns: 


transform(X, copy=True)
[source]
Transform a count matrix to a tf or tfidf representation
Parameters: 


Returns: 

sklearn.feature_extraction.text.TfidfTransformer
© 2007–2018 The scikitlearn developers
Licensed under the 3clause BSD License.
http://scikitlearn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html