The tf.learn API provides (among other things) a rich set of tools for working with linear models in TensorFlow. This document provides an overview of those tools. It explains:

- what a linear model is.
- why you might want to use a linear model.
- how tf.learn makes it easy to build linear models in TensorFlow.
- how you can use tf.learn to combine linear models with deep learning to get the advantages of both.

Read this overview to decide whether the tf.learn linear model tools might be useful to you. Then do the Linear Models tutorial to give it a try. This overview uses code samples from the tutorial, but the tutorial walks through the code in greater detail.

To understand this overview it will help to have some familiarity with basic machine learning concepts, and also with tf.learn.

A *linear model* uses a single weighted sum of features to make a prediction. For example, if you have data on age, years of education, and weekly hours of work for a population, you can learn weights for each of those numbers so that their weighted sum estimates a person's salary. You can also use linear models for classification.

Some linear models transform the weighted sum into a more convenient form. For example, *logistic regression* plugs the weighted sum into the logistic function to turn the output into a value between 0 and 1. But you still just have one weight for each input feature.

Why would you want to use so simple a model when recent research has demonstrated the power of more complex neural networks with many layers?

Linear models:

- train quickly, compared to deep neural nets.
- can work well on very large feature sets.
- can be trained with algorithms that don't require a lot of fiddling with learning rates, etc.
- can be interpreted and debugged more easily than neural nets. You can examine the weights assigned to each feature to figure out what's having the biggest impact on a prediction.
- provide an excellent starting point for learning about machine learning.
- are widely used in industry.

You can build a linear model from scratch in TensorFlow without the help of a special API. But tf.learn provides some tools that make it easier to build effective large-scale linear models.

Much of the work of designing a linear model consists of transforming raw data into suitable input features. tf.learn uses the `FeatureColumn`

abstraction to enable these transformations.

A `FeatureColumn`

represents a single feature in your data. A `FeatureColumn`

may represent a quantity like 'height', or it may represent a category like 'eye_color' where the value is drawn from a set of discrete possibilities like {'blue', 'brown', 'green'}.

In the case of both *continuous features* like 'height' and *categorical features* like 'eye_color', a single value in the data might get transformed into a sequence of numbers before it is input into the model. The `FeatureColumn`

abstraction lets you manipulate the feature as a single semantic unit in spite of this fact. You can specify transformations and select features to include without dealing with specific indices in the tensors you feed into the model.

Categorical features in linear models are typically translated into a sparse vector in which each possible value has a corresponding index or id. For example, if there are only three possible eye colors you can represent 'eye_color' as a length 3 vector: 'brown' would become [1, 0, 0], 'blue' would become [0, 1, 0] and 'green' would become [0, 0, 1]. These vectors are called "sparse" because they may be very long, with many zeros, when the set of possible values is very large (such as all English words).

While you don't need to use sparse columns to use tf.learn linear models, one of the strengths of linear models is their ability to deal with large sparse vectors. Sparse features are a primary use case for the tf.learn linear model tools.

`FeatureColumn`

handles the conversion of categorical values into vectors automatically, with code like this:

eye_color = tf.contrib.layers.sparse_column_with_keys( column_name="eye_color", keys=["blue", "brown", "green"])

where `eye_color`

is the name of a column in your source data.

You can also generate `FeatureColumn`

s for categorical features for which you don't know all possible values. For this case you would use `sparse_column_with_hash_bucket()`

, which uses a hash function to assign indices to feature values.

education = tf.contrib.layers.sparse_column_with_hash_bucket(\ "education", hash_bucket_size=1000)

Because linear models assign independent weights to separate features, they can't learn the relative importance of specific combinations of feature values. If you have a feature 'favorite_sport' and a feature 'home_city' and you're trying to predict whether a person likes to wear red, your linear model won't be able to learn that baseball fans from St. Louis especially like to wear red.

You can get around this limitation by creating a new feature 'favorite_sport_x_home_city'. The value of this feature for a given person is just the concatenation of the values of the two source features: 'baseball_x_stlouis', for example. This sort of combination feature is called a *feature cross*.

The `crossed_column()`

method makes it easy to set up feature crosses:

sport = tf.contrib.layers.sparse_column_with_hash_bucket(\ "sport", hash_bucket_size=1000) city = tf.contrib.layers.sparse_column_with_hash_bucket(\ "city", hash_bucket_size=1000) sport_x_city = tf.contrib.layers.crossed_column( [sport, city], hash_bucket_size=int(1e4))

You can specify a continuous feature like so:

age = tf.contrib.layers.real_valued_column("age")

Although, as a single real number, a continuous feature can often be input directly into the model, tf.learn offers useful transformations for this sort of column as well.

*Bucketization* turns a continuous column into a categorical column. This transformation lets you use continuous features in feature crosses, or learn cases where specific value ranges have particular importance.

Bucketization divides the range of possible values into subranges called buckets:

age_buckets = tf.contrib.layers.bucketized_column( age, boundaries=[18, 25, 30, 35, 40, 45, 50, 55, 60, 65])

The bucket into which a value falls becomes the categorical label for that value.

`FeatureColumn`

s provide a specification for the input data for your model, indicating how to represent and transform the data. But they do not provide the data itself. You provide the data through an input function.

The input function must return a dictionary of tensors. Each key corresponds to the name of a `FeatureColumn`

. Each key's value is a tensor containing the values of that feature for all data instances. See Building Input Functions with tf.contrib.learn for a more comprehensive look at input functions, and `input_fn`

in the linear models tutorial code for an example implementation of an input function.

The input function is passed to the `fit()`

and `evaluate()`

calls that initiate training and testing, as described in the next section.

tf.learn's estimator classes provide a unified training and evaluation harness for regression and classification models. They take care of the details of the training and evaluation loops and allow the user to focus on model inputs and architecture.

To build a linear estimator, you can use either the `tf.contrib.learn.LinearClassifier`

estimator or the `tf.contrib.learn.LinearRegressor`

estimator, for classification and regression respectively.

As with all tf.learn estimators, to run the estimator you just:

- Instantiate the estimator class. For the two linear estimator classes, you pass a list of
`FeatureColumn`

s to the constructor. - Call the estimator's
`fit()`

method to train it. - Call the estimator's
`evaluate()`

method to see how it does.

For example:

e = tf.contrib.learn.LinearClassifier(feature_columns=[ native_country, education, occupation, workclass, marital_status, race, age_buckets, education_x_occupation, age_buckets_x_race_x_occupation], model_dir=YOUR_MODEL_DIRECTORY) e.fit(input_fn=input_fn_train, steps=200) # Evaluate for one step (one pass through the test data). results = e.evaluate(input_fn=input_fn_test, steps=1) # Print the stats for the evaluation. for key in sorted(results): print("%s: %s" % (key, results[key]))

The tf.learn API also provides an estimator class that lets you jointly train a linear model and a deep neural network. This novel approach combines the ability of linear models to "memorize" key features with the generalization ability of neural nets. Use `tf.contrib.learn.DNNLinearCombinedClassifier`

to create this sort of "wide and deep" model:

e = tf.contrib.learn.DNNLinearCombinedClassifier( model_dir=YOUR_MODEL_DIR, linear_feature_columns=wide_columns, dnn_feature_columns=deep_columns, dnn_hidden_units=[100, 50])

For more information, see the Wide and Deep Learning tutorial.

© 2017 The TensorFlow Authors. All rights reserved.

Licensed under the Creative Commons Attribution License 3.0.

Code samples licensed under the Apache 2.0 License.

https://www.tensorflow.org/tutorials/linear