View source on GitHub |

Optimizer that implements the RMSprop algorithm.

Inherits From: `Optimizer`

tf.keras.optimizers.RMSprop( learning_rate=0.001, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False, name='RMSprop', **kwargs )

A detailed description of rmsprop.

- maintain a moving (discounted) average of the square of gradients
- divide gradient by the root of this average

$$mean_square_t = rho * mean_square{t-1} + (1-rho) * gradient ** 2$$

$$mom_t = momentum * mom_{t-1} + learning_rate * gradient / \sqrt{ / mean_square_t + \epsilon}$$

$$variable_t := variable_{t-1} - mom_t$$

This implementation of RMSprop uses plain momentum, not Nesterov momentum.

The centered version additionally maintains a moving average of the gradients, and uses that average to estimate the variance:

$$mean_grad_t = rho * mean_grad_{t-1} + (1-rho) * gradient$$

$$mean_square_t = rho * mean_square_{t-1} + (1-rho) * gradient ** 2$$

$$mom_t = momentum * mom_{t-1} + learning_rate * gradient / sqrt(mean_square_t - mean_grad_t**2 + epsilon)$$

$$variable_t := variable_{t-1} - mom_t$$

References See ([pdf] http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf).

Args | |
---|---|

`learning_rate` | A Tensor or a floating point value. The learning rate. |

`rho` | Discounting factor for the history/coming gradient |

`momentum` | A scalar tensor. |

`epsilon` | Small value to avoid zero denominator. |

`centered` | If True, gradients are normalized by the estimated variance of the gradient; if False, by the uncentered second moment. Setting this to True may help with training, but is slightly more expensive in terms of computation and memory. Defaults to False. |

`name` | Optional name prefix for the operations created when applying gradients. Defaults to "RMSprop". @compatibility(eager) When eager execution is enabled, `learning_rate` , `decay` , `momentum` , and `epsilon` can each be a callable that takes no arguments and returns the actual value to use. This can be useful for changing these values across different invocations of optimizer functions. @end_compatibility |

`**kwargs` | keyword arguments. Allowed to be {`clipnorm` , `clipvalue` , `lr` , `decay` }. `clipnorm` is clip gradients by norm; `clipvalue` is clip gradients by value, `decay` is included for backward compatibility to allow time inverse decay of learning rate. `lr` is included for backward compatibility, recommended to use `learning_rate` instead. |

Attributes | |
---|---|

`iterations` | Variable. The number of training steps this Optimizer has run. |

`weights` | Returns variables of this Optimizer based on the order created. |

`add_slot`

add_slot( var, slot_name, initializer='zeros' )

Add a new slot variable for `var`

.

`add_weight`

add_weight( name, shape, dtype=None, initializer='zeros', trainable=None, synchronization=tf.VariableSynchronization.AUTO, aggregation=tf.VariableAggregation.NONE )

`apply_gradients`

apply_gradients( grads_and_vars, name=None )

Apply gradients to variables.

This is the second part of `minimize()`

. It returns an `Operation`

that applies gradients.

Args | |
---|---|

`grads_and_vars` | List of (gradient, variable) pairs. |

`name` | Optional name for the returned operation. Default to the name passed to the `Optimizer` constructor. |

Returns | |
---|---|

An `Operation` that applies the specified gradients. The `iterations` will be automatically increased by 1. |

Raises | |
---|---|

`TypeError` | If `grads_and_vars` is malformed. |

`ValueError` | If none of the variables have gradients. |

`from_config`

@classmethod from_config( config, custom_objects=None )

Creates an optimizer from its config.

This method is the reverse of `get_config`

, capable of instantiating the same optimizer from the config dictionary.

Arguments | |
---|---|

`config` | A Python dictionary, typically the output of get_config. |

`custom_objects` | A Python dictionary mapping names to additional Python objects used to create this optimizer, such as a function used for a hyperparameter. |

Returns | |
---|---|

An optimizer instance. |

`get_config`

get_config()

Returns the config of the optimimizer.

An optimizer config is a Python dictionary (serializable) containing the configuration of an optimizer. The same optimizer can be reinstantiated later (without any saved state) from this configuration.

Returns | |
---|---|

Python dictionary. |

`get_gradients`

get_gradients( loss, params )

Returns gradients of `loss`

with respect to `params`

.

Arguments | |
---|---|

`loss` | Loss tensor. |

`params` | List of variables. |

Returns | |
---|---|

List of gradient tensors. |

Raises | |
---|---|

`ValueError` | In case any gradient cannot be computed (e.g. if gradient function not implemented). |

`get_slot`

get_slot( var, slot_name )

`get_slot_names`

get_slot_names()

A list of names for this optimizer's slots.

`get_updates`

get_updates( loss, params )

`get_weights`

get_weights()

`minimize`

minimize( loss, var_list, grad_loss=None, name=None )

Minimize `loss`

by updating `var_list`

.

This method simply computes gradient using `tf.GradientTape`

and calls `apply_gradients()`

. If you want to process the gradient before applying then call `tf.GradientTape`

and `apply_gradients()`

explicitly instead of using this function.

Args | |
---|---|

`loss` | A callable taking no arguments which returns the value to minimize. |

`var_list` | list or tuple of `Variable` objects to update to minimize `loss` , or a callable returning the list or tuple of `Variable` objects. Use callable when the variable list would otherwise be incomplete before `minimize` since the variables are created at the first time `loss` is called. |

`grad_loss` | Optional. A `Tensor` holding the gradient computed for `loss` . |

`name` | Optional name for the returned operation. |

Returns | |
---|---|

An Operation that updates the variables in `var_list` . If `global_step` was not `None` , that operation also increments `global_step` . |

Raises | |
---|---|

`ValueError` | If some of the variables are not `Variable` objects. |

`set_weights`

set_weights( weights )

`variables`

variables()

Returns variables of this Optimizer based on the order created.

© 2020 The TensorFlow Authors. All rights reserved.

Licensed under the Creative Commons Attribution License 3.0.

Code samples licensed under the Apache 2.0 License.

https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/keras/optimizers/RMSprop