View source on GitHub |

Optimizer that implements the Adadelta algorithm.

Inherits From: `Optimizer`

tf.keras.optimizers.Adadelta( learning_rate=0.001, rho=0.95, epsilon=1e-07, name='Adadelta', **kwargs )

Adadelta optimization is a stochastic gradient descent method that is based on adaptive learning rate per dimension to address two drawbacks:

1) the continual decay of learning rates throughout training 2) the need for a manually selected global learning rate

Two accumulation steps are required:

1) the accumulation of gradients squared, 2) the accumulation of updates squared.

$$E[g^2]_0 := 0 \text{(Initialize gradient 2nd order moment vector)}$$

$$E[\Delta x^2]_0 := 0 \text{(Initialize 2nd order variable update)}$$

$$t := t + 1$$

$$E[g^2]_t := \rho * E[g^2]_{t-1} + (1 - \rho) * g^2$$

$$\Delta x_t = -RMS[\Delta x]_{t-1} * g_t / RMS[g]_t$$

$$E[\Delta x^2]_t := \rho * E[\Delta x^2]_{t-1} + (1 - \rho) * \Delta x_t^2$$

$$x_t := x_{t-1} + \Delta x_{t}$$

References See M. D. Zeiler (pdf)

Args | |
---|---|

`learning_rate` | A `Tensor` or a floating point value. The learning rate. To match the exact form in the original paper use 1.0. |

`rho` | A `Tensor` or a floating point value. The decay rate. |

`epsilon` | A `Tensor` or a floating point value. A constant epsilon used to better conditioning the grad update. |

`name` | Optional name prefix for the operations created when applying gradients. Defaults to "Adadelta". |

`**kwargs` | keyword arguments. Allowed to be {`clipnorm` , `clipvalue` , `lr` , `decay` }. `clipnorm` is clip gradients by norm; `clipvalue` is clip gradients by value, `decay` is included for backward compatibility to allow time inverse decay of learning rate. `lr` is included for backward compatibility, recommended to use `learning_rate` instead. |

Attributes | |
---|---|

`iterations` | Variable. The number of training steps this Optimizer has run. |

`weights` | Returns variables of this Optimizer based on the order created. |

`add_slot`

add_slot( var, slot_name, initializer='zeros' )

Add a new slot variable for `var`

.

`add_weight`

add_weight( name, shape, dtype=None, initializer='zeros', trainable=None, synchronization=tf.VariableSynchronization.AUTO, aggregation=tf.VariableAggregation.NONE )

`apply_gradients`

apply_gradients( grads_and_vars, name=None )

Apply gradients to variables.

This is the second part of `minimize()`

. It returns an `Operation`

that applies gradients.

Args | |
---|---|

`grads_and_vars` | List of (gradient, variable) pairs. |

`name` | Optional name for the returned operation. Default to the name passed to the `Optimizer` constructor. |

Returns | |
---|---|

An `Operation` that applies the specified gradients. The `iterations` will be automatically increased by 1. |

Raises | |
---|---|

`TypeError` | If `grads_and_vars` is malformed. |

`ValueError` | If none of the variables have gradients. |

`from_config`

@classmethod from_config( config, custom_objects=None )

Creates an optimizer from its config.

This method is the reverse of `get_config`

, capable of instantiating the same optimizer from the config dictionary.

Arguments | |
---|---|

`config` | A Python dictionary, typically the output of get_config. |

`custom_objects` | A Python dictionary mapping names to additional Python objects used to create this optimizer, such as a function used for a hyperparameter. |

Returns | |
---|---|

An optimizer instance. |

`get_config`

get_config()

Returns the config of the optimimizer.

An optimizer config is a Python dictionary (serializable) containing the configuration of an optimizer. The same optimizer can be reinstantiated later (without any saved state) from this configuration.

Returns | |
---|---|

Python dictionary. |

`get_gradients`

get_gradients( loss, params )

Returns gradients of `loss`

with respect to `params`

.

Arguments | |
---|---|

`loss` | Loss tensor. |

`params` | List of variables. |

Returns | |
---|---|

List of gradient tensors. |

Raises | |
---|---|

`ValueError` | In case any gradient cannot be computed (e.g. if gradient function not implemented). |

`get_slot`

get_slot( var, slot_name )

`get_slot_names`

get_slot_names()

A list of names for this optimizer's slots.

`get_updates`

get_updates( loss, params )

`get_weights`

get_weights()

`minimize`

minimize( loss, var_list, grad_loss=None, name=None )

Minimize `loss`

by updating `var_list`

.

This method simply computes gradient using `tf.GradientTape`

and calls `apply_gradients()`

. If you want to process the gradient before applying then call `tf.GradientTape`

and `apply_gradients()`

explicitly instead of using this function.

Args | |
---|---|

`loss` | A callable taking no arguments which returns the value to minimize. |

`var_list` | list or tuple of `Variable` objects to update to minimize `loss` , or a callable returning the list or tuple of `Variable` objects. Use callable when the variable list would otherwise be incomplete before `minimize` since the variables are created at the first time `loss` is called. |

`grad_loss` | Optional. A `Tensor` holding the gradient computed for `loss` . |

`name` | Optional name for the returned operation. |

Returns | |
---|---|

An Operation that updates the variables in `var_list` . If `global_step` was not `None` , that operation also increments `global_step` . |

Raises | |
---|---|

`ValueError` | If some of the variables are not `Variable` objects. |

`set_weights`

set_weights( weights )

`variables`

variables()

Returns variables of this Optimizer based on the order created.

© 2020 The TensorFlow Authors. All rights reserved.

Licensed under the Creative Commons Attribution License 3.0.

Code samples licensed under the Apache 2.0 License.

https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/keras/optimizers/Adadelta