Indicates how a distributed variable will be aggregated.
tf.distribute.Strategy
distributes a model by making multiple copies (called "replicas") acting data-parallel on different elements of the input batch. When performing some variable-update operation, say var.assign_add(x)
, in a model, we need to resolve how to combine the different values for x
computed in the different replicas.
NONE
: This is the default, giving an error if you use a variable-update operation with multiple replicas.SUM
: Add the updates across replicas.MEAN
: Take the arithmetic mean ("average") of the updates across replicas.ONLY_FIRST_REPLICA
: This is for when every replica is performing the same update, but we only want to perform the update once. Used, e.g., for the global step counter.ONLY_FIRST_TOWER
: Deprecated alias for ONLY_FIRST_REPLICA
.MEAN
NONE
ONLY_FIRST_REPLICA
SUM
© 2020 The TensorFlow Authors. All rights reserved.
Licensed under the Creative Commons Attribution License 3.0.
Code samples licensed under the Apache 2.0 License.
https://www.tensorflow.org/versions/r2.3/api_docs/python/tf/compat/v1/VariableAggregation