View source on GitHub |
ClusterResolver for Kubernetes.
Inherits From: ClusterResolver
tf.distribute.cluster_resolver.KubernetesClusterResolver( job_to_label_mapping=None, tf_server_port=8470, rpc_layer='grpc', override_client=None )
This is an implementation of cluster resolvers for Kubernetes. When given the the Kubernetes namespace and label selector for pods, we will retrieve the pod IP addresses of all running pods matching the selector, and return a ClusterSpec based on that information.
Note: it cannot retrievetask_type
,task_id
orrpc_layer
. To use it with some distribution strategies liketf.distribute.experimental.MultiWorkerMirroredStrategy
, you will need to specifytask_type
andtask_id
by setting these attributes.
Usage example with tf.distribute.Strategy:
# On worker 0 cluster_resolver = KubernetesClusterResolver( {"worker": ["job-name=worker-cluster-a", "job-name=worker-cluster-b"]}) cluster_resolver.task_type = "worker" cluster_resolver.task_id = 0 strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy( cluster_resolver=cluster_resolver) # On worker 1 cluster_resolver = KubernetesClusterResolver( {"worker": ["job-name=worker-cluster-a", "job-name=worker-cluster-b"]}) cluster_resolver.task_type = "worker" cluster_resolver.task_id = 1 strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy( cluster_resolver=cluster_resolver)
Args | |
---|---|
job_to_label_mapping | A mapping of TensorFlow jobs to label selectors. This allows users to specify many TensorFlow jobs in one Cluster Resolver, and each job can have pods belong with different label selectors. For example, a sample mapping might be {'worker': ['job-name=worker-cluster-a', 'job-name=worker-cluster-b'], 'ps': ['job-name=ps-1', 'job-name=ps-2']} |
tf_server_port | The port the TensorFlow server is listening on. |
rpc_layer | (Optional) The RPC layer TensorFlow should use to communicate between tasks in Kubernetes. Defaults to 'grpc'. |
override_client | The Kubernetes client (usually automatically retrieved using from kubernetes import client as k8sclient ). If you pass this in, you are responsible for setting Kubernetes credentials manually. |
Raises | |
---|---|
ImportError | If the Kubernetes Python client is not installed and no override_client is passed in. |
RuntimeError | If autoresolve_task is not a boolean or a callable. |
Attributes | |
---|---|
environment | Returns the current environment which TensorFlow is running in. There are two possible return values, "google" (when TensorFlow is running in a Google-internal environment) or an empty string (when TensorFlow is running elsewhere). If you are implementing a ClusterResolver that works in both the Google environment and the open-source world (for instance, a TPU ClusterResolver or similar), you will have to return the appropriate string depending on the environment, which you will have to detect. Otherwise, if you are implementing a ClusterResolver that will only work in open-source TensorFlow, you do not need to implement this property. |
task_id | Returns the task id this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task id, which is the index of the instance within its task type. This is useful when user needs to run specific code according to task index. For example, cluster_spec = tf.train.ClusterSpec({ "ps": ["localhost:2222", "localhost:2223"], "worker": ["localhost:2224", "localhost:2225", "localhost:2226"] }) # SimpleClusterResolver is used here for illustration; other cluster # resolvers may be used for other source of task type/id. simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker", task_id=0) ... if cluster_resolver.task_type == 'worker' and cluster_resolver.task_id == 0: # Perform something that's only applicable on 'worker' type, id 0. This # block will run on this particular instance since we've specified this # task to be a 'worker', id 0 in above cluster resolver. else: # Perform something that's only applicable on other ids. This block will # not run on this particular instance. Returns For more information, please see |
task_type | Returns the task type this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task type. Valid task types in TensorFlow include 'chief': a worker that is designated with more responsibility, 'worker': a regular worker for training/evaluation, 'ps': a parameter server, or 'evaluator': an evaluator that evaluates the checkpoints for metrics. See Multi-worker configuration for more information about 'chief' and 'worker' task type, which are most commonly used. Having access to such information is useful when user needs to run specific code according to task types. For example, cluster_spec = tf.train.ClusterSpec({ "ps": ["localhost:2222", "localhost:2223"], "worker": ["localhost:2224", "localhost:2225", "localhost:2226"] }) # SimpleClusterResolver is used here for illustration; other cluster # resolvers may be used for other source of task type/id. simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker", task_id=1) ... if cluster_resolver.task_type == 'worker': # Perform something that's only applicable on workers. This block # will run on this particular instance since we've specified this task to # be a worker in above cluster resolver. elif cluster_resolver.task_type == 'ps': # Perform something that's only applicable on parameter servers. This # block will not run on this particular instance. Returns For more information, please see |
cluster_spec
cluster_spec()
Returns a ClusterSpec object based on the latest info from Kubernetes.
We retrieve the information from the Kubernetes master every time this method is called.
Returns | |
---|---|
A ClusterSpec containing host information returned from Kubernetes. |
Raises | |
---|---|
RuntimeError | If any of the pods returned by the master is not in the Running phase. |
master
master( task_type=None, task_id=None, rpc_layer=None )
Returns the master address to use when creating a session.
You must have set the task_type and task_id object properties before calling this function, or pass in the task_type
and task_id
parameters when using this function. If you do both, the function parameters will override the object properties.
Note: this is only useful for TensorFlow 1.x.
Args | |
---|---|
task_type | (Optional) The type of the TensorFlow task of the master. |
task_id | (Optional) The index of the TensorFlow task of the master. |
rpc_layer | (Optional) The RPC protocol for the given cluster. |
Returns | |
---|---|
The name or URL of the session master. |
num_accelerators
num_accelerators( task_type=None, task_id=None, config_proto=None )
Returns the number of accelerator cores per worker.
This returns the number of accelerator cores (such as GPUs and TPUs) available per worker.
Optionally, we allow callers to specify the task_type, and task_id, for if they want to target a specific TensorFlow task to query the number of accelerators. This is to support heterogenous environments, where the number of accelerators cores per host is different.
Args | |
---|---|
task_type | (Optional) The type of the TensorFlow task of the machine we want to query. |
task_id | (Optional) The index of the TensorFlow task of the machine we want to query. |
config_proto | (Optional) Configuration for starting a new session to query how many accelerator cores it has. |
Returns | |
---|---|
A map of accelerator types to number of cores. |
© 2020 The TensorFlow Authors. All rights reserved.
Licensed under the Creative Commons Attribution License 3.0.
Code samples licensed under the Apache 2.0 License.
https://www.tensorflow.org/versions/r2.4/api_docs/python/tf/distribute/cluster_resolver/KubernetesClusterResolver