tf.contrib.kfac.fisher_blocks.FullyConnectedKFACBasicFB

Class FullyConnectedKFACBasicFB

Inherits From: KroneckerProductFB

Defined in tensorflow/contrib/kfac/python/ops/fisher_blocks.py.

K-FAC FisherBlock for fully-connected (dense) layers.

This uses the Kronecker-factorized approximation from the original K-FAC paper (https://arxiv.org/abs/1503.05671)

Properties

num_registered_towers

Methods

__init__

__init__(
    layer_collection,
    has_bias=False
)

Creates a FullyConnectedKFACBasicFB block.

Args:

• layer_collection: The collection of all layers in the K-FAC approximate Fisher information matrix to which this FisherBlock belongs.
• has_bias: Whether the component Kronecker factors have an additive bias. (Default: False)

full_fisher_block

full_fisher_block()

Explicitly constructs the full Fisher block.

Used for testing purposes. (In general, the result may be very large.)

Returns:

The full Fisher block.

instantiate_factors

instantiate_factors(
    grads_list,
    damping
)

Instantiate Kronecker Factors for this FisherBlock.

Args:

• grads_list: List of list of Tensors. grads_list[i][j] is the gradient of the loss with respect to 'outputs' from source 'i' and tower 'j'. Each Tensor has shape [tower_minibatch_size, output_size].
• damping: 0-D Tensor or float. 'damping' * identity is approximately added to this FisherBlock's Fisher approximation.

multiply

multiply(vector)

Multiplies the vector by the (damped) block.

Args:

• vector: The vector (a Tensor or tuple of Tensors) to be multiplied.

Returns:

The vector left-multiplied by the (damped) block.

multiply_inverse

multiply_inverse(vector)

Multiplies the vector by the (damped) inverse of the block.

Args:

• vector: The vector (a Tensor or tuple of Tensors) to be multiplied.

Returns:

The vector left-multiplied by the (damped) inverse of the block.

multiply_matpower

multiply_matpower(
    vector,
    exp
)

Multiplies the vector by the (damped) matrix-power of the block.

Args:

• vector: The vector (a Tensor or tuple of Tensors) to be multiplied.
• exp: A float representing the power to raise the block by before multiplying it by the vector.

Returns:

The vector left-multiplied by the (damped) matrix-power of the block.

register_additional_tower

register_additional_tower(
    inputs,
    outputs
)

register_inverse

register_inverse()

Registers a matrix inverse to be computed by the block.

register_matpower

register_matpower(exp)

Registers a matrix power to be computed by the block.

Args:

• exp: A float representing the power to raise the block by.

tensors_to_compute_grads

tensors_to_compute_grads()

Tensors to compute derivative of loss with respect to.