tf.contrib.kfac.fisher_blocks.NaiveDiagonalFB

Class NaiveDiagonalFB

Inherits From: FisherBlock

Defined in tensorflow/contrib/kfac/python/ops/fisher_blocks.py.

FisherBlock using a diagonal matrix approximation.

This type of approximation is generically applicable but quite primitive.

Note that this uses the naive "square the sum estimator", and so is applicable to any type of parameter in principle, but has very high variance.

Properties

num_registered_towers

Number of towers registered for this FisherBlock.

Typically equal to the number of towers in a multi-tower setup.

Methods

__init__

__init__(
    layer_collection,
    params
)

Creates a NaiveDiagonalFB block.

Args:

• layer_collection: The collection of all layers in the K-FAC approximate Fisher information matrix to which this FisherBlock belongs.
• params: The parameters of this layer (Tensor or tuple of Tensors).

full_fisher_block

full_fisher_block()

instantiate_factors

instantiate_factors(
    grads_list,
    damping
)

Creates and registers the component factors of this Fisher block.

Args:

• grads_list: A list gradients (each a Tensor or tuple of Tensors) with respect to the tensors returned by tensors_to_compute_grads() that are to be used to estimate the block.
• damping: The damping factor (float or Tensor).

multiply

multiply(vector)

Multiplies the vector by the (damped) block.

Args:

• vector: The vector (a Tensor or tuple of Tensors) to be multiplied.

Returns:

The vector left-multiplied by the (damped) block.

multiply_inverse

multiply_inverse(vector)

Multiplies the vector by the (damped) inverse of the block.

Args:

• vector: The vector (a Tensor or tuple of Tensors) to be multiplied.

Returns:

The vector left-multiplied by the (damped) inverse of the block.

multiply_matpower

multiply_matpower(
    vector,
    exp
)

Multiplies the vector by the (damped) matrix-power of the block.

Args:

• vector: The vector (a Tensor or tuple of Tensors) to be multiplied.
• exp: A float representing the power to raise the block by before multiplying it by the vector.

Returns:

The vector left-multiplied by the (damped) matrix-power of the block.

register_additional_tower

register_additional_tower(batch_size)

Register an additional tower.

Args:

• batch_size: The batch size, used in the covariance estimator.

register_inverse

register_inverse()

Registers a matrix inverse to be computed by the block.

register_matpower

register_matpower(exp)

Registers a matrix power to be computed by the block.

Args:

• exp: A float representing the power to raise the block by.

tensors_to_compute_grads

tensors_to_compute_grads()

Returns the Tensor(s) with respect to which this FisherBlock needs grads.