Package me.damoebe.architectures.transformer.mha

Class MultiHeadAttention<H extends Head>

java.lang.Object

me.damoebe.architectures.transformer.mha.MultiHeadAttention<H>

Type Parameters:

H - The type of head that is used stored in the class

All Implemented Interfaces:

Cloneable, Backpropagation

public class MultiHeadAttention<H extends Head>
extends Object
implements Cloneable, Backpropagation

A container class for Heads

Constructor Summary

Constructors

Constructor	Description
MultiHeadAttention(Class<H> c, int headAmount, int inputEmbeddingAmount, int inputEmbeddingSize, boolean masked, double learningRate)	Main constructor for the MultiHeadAttention class

Method Summary

Modifier and Type	Method	Description
double[][]	backPropagate(double[][] deltas)	Updates All head QKV weights as well as the weights which are used to merge the head matrices
MultiHeadAttention<H>	clone()	Clones a MultiHeadAttention object.
static double[][]	concatMatrices(List<double[][]> matrices)	Merges a list of matrices.
Sequence	generateOutputFor(Sequence[] input)	Generates an output for a list of inputs(embedding lists).
int	getEmbeddingAmount()	Getter for the used embedding amount of this MHA.
int	getEmbeddingSize()	Getter for the used embedding size of this MHA.
List<H>	getHeads()	Getter for the heads stored in the MultiHeadAttention

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

MultiHeadAttention

public MultiHeadAttention(Class<H> c,
 int headAmount,
 int inputEmbeddingAmount,
 int inputEmbeddingSize,
 boolean masked,
 double learningRate)

Main constructor for the MultiHeadAttention class

Parameters:

c - The class that is being used for the heads

headAmount - The amount of heads this MHA object should have

inputEmbeddingAmount - The amount of the input embeddings

inputEmbeddingSize - The size of the input embeddings

Method Details

generateOutputFor

public Sequence generateOutputFor(Sequence[] input)

Generates an output for a list of inputs(embedding lists).

Parameters:

input - An input as a list of embedding lists -> see EDHead doc if you are using a list size larger than 1

Returns:

The output embedding list

backPropagate

public double[][] backPropagate(double[][] deltas)

Updates All head QKV weights as well as the weights which are used to merge the head matrices

Specified by:

backPropagate in interface Backpropagation

Parameters:

deltas - The first hidden-layer deltas of the block's mlp

Returns:

The new deltas used for the next backpropagation step.

clone

public MultiHeadAttention<H> clone()

Clones a MultiHeadAttention object.

Overrides:

clone in class Object

Returns:

The cloned object.

concatMatrices

public static double[][] concatMatrices(List<double[][]> matrices)

Merges a list of matrices.

Parameters:

matrices - the matrices that will be concat

Returns:

the merged matrix as a 2d array

getEmbeddingAmount

public int getEmbeddingAmount()

Getter for the used embedding amount of this MHA.

Returns:

The embedding Amount

getEmbeddingSize

public int getEmbeddingSize()

Getter for the used embedding size of this MHA.

Returns:

the embedding size.

getHeads

public List<H> getHeads()

Getter for the heads stored in the MultiHeadAttention

Returns:

A list of the heads.