Package me.damoebe.architectures.transformer.mha

Klasse MultiHeadAttention<H extends Head>

java.lang.Object

me.damoebe.architectures.transformer.mha.MultiHeadAttention<H>

Typparameter:

H - The type of head that is used stored in the class

public class MultiHeadAttention<H extends Head>
extends Object

A container class for Heads

Konstruktorübersicht

Konstruktoren

Konstruktor	Beschreibung
MultiHeadAttention(Class<H> c, int headAmount, int inputEmbeddingAmounts, int inputEmbeddingSize, boolean masked)	Main constructor for the MultiHeadAttention class

Methodenübersicht

Modifizierer und Typ	Methode	Beschreibung
MultiHeadAttention<H>	clone()	Clones a MultiHeadAttention object.
static double[][]	concatMatrices(List<double[][]> matrices)	Merges a list of matrices.
Sequence	generateOutputFor(Sequence[] input)	Generates an output for a list of inputs(embedding lists).
int	getEmbeddingAmount()	Getter for the used embedding amount of this MHA.
int	getEmbeddingSize()	Getter for the used embedding size of this MHA.
List<H>	getHeads()	Getter for the heads stored in the MultiHeadAttention
void	updateAllWeights(double[][] deltas)	Updates All head QKV weights as well as the weights which are used to merge the head matrices

Von Klasse geerbte Methoden java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Konstruktordetails

MultiHeadAttention

public MultiHeadAttention(Class<H> c,
 int headAmount,
 int inputEmbeddingAmounts,
 int inputEmbeddingSize,
 boolean masked)

Main constructor for the MultiHeadAttention class

Parameter:

c - The class that is being used for the heads

headAmount - The amount of heads this MHA object should have

inputEmbeddingAmounts - The amount of the input embeddings

inputEmbeddingSize - The size of the input embeddings

Methodendetails

generateOutputFor

public Sequence generateOutputFor(Sequence[] input)

Generates an output for a list of inputs(embedding lists).

Parameter:

input - An input as a list of embedding lists -> see EDHead doc if you are using a list size larger than 1

Gibt zurück:

The output embedding list

concatMatrices

public static double[][] concatMatrices(List<double[][]> matrices)

Merges a list of matrices.

Parameter:

matrices - the matrices that will be concat

Gibt zurück:

the merged matrix as a 2d array

updateAllWeights

public void updateAllWeights(double[][] deltas)

Updates All head QKV weights as well as the weights which are used to merge the head matrices

Parameter:

deltas - The first hidden-layer deltas of the block's mlp

clone

public MultiHeadAttention<H> clone()

Clones a MultiHeadAttention object.

Setzt außer Kraft:

clone in Klasse Object

Gibt zurück:

The cloned object.

getEmbeddingAmount

public int getEmbeddingAmount()

Getter for the used embedding amount of this MHA.

Gibt zurück:

The embedding Amount

getEmbeddingSize

public int getEmbeddingSize()

Getter for the used embedding size of this MHA.

Gibt zurück:

the embedding size.

getHeads

public List<H> getHeads()

Getter for the heads stored in the MultiHeadAttention

Gibt zurück:

A list of the heads.