Package me.damoebe.architectures.transformer.mha

Class Head

java.lang.Object

me.damoebe.architectures.transformer.mha.Head

All Implemented Interfaces:

Backpropagation

Direct Known Subclasses:

EDHead

public class Head
extends Object
implements Backpropagation

The Decoder and Encoder Head Object class

Field Summary

Fields

Modifier and Type	Field	Description
protected List<double[][]>	biases	All biases of the head in one list, (size = inputEmbeddingSize*inputEmbeddingAmount*3) -> used to calculate the queries, keys and values.
protected final int	inputEmbeddingAmount	The amount of embeddings that are allowed to be inserted.
protected final int	inputEmbeddingSize	The vector dimension (size) that each inserted embedding must have
protected final double	learningRate
protected final boolean	masked
protected List<double[][]>	weights	All weights of the head in one list, (size = inputEmbeddingSize*inputEmbeddingAmount*3) -> used to calculate the queries, keys and values.

Constructor Summary

Constructors

Constructor	Description
Head(int inputEmbeddingAmount, int inputEmbeddingSize, boolean masked, double learningRate)	The main constructor of the decoder Head Identical to the super constructor

Method Summary

Modifier and Type	Method	Description
double[][]	backPropagate(double[][] deltas)	BACKPROPAGATION IMPLEMENTATION Mathematical Derivation: Forward Pass: 1.
protected List<double[][]>	generateQKVMatrices(Sequence[] inputEmbeddingLists)	Generates a list of query, key and value matrices based on the heads weights and biases.
int	getInputEmbeddingAmount()
int	getInputEmbeddingSize()
double[][]	getOutput()	Generates an output of this head by using the current attention, which is multiplied with the current value-vectors
void	insertInput(Sequence... inputEmbeddingLists)	Inserts an input sequence into this head and calculated the attention

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

weights

protected List<double[][]> weights

All weights of the head in one list, (size = inputEmbeddingSize*inputEmbeddingAmount*3) -> used to calculate the queries, keys and values.

biases

protected List<double[][]> biases

All biases of the head in one list, (size = inputEmbeddingSize*inputEmbeddingAmount*3) -> used to calculate the queries, keys and values.

inputEmbeddingSize

protected final int inputEmbeddingSize

The vector dimension (size) that each inserted embedding must have

inputEmbeddingAmount

protected final int inputEmbeddingAmount

The amount of embeddings that are allowed to be inserted.

masked

protected final boolean masked

learningRate

protected final double learningRate

Constructor Details

Head

public Head(int inputEmbeddingAmount,
 int inputEmbeddingSize,
 boolean masked,
 double learningRate)

The main constructor of the decoder Head Identical to the super constructor

Parameters:

inputEmbeddingAmount - The amount of input embeddings

inputEmbeddingSize - The size of each input embedding

Method Details

insertInput

public void insertInput(Sequence... inputEmbeddingLists)
                 throws Exception

Inserts an input sequence into this head and calculated the attention

Parameters:

inputEmbeddingLists - The input Embedding lists -> here it can only contain one embedding-list

Throws:

Exception - if the inserted input embeddings don't fit the expectations

generateQKVMatrices

protected List<double[][]> generateQKVMatrices(Sequence[] inputEmbeddingLists)
                                        throws Exception

Generates a list of query, key and value matrices based on the heads weights and biases.

Parameters:

inputEmbeddingLists - The input Embedding lists -> here it can only contain one embedding-list

Returns:

a list of query, key and value matrices

Throws:

Exception

backPropagate

public double[][] backPropagate(double[][] deltas)

BACKPROPAGATION IMPLEMENTATION Mathematical Derivation: Forward Pass: 1. Q = X * W_q + b_q (queries) 2. K = X * W_k + b_k (keys) 3. V = X * W_v + b_v (values) 4. scores = Q * K^T / sqrt(d_k) (scaled dot product) 5. attention = softmax(scores) (attention weights) 6. output = attention * V (head output) Backward Pass (Chain Rule): Given dL/dOutput from the next layer: 1. dL/dAttention = dL/dOutput * V^T 2. dL/dScores = softmax_derivative(scores) * dL/dAttention 3. dL/dQ = dL/dScores * K / sqrt(d_k) 4. dL/dK = dL/dScores^T * Q / sqrt(d_k) 5. dL/dV = dL/dOutput^T * attention 6. dL/dW_q = X^T * dL/dQ 7. dL/dW_k = X^T * dL/dK 8. dL/dW_v = X^T * dL/dV 9. dL/dInput = dL/dQ * W_q^T + dL/dK * W_k^T + dL/dV * W_v^T

Specified by:

backPropagate in interface Backpropagation

Parameters:

deltas - The delta from the mha of this head (dL/dOutput)

Returns:

The new deltas for the next backpropagation step (dL/dInput)

getOutput

public double[][] getOutput()

Generates an output of this head by using the current attention, which is multiplied with the current value-vectors

Returns:

The output for the input Embeddings as a matrix.

getInputEmbeddingSize

public int getInputEmbeddingSize()

getInputEmbeddingAmount

public int getInputEmbeddingAmount()