Compressive Transformer

The Compressive Transformer is a type of neural network architecture designed to improve long-range sequence modeling by compressing past hidden states to extend memory capacity. It enhances the Transformer model by maintaining a compressed memory of previous activations, enabling efficient handling of longer sequences.

Read More →

Defensive distillation

Defensive distillation is a technique used to enhance the robustness of machine learning models, particularly neural networks, against adversarial attacks by training them on softened output probabilities. It modifies the training process to reduce model sensitivity to small input perturbations that can cause misclassification.

Read More →

Self-consistency decoding

Self-consistency decoding is a method used in natural language processing and artificial intelligence to improve the accuracy of model-generated responses by aggregating multiple outputs and selecting the most consistent answer. This approach enhances the reliability of language models by addressing variability in their outputs.

Read More →