Transformer-XL
Transformer-XL is an advanced neural network architecture that extends the Transformer model by introducing recurrence and segment-level recurrence mechanisms to better capture long-range dependencies in sequential data.
Technology & Innovation
Hopfield network
A Hopfield network is a form of recurrent artificial neural network that serves as content-addressable (associative) memory, enabling the retrieval of stored patterns.
Self-supervised learning
Self-supervised learning is a machine learning paradigm that uses unlabeled data to train models, allowing them to learn useful representations without extensive human annotation.
PointMLP (point cloud MLP)
PointMLP is a neural network architecture designed for processing three-dimensional point cloud data using multi-layer perceptrons (MLPs). It aims to efficiently capture local and global geometric features for tasks such as classification and segmentation in 3D vision applications.
Deep Q-network (DQN) for autonomous driving
The Deep Q-Network (DQN) is a pivotal reinforcement learning architecture applied in autonomous driving, enabling vehicles to make decisions based on visual inputs.
Relational graph convolutional network (R-GCN)
A Relational Graph Convolutional Network (R-GCN) is a type of neural network designed to operate on graph-structured data with multiple types of edges, enabling effective learning over relational data. It extends traditional graph convolutional networks by incorporating relation-specific transformations, making it particularly useful for knowledge graphs and multi-relational data.
Monte Carlo dropout for uncertainty estimation
Monte Carlo dropout is a technique used in deep learning to estimate model uncertainty by performing stochastic forward passes with dropout enabled during inference. This approach allows neural networks to approximate Bayesian inference and quantify predictive uncertainty without major changes to the model architecture.
Semi-supervised learning
Semi-supervised learning is a machine learning approach that uses both labeled and unlabeled data to improve model accuracy, bridging the gap between supervised and unsupervised learning.
ByT5
ByT5 is a transformer-based model designed for natural language processing tasks that operates directly on byte-level input rather than traditional tokenized text. It aims to improve language understanding across multiple languages and domains by avoiding tokenization issues.
Causality in AI
Causality in AI refers to the understanding and modeling of cause-and-effect relationships within artificial intelligence systems.