Introduction

One of the central goals of machine learning is learning useful representations of data. Interestingly, the most effective systems for representation learning are often not purely generative models. Instead of reconstructing images pixel-by-pixel, many successful approaches focus on extracting compact and meaningful embeddings that preserve the semantic content of the input.

A natural way to achieve this is through joint embedding architectures. The core idea is simple: if two images represent the same underlying scene — even after corruption, augmentation, or transformation — their representations should remain similar. This principle is at the heart of Siamese neural networks [1], one of the earliest representation learning frameworks.

Siamese networks were originally introduced for tasks such as fraudulent signature detection. The architecture consists of two identical neural networks that share weights and process two input images independently. Rather than generating images, the networks produce embedding vectors that capture the semantic structure of the inputs.

During training, the model receives both positive and negative pairs. Positive pairs contain signatures from the same person, while negative pairs contain forged signatures. The objective is to make embeddings from positive pairs as similar as possible and embeddings from negative pairs as different as possible. Once trained, the model can compare the embedding of a new signature against a reference signature to determine whether it is authentic.

What makes this approach powerful is that the network learns meaningful internal representations without explicitly generating or reconstructing images.

The Problem of Representation Collapse

Joint embedding methods, however, introduce an important challenge. If the objective is simply to maximize similarity between embeddings, the network may converge to a trivial solution where it outputs the same embedding vector for every input. This phenomenon is known as representation collapse.

Early Siamese approaches addressed this problem using what is now known as contrastive learning. By introducing both positive and negative examples, the model learns not only which samples should be similar, but also which should be distinct. Contrastive methods have been highly successful for both images and videos, but they often require large batch sizes, many negative samples, and significant computational resources.

“Because of the curse of dimensionality, in the worst case, the number of contrastive examples may grow exponentially with the dimension of the representation. This is the main reason why I argue against contrastive methods.”

— Yann LeCun, A Path Towards Autonomous Machine Intelligence [2]

From Neuroscience to Barlow Twins

At the same time, fully generative approaches were also found to be inefficient for self-supervised representation learning. This motivated researchers to search for alternative principles inspired by neuroscience.

One important inspiration came from the work of computational neuroscientist Horace Barlow, who hypothesized in 1961 that biological visual systems operate by reducing redundancy between neurons [3]. Later, researchers proposed applying this idea directly to representation learning.

In joint embedding architectures, the goal is already to make representations of different views of the same image similar. However, similarity alone is not sufficient, since it can still lead to collapse. Barlow’s idea suggests an additional constraint: reduce redundancy between different dimensions of the representation.

To measure this redundancy, the outputs of different neurons are compared using cross-correlation. By computing the correlations between all feature dimensions across two augmented views, we obtain a cross-correlation matrix. Ideally:

• corresponding dimensions should be highly correlated,

• different dimensions should remain decorrelated.

In other words, the cross-correlation matrix should approach the identity matrix. This simple but powerful idea became the foundation of Barlow Twins [4].

How the Barlow Twins Model Works

Barlow Twins learns representations using an encoder network \(f(\cdot)\) that maps an input image to an embedding vector. The encoder consists of a ResNet-50 backbone followed by a projection head composed of three linear layers with 8192 units each. The first two layers are followed by Batch Normalization and ReLU activations.

During training, we first sample a batch of images \(x\) from the dataset. For each image, we generate two randomly augmented views using a transformation function. These two views are denoted as \(x_1\) and \(x_2\). Both views are passed through the same encoder network to produce the embedding vectors:

\[ z_1 = f(x_1), \qquad z_2 = f(x_2) \]

The embeddings are then normalized along the batch dimension rather than the feature dimension, which is more commonly used in other representation learning methods.

The Barlow Twins objective function is defined as:

\[ L_{BT} \triangleq \sum_i (1 - C_{ii})^2 + \lambda \sum_i \sum_{j \neq i} C_{ij}^2 \]

where:

• \(\lambda\) is a positive constant that balances the importance of the invariance term and the redundancy reduction term.

• \(C\) is the cross-correlation matrix computed between the normalized embeddings \(z_1\) and \(z_2\).

\[ C_{ij} \triangleq \frac{\sum_b z^{A}_{b,i} z^{B}_{b,j}} {\sqrt{\sum_b (z^{A}_{b,i})^2}\sqrt{\sum_b (z^{B}_{b,j})^2}} \]

• The first term of the loss is the invariance term. It encourages the diagonal entries of the cross-correlation matrix to be equal to one. This makes corresponding embedding dimensions from two augmented views highly correlated, ensuring that the two representations remain similar.

• The second term is the redundancy reduction term. It encourages the off-diagonal entries of the cross-correlation matrix to be close to zero. As a result, different embedding dimensions become decorrelated, reducing redundancy and encouraging each dimension to capture distinct information.