Multimodal Fusion

Strategies for combining embeddings from different data modalities (text, image, audio, user behavior).

Quick Start

from corerec.multimodal.fusion_strategies import AttentionFusion

fusion = AttentionFusion(modality_dims=[768, 512, 128], output_dim=256)
fused = fusion(text_emb, image_emb, audio_emb)

Fusion Strategies

Multi-modal fusion strategies for combining different data types.

Modern recsys uses text, images, audio, user behavior all together. This module provides ways to combine these embeddings.

class corerec.multimodal.fusion_strategies.FusionStrategy(*values)

Bases: Enum

Different ways to combine modalities.

CONCAT = 'concat'

WEIGHTED = 'weighted'

ATTENTION = 'attention'

GATED = 'gated'

BILINEAR = 'bilinear'

TUCKER = 'tucker'

class corerec.multimodal.fusion_strategies.ConcatFusion(*args: Any, **kwargs: Any)

Bases: Module

Simplest fusion: concatenate embeddings and project. Fast but doesn’t model cross-modal interactions well.

forward(embeddings: Dict[str, torch.Tensor]) → torch.Tensor

embeddings: dict with keys matching input_dims, values are [batch, dim] tensors

class corerec.multimodal.fusion_strategies.WeightedFusion(*args: Any, **kwargs: Any)

Bases: Module

Learned weighted sum of modalities. Good when modalities have different importance.

forward(embeddings: Dict[str, torch.Tensor]) → torch.Tensor

embeddings: dict with [batch, embedding_dim] tensors All embeddings must have same dimension.

class corerec.multimodal.fusion_strategies.AttentionFusion(*args: Any, **kwargs: Any)

Bases: Module

Use attention to dynamically weight modalities. More flexible than fixed weights, adapts per instance.

forward(embeddings: Dict[str, torch.Tensor]) → torch.Tensor

Use attention to aggregate modality embeddings.

class corerec.multimodal.fusion_strategies.GatedFusion(*args: Any, **kwargs: Any)

Bases: Module

Gated fusion - learns to filter/select from each modality.

Used in VQA and other multi-modal tasks. Each modality gets a gate that controls its contribution.

forward(embeddings: Dict[str, torch.Tensor]) → torch.Tensor

Apply gating to each modality then combine.

class corerec.multimodal.fusion_strategies.BilinearFusion(*args: Any, **kwargs: Any)

Bases: Module

Bilinear pooling for two modalities. Captures multiplicative interactions between features.

Commonly used for VQA (vision + language).

forward(emb1: torch.Tensor, emb2: torch.Tensor) → torch.Tensor

emb1: [batch, dim1] emb2: [batch, dim2]

class corerec.multimodal.fusion_strategies.MultiModalFusion(*args: Any, **kwargs: Any)

Bases: Module

Unified interface for multi-modal fusion. Handles arbitrary number of modalities with flexible strategies.

forward(raw_embeddings: Dict[str, torch.Tensor]) → torch.Tensor

Fuse embeddings from different modalities.

raw_embeddings: dict with modality name -> [batch, modality_dim] tensor Returns: [batch, output_dim] fused representation