EviCLIP: Bridging the Semantic Gap in Multimodal Movie Review Helpfulness

This sentence carries enormous semantic weight — yet a conventional transformer model can only parse its lexical surface. It registers positive sentiment and descriptive intensity, but the true meaning lives in a specific cinematic frame: a spacecraft alone in the void of space. That frame is completely absent from the text.

In recommender systems, review helpfulness prediction is a critical ranking signal. Existing multimodal models — UNITER, ViLBERT, MCR — share a fatal assumption: that any image paired with the text is inherently relevant. Pair an Interstellar review discussing a desolate wide shot with the generic movie poster, and you inject noise rather than insight. This is the Semantic Gap.

Formally, a unimodal model estimates:

But the review implicitly references a latent visual variable $v^*$ — the mental image invoked by the reviewer. The correct formulation should be:

Prior multimodal systems approximate this as $P(y \mid r, \tilde{v})$ where $\tilde{v} \neq v^*$. We define the Semantic Gap as:

We curated 40,000+ IMDb user reviews across 10 films spanning diverse narrative genres — sci-fi, fantasy, crime, superhero, and drama — paired with 24,000+ high-resolution keyframes extracted via shot-boundary detection. Each film was processed in its own Jupyter notebook.

Helpfulness Score & Binarisation

Each review carries a raw helpfulness ratio from IMDb community votes. Let $H$ be helpful votes and $U$ unhelpful votes:

$S$ is heteroscedastic — a review with 2/2 votes and 20/20 votes both score 1.0, but the latter carries far more statistical weight. We therefore excluded all samples with $H + U < 5$ to stabilise the target variable.

We binarised via threshold $\tau$, optimised for Balanced Accuracy across the range $\tau \in [0.4, 0.7]$:

The architectural centrepiece of EviCLIP is that image selection must be dynamic, not static. For each review $r_i$, we retrieve the single most semantically aligned frame $f^*$ from the $N$ candidate frames of its corresponding film.

Using CLIP's dual encoders, we project both modalities into a shared $d$-dimensional embedding space:

Semantic alignment is measured by cosine similarity, and frame selection becomes a nearest-neighbour retrieval problem:

CLIP's contrastive training objective ensures matched text-image pairs cluster in embedding space:

import torch
import torch.nn.functional as F

def get_optimal_frame(review_emb: torch.Tensor,
                        frame_embs: torch.Tensor) -> tuple:
    """
    Select the keyframe most semantically aligned with the review.
    
    Args:
        review_emb  : (1, 512)  – CLIP text embedding
        frame_embs  : (N, 512)  – CLIP image embeddings for N candidate frames
    Returns:
        (best_idx, alignment_score)
    """
    # L2-normalise both modalities
    review_emb = F.normalize(review_emb, p=2, dim=-1)
    frame_embs = F.normalize(frame_embs, p=2, dim=-1)

    # Cosine similarity matrix: (1, N)
    similarities = torch.matmul(review_emb, frame_embs.T)

    # Greedy argmax retrieval → f*
    best_idx   = torch.argmax(similarities, dim=-1)
    best_score = similarities[0, best_idx]

    return best_idx.item(), best_score.item()

Once the aligned pair $(r_i, f^*)$ is established, we encode each modality independently via dedicated transformers, following a late-interaction dual-encoder paradigm that preserves modality-specific abstraction before fusion.

Text: BERT

Vision: ViT

Both encoders project into the same $\mathbb{R}^{768}$ semantic manifold. Naïve concatenation $h_{\text{concat}} = [h_{\text{text}};\, h_{\text{visual}}]$ implicitly assumes equal modality reliability — a critical oversimplification that motivates our evidential fusion stage.

In high-context cinematic domains, text and visual signals routinely disagree. A hyperbolic review of Inception may score 0.9 helpfulness from text alone — but if the retrieved frame is poorly aligned, forcing equal-weight fusion introduces systematic noise.

We resolve this via Dempster–Shafer Theory (DST), which models evidence as belief mass functions rather than scalar weights. Define the binary hypothesis frame:

Conflict Mass & Dempster's Rule

Worked Numerical Example

Suppose for an Inception review, the BERT encoder assigns high text confidence but ViT sees a poorly-aligned frame:

Training Objective

We benchmarked three fusion strategies — Evidential (DS-theory), Attention (cross-modal attention), and Simple (concatenation) — across five metrics: Accuracy, F1, AUC-ROC, MAP, and NDCG@5. Highlighted cells indicate the best-performing strategy per film per metric.

To isolate each architectural component's contribution, we systematically ablated modules and observed performance degradation on the Interstellar test set (AUC reported):

CLIP alignment contributes the largest single improvement (~7–8 AUC points), confirming that which image is chosen matters more than how it is fused. Evidential fusion contributes primarily to calibration stability and ranking robustness rather than raw separability.

Beyond accuracy, model reliability is measured via Expected Calibration Error (ECE) — the gap between predicted confidence and empirical accuracy across $M$ probability bins:

Evidential fusion reduced ECE compared to attention-based fusion across all evaluated films. The conflict mass $K$ functions as an implicit uncertainty detector: when modalities disagree ($K \uparrow$), fused belief mass redistributes toward the ignorance set $m(\Theta)$, producing conservative, appropriately uncertain predictions instead of overconfident scores.

Systematic error analysis revealed four primary failure clusters:

• Metaphorical language: Reviews like "a symphony of visual chaos" trigger CLIP to retrieve visually noisy frames with high cosine similarity but zero semantic alignment with the reviewer's intended reference.
• Non-visual reviews: Music, dialogue, or acting-focused critiques have no meaningful frame to retrieve — CLIP returns an arbitrary high-similarity frame.
• Extremely short reviews (<15 tokens) produce low-confidence CLIP embeddings with high distributional variance.
• Rare or visually sparse films with fewer than 500 keyframes reduce the quality of the retrieval pool.

Frame-alignment complexity per review is $\mathcal{O}(N_f \cdot d)$ where $N_f$ is the frame count and $d=512$ is the CLIP embedding dimension. Since frame embeddings are pre-computed offline, runtime cost reduces to a single matrix multiplication.

Future work may leverage approximate nearest-neighbour search (FAISS, HNSW) to scale alignment to sub-linear complexity for large frame collections.

• Designed and implemented the end-to-end CLIP dynamic alignment pipeline — including shot-boundary extraction, embedding pre-computation, and greedy frame retrieval.
• Derived and implemented the Dempster–Shafer evidential fusion layer from first principles, including conflict mass computation and belief normalisation.
• Engineered the threshold calibration sweep over $\tau \in [0.4, 0.7]$, identifying $\tau = 0.54$ as the optimal balanced-accuracy operating point.
• Conducted full cross-genre benchmarking across 10 films (individual Jupyter notebooks), interpreting per-genre performance variation and correlating it with visual/narrative density.
• Diagnosed gradient instability in early attention-fusion training via loss variance analysis; proposed and validated evidential normalisation as the stabilising mechanism.
• Developed the full visualisation dashboard for training dynamics, validation curves, ranking metric summaries, and genre-level performance comparisons.
• Co-authored the paper, contributing the theoretical framing of the Semantic Gap, all mathematical derivations, and the empirical analysis sections.

EviCLIP currently grounds each review on a single keyframe, ignoring temporal cinematographic context. For reviews referencing narrative arcs or editing rhythms, the correct formulation should condition on the full frame sequence:

Priority research directions:

• Temporal visual grounding via multi-frame aggregation or video-language pre-training
• Adaptive fusion weighting conditioned on predicted genre distribution
• ANN-based frame retrieval (FAISS) for sub-linear scaling to large film libraries
• Cross-cultural review style adaptation and debiasing for non-English IMDb content
• Extension to other review domains: books (cover art), music (album artwork), restaurants (photo galleries)

• [1] CLIP: Radford, A., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML 2021. arXiv:2103.00020
• [2] Dempster–Shafer Fusion: Alzahem, A., et al. (2024). Feature fusion for improved classification: Combining Dempster-Shafer theory and multiple CNN architectures. arXiv:2405.20230
• [3] BERT: Devlin, J., et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT 2019.
• [4] ViT: Dosovitskiy, A., et al. (2021). An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale. ICLR 2021.
• [5] MCR Baseline: Liu, J., et al. (2021). Multi-perspective Coherent Reasoning for Helpfulness Prediction of Multimodal Reviews. ACL 2021.
• [6] SANCL: (2022). Sentiment-Aware Multi-modal Dialogue Systems. COLING 2022. ACL Anthology
• [7] This Paper: Hamza, M., Rai, K., Sarkar, S. (2025). Movie Review Helpfulness Prediction: A Multimodal Framework. IEEE ISAI-NLP 2025. IEEE Xplore