NeoMME Multimodal Encoder: Visual RAG for AI Video and Ecommerce

By VEONIB | 2026-10-17

Quick Answer

NeoMME is a family of 260M and 800M multilingual multimodal encoders from H company that process text tokens and raw image patches inside a single bidirectional Transformer, without a separate vision tower or a causal language model. Its retrieval variant, NeoMME-Retriever, ranks document page images directly, runs roughly twice as fast as ColModernVBERT at matched resolution, and reduces late-interaction index storage by 255× while retaining over 95% of baseline nDCG@10.

TL;DR

Table of Contents

According to NeoMME: an efficient Multimodal-native and Multilingual Encoder published by Hugging Face on 2026-09-03, H company has introduced a family of multilingual multimodal encoders that skip the conventional vision-tower-plus-language-model recipe. NeoMME ships in 260M and 800M parameter sizes, processes text tokens and raw image patches inside one bidirectional Transformer, and is trained from scratch with a masked discrete-diffusion objective. Its retrieval variant, NeoMME-Retriever, ranks document page images directly and bypasses OCR preprocessing entirely.

The release matters beyond academic document search. Visual retrieval is an underappreciated infrastructure layer in ecommerce content operations: it decides which product image, specification sheet, or packaging asset an AI system "sees" before it writes a script or generates a video. In this analysis, VEONIB examines what NeoMME changes technically, where it fits in AI video production pipelines, and which ecommerce teams should test it now rather than later.

Hero Image Alt Text: NeoMME multimodal encoder architecture for visual document retrieval and ecommerce AI video retrieval-augmented generation Caption: NeoMME processes text tokens and raw image patches in a single bidirectional Transformer, powering page-image retrieval for visual RAG. OG Image Title: NeoMME Multimodal Encoder — Visual RAG for AI Video and Ecommerce Suggested Visual: A clean technical diagram showing a document page image splitting into 32×32 patches and text tokens entering one shared Transformer encoder, with arrows pointing to dense and late-interaction embeddings feeding a video script generator.

What NeoMME Is and Why H Company Built It

NeoMME (pronounced "nee-oh-me") is described in the source as a multilingual, multimodal foundation encoder that generates vector representations for input text, images, or both using a single Transformer encoder. Original Fact: It is not based on an existing pretrained vision tower, text encoder, or text decoder. The two released checkpoints — NeoMME-260M and NeoMME-800M — are published under the Apache 2.0 license and integrated into Hugging Face Transformers, with a technical report registered as arXiv 2609.01657.

The motivation is architectural rather than cosmetic. Many visual document retrievers are adapted from pretrained generative visual language models: a vision encoder produces features, a projector maps them into the language model's space, and a causal decoder processes the combined representation. Retrieval, classification, and token labeling do not generate text autoregressively, so the causal decoder and its parameter budget are largely wasted.

Where ModernBERT improved bidirectional text encoders, and ModernVBERT paired a ModernBERT-style text encoder with a SigLIP2 vision tower, NeoMME removes the external vision tower altogether. The result is a model whose image path and text path share the same parameters — which simplifies pretraining, fine-tuning, parallelism, and serving across modalities.

VEONIB Insight

This matters because retrieval quality, not generation quality, increasingly bottlenecks ecommerce AI video output. A generator can only work with the product context it is given. When a merchant's content library contains PDF spec sheets, packaging renders, and lifestyle photography, an OCR-dependent retriever silently drops layout, font hierarchy, and table structure. NeoMME's page-image approach preserves those signals by design.

For AI video generation specifically, this points to a practical shift: treat asset retrieval as a first-class pipeline stage rather than a preprocessing chore. Businesses building product video systems should adopt visual retrieval now, because it is cheap to run and degrades gracefully. Teams with small catalogs and clean, text-heavy metadata can reasonably wait, since keyword search may already be sufficient below a few hundred SKUs.

Inside the NeoMME Architecture: One Transformer for Images and Text

Original Fact: NeoMME-260M and NeoMME-800M share identical architecture. Text inputs use factorized token embeddings, while images are divided into a grid of non-overlapping 32×32 patches and projected through a small MLP. Both token streams enter the same Transformer encoder. Because images keep their native aspect ratio and size, the model can allocate more tokens to an information-dense document page than to a smaller, simpler image.

The context length is 16,384 tokens, which the source notes is enough for up to two standard 3840×2160 4K UHD images. Attention is hybrid: most layers use symmetric sliding-window attention, while every sixth layer and the final layer apply global attention. This alternating pattern controls compute cost while preserving long-range coherence.

The encoder stack also incorporates grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. A from-scratch BPE tokenizer with a 131k-token vocabulary was trained on multilingual text, code, mathematics, and machine-produced image transcripts.

Suggested visual: a layer-stack diagram showing alternating sliding-window and global-attention blocks, with the every-sixth-layer global pattern highlighted.

VEONIB Insight

The architectural choice that matters most for production teams is dynamic image resolution. Ecommerce assets are wildly inconsistent: a 400×400 thumbnail, a 3000px packaging render, and a 1080×1920 vertical lifestyle shot all coexist in one catalog. A fixed-resolution encoder forces lossy downscaling that erases small print such as ingredient lists, sizing charts, or compliance labels — exactly the details that make product video claims accurate.

The catch is token budget. A 16,384-token context is generous for documents but tight for video. NeoMME is an encoder, not a temporal model: it has no frame-to-frame motion representation, so it cannot evaluate camera movement or clip continuity. Adopt it for retrieval and grounding, not for judging video output quality.

How NeoMME Learns: Masked Discrete-Diffusion Pretraining

Original Fact: NeoMME is pretrained from scratch as a discrete masked-diffusion text denoiser. For text-only examples, a corruption rate is sampled uniformly between 0 and 1, and each eligible token is masked independently at that rate. Multimodal examples use corruption rates between 0.3 and 1, keeping image patches visible while the model reconstructs masked text.

The design intent is to prevent shortcuts. With light masking, a model can recover a missing word from surrounding text alone — the source's own example is that "cat" plausibly completes "The [MASK] sat on the mat" without any image. Heavy masking removes that language-only signal and forces the model to rely on visible image evidence, producing image-grounded descriptions rather than plausible guesses.

Pretraining mixes multilingual text, code, mathematics, natural images, and document images. Each model processes about 524 billion packed input tokens, including 290 billion text-only tokens. That is a comparatively small text budget next to ModernBERT's roughly 2 trillion training tokens, which is why H company selected the NorMuon optimizer to improve data efficiency.

VEONIB Insight

The masked-diffusion framing explains an important commercial property: NeoMME is trained to extract evidence from pixels rather than to hallucinate fluent text. For ecommerce, that reduces a specific and expensive failure mode — video scripts that invent product attributes. A retrievable embedding grounded in the actual packaging image is a safer foundation for downstream generation than a caption model that optimizes for plausibility.

The data-efficiency angle is equally relevant. A 524B-token pretraining run is still expensive, but it is materially cheaper than the multi-trillion-token regime used by frontier multimodal models. For teams evaluating whether to fine-tune rather than merely call an API, a smaller, openly licensed backbone lowers both cost and legal friction.

NeoMME-Retriever: From OCR Pipelines to Page-Image Retrieval

Original Fact: NeoMME-Retriever applies the page-image methodology introduced by ColPali. Instead of retrieving text chunks extracted by OCR, it ranks document page screenshots. The source states this preserves layout, charts, tables, font type and size, and other visual cues that even a perfect OCR model would not capture.

The retriever uses a dual-head design that returns both dense and late-interaction embeddings in one forward pass. It achieves these results at compact model sizes: both NeoMME variants sit on the ViDoRe v3 Pareto frontier for nDCG@10 versus model size. At a matched 2048×2048 input size on an NVIDIA L40S GPU, the 260M model encodes roughly 51 pages per second — about twice ColModernVBERT's throughput.

Storage is the other headline. Hierarchical token pooling combined with asymmetric quantization reduces late-interaction index storage from approximately 1.5 MB to 6 kB per page, a 255× reduction, while retaining more than 95% of baseline nDCG@10. The models are also fine-tunable through Sentence Transformers, and a visual RAG demo is published as a Hugging Face Space.

VEONIB Insight

The 255× storage reduction is the number most likely to change deployment decisions. Late-interaction retrieval has historically been penalized for index bloat, which makes large-scale page-image search expensive to host. At 6 kB per page, indexing one million product pages implies roughly 6 GB of late-interaction vectors before overhead — a range that fits comfortably on a single well-provisioned server rather than a distributed vector cluster.

Practically, this moves visual retrieval from a research demo into a viable internal service for mid-market merchants. The recommended adoption path is incremental: index existing PDF spec sheets and packaging artwork first, measure retrieval precision against your current keyword search, then expand to full catalog coverage.

NeoMME vs ColPali and ColModernVBERT: A Retrieval Comparison

The table below compares retrieval architectures as described in the source. Cells marked "Not specified in the original source" are absent from the published material and should not be treated as measured values.

Model / Approach Backbone Design External Vision Tower Throughput (2048×2048, L40S) Late-Interaction Index Size Reported Strength
NeoMME-260M Single bidirectional Transformer No ~51 pages/second ~6 kB per page after pooling and quantization Best quality-per-parameter on ViDoRe v3
NeoMME-800M Single bidirectional Transformer No Not specified in the original source ~6 kB per page after pooling and quantization Higher-capacity variant, same architecture
ColModernVBERT ModernBERT-style text encoder plus vision tower Yes (SigLIP2) Roughly half of NeoMME-260M Not specified in the original source Baseline throughput comparison
ColPali-style VLM retrievers Vision encoder, projector, causal decoder Yes Not specified in the original source Baseline of roughly 1.5 MB per page before compression Origin of the page-image retrieval method
ModernVBERT Bidirectional text encoder plus pretrained vision tower Yes (SigLIP2) Not specified in the original source Not applicable Efficient bidirectional text encoding

Two caveats are worth stating plainly. First, throughput comparisons depend on hardware, batch size, and input resolution; the 51 pages/second figure is specific to an NVIDIA L40S at 2048×2048. Second, ViDoRe v3 evaluates visual document retrieval quality, not video generation quality — the benchmark says nothing about motion, temporal consistency, or commercial readiness for advertising.

VEONIB Insight

The competitive story is not "NeoMME beats ColPali." It is that retrieval architectures are converging on removing unnecessary generative machinery. ColPali established that page images beat extracted text for document search. NeoMME pushes the same logic one step further by removing the pretrained vision tower and the causal decoder, arguing that a bidirectional encoder is the correct inductive bias for a ranking task.

For buyers, this reframes vendor evaluation. Instead of asking which model scores highest on a leaderboard, ask which model meets a latency and storage budget at your catalog size. A model that is 5% better on nDCG@10 but 10× more expensive to index is the wrong choice for a 50,000-SKU Shopify store.

What Visual RAG Means for Ecommerce Teams

Retrieval-augmented generation (RAG) is usually discussed as a text technique: embed documents, retrieve relevant chunks, inject them into a prompt. Visual RAG extends the same pattern to page images and product photography, which is where most ecommerce product knowledge actually lives.

Consider what a merchant's knowledge base genuinely contains: supplier PDF spec sheets, annotated packaging dielines, sizing tables rendered as images, certification scans, and marketplace listing screenshots. A text-only RAG pipeline must OCR all of it, losing column alignment in tables and discarding visual emphasis. NeoMME-Retriever sidesteps OCR and ranks the page images themselves.

VEONIB Insight

The business case is risk reduction before it is creative uplift. Generative models, including OpenAI GPT systems, Google AI Gemini models, and Anthropic Claude, all produce better output when supplied with accurate retrieved context. Retrieval is the mechanism that converts a plausible-sounding script into a verifiable one.

Should teams adopt now? If your catalog exceeds a few thousand SKUs, or if your product claims carry regulatory weight — supplements, cosmetics, electronics — the answer is yes. If you sell a handful of simple products with clean structured data, the marginal gain is small, and effort is better spent on prompt design and human review workflows.

Fitting NeoMME Into the VEONIB AI Video Workflow

NeoMME is not a video model. It does not generate frames, animate avatars, clone voices, or render text overlays. Any expectation that it replaces tools such as Runway, Pika, HeyGen, or MiniMax Hailuo is misplaced. What it does provide is a reliable retrieval and grounding layer.

Mapping NeoMME onto a standard ecommerce video pipeline clarifies its role:

Pipeline Stage NeoMME Relevance Notes
Product URL None Input collection happens upstream
Product Analysis High Retrieves spec sheets, packaging images, and listing pages as visual evidence
Script Medium Supplies grounded source material for script generation
Storyboard Medium Can surface reference layouts and existing brand assets
Image Prompt Low to medium Helps select correct reference imagery
Video Prompt Low Text-centric; prompt authoring remains an LLM task
AI Video None Requires a generative video model
Voice None Requires a text-to-speech or voice-cloning system
Subtitle None Requires a rendering or captioning tool
Publishing None Platform integration layer

Two dimensions deserve honest treatment. Character and product consistency are properties of the generative model, not the retriever, so NeoMME cannot improve them directly — although better reference retrieval often improves consistency indirectly by giving the generator cleaner inputs. Text rendering quality, camera movement, and motion fidelity are likewise outside NeoMME's scope. Estimated cost efficiency is favorable: Apache 2.0 licensing plus ~51 pages/second throughput on a single L40S means indexing cost is dominated by one-time compute, not per-query fees.

Recommended ecommerce use cases include Amazon product videos where specification accuracy is audited, Shopify product pages requiring consistent brand assets across large catalogs, and product demo videos built from technical documentation. For UGC-style and lifestyle videos, NeoMME's contribution is smaller, since those formats depend more on performance and pacing than on documented specifications.

VEONIB Insight

The correct mental model is that NeoMME strengthens the front half of the pipeline and leaves the back half untouched. That is not a limitation to apologize for; it is the division of labor that makes end-to-end automation tractable. Generative video models are improving faster than retrieval infrastructure, so the retrieval layer is where durable, compounding gains currently sit.

Adopt NeoMME for product analysis and asset grounding today if you operate a large or document-heavy catalog. Wait if your bottleneck is creative quality or platform distribution rather than product knowledge accuracy — retrieval will not fix a bad hook or a miscast voice.

Recommendations

Shopify Merchants. Index supplier PDFs and brand guideline documents as page images rather than extracting text. Start with your top 200 revenue SKUs and measure whether generated scripts cite specifications more accurately than your current process.

Amazon Sellers. Prioritize retrieval for categories with compliance scrutiny, including supplements, cosmetics, and consumer electronics. Ground every generated claim in a retrievable page so listing accuracy can be audited after the fact.

AI Developers. Test the Apache 2.0 NeoMME checkpoints in Hugging Face Transformers before committing to a proprietary retrieval API. Benchmark at your actual catalog scale, and validate the 6 kB per page storage figure against your own document mix rather than assuming it transfers.

SaaS Founders. Treat visual retrieval as a product capability rather than an internal tool. Merchants will pay for accurate grounding, and a 255× index reduction materially improves your unit economics versus late-interaction baselines.

Content Marketers. Use visual RAG to build a single source of truth for product facts, then let writers and models draw from it. This reduces review cycles because reviewers check citations instead of re-verifying every claim.

Video Creators. Add a retrieval step before prompting any generator. Supplying a generator with the correct reference image is usually faster than iterating on prompt language to correct a wrong product depiction.

FAQ

What is NeoMME in simple terms? NeoMME is a multilingual multimodal encoder from H company that turns text and images into vector representations using a single bidirectional Transformer, without a separate vision tower or causal language model. Its retriever variant ranks document page images directly.

Can NeoMME generate videos? No. NeoMME is an encoder and retrieval model. It produces embeddings for search and grounding, and it does not generate images, video, audio, or text autoregressively.

How much faster is NeoMME-Retriever than ColModernVBERT? At a matched 2048×2048 input size on an NVIDIA L40S GPU, the 260M NeoMME model encodes about 51 pages per second, roughly twice ColModernVBERT's throughput according to the source article.

How much storage does a late-interaction index require with NeoMME? Hierarchical token pooling and asymmetric quantization reduce index storage from about 1.5 MB to 6 kB per page — 255× smaller — while retaining more than 95% of baseline nDCG@10.

Is NeoMME free to use commercially? Yes. All release model checkpoints are published under the Apache 2.0 license, as stated in the source article. Deployment costs are therefore dominated by your own compute.

Does NeoMME support non-English documents? The model is described as multilingual. Its BPE tokenizer uses a 131k-token vocabulary trained from scratch on multilingual text, code, mathematics, and machine-produced image transcripts.

References

Sources

Try VEONIB

VEONIB turns a product URL into product analysis, video scripts, storyboards, image prompts, video prompts, and finished AI marketing videos automatically. Teams that pair grounded retrieval with a structured generation pipeline can move from a single listing link to publishable creative without manual asset hunting. Explore the VEONIB AI video generator to see how the workflow fits your catalog.

Credibility Assessment

Information drawn directly from the source includes the NeoMME model sizes, license, architecture components, attention pattern, context length, tokenizer vocabulary, pretraining token counts, corruption-rate ranges, benchmark placement on ViDoRe v3, the 51 pages/second throughput figure at 2048×2048 on an NVIDIA L40S GPU, the roughly twofold throughput advantage over ColModernVBERT, and the 1.5 MB to 6 kB per page storage reduction with over 95% nDCG@10 retention.

Conclusions that are VEONIB's own analysis include the ecommerce adoption guidance, the pipeline-mapping table, the assessment of where NeoMME does and does not fit into AI video production, and the recommendations for merchants, sellers, developers, and creators. These reflect editorial judgment rather than published measurements.

Information that remains uncertain or unspecified includes exact throughput figures for ColPali-style retrievers and ModernVBERT, the 800M model's throughput, storage figures for competing approaches, pricing or hosting costs, and any direct relationship between retrieval quality and end-video performance metrics. Where the source did not provide a value, this article says so explicitly rather than estimating.