Score 10,000 candidate passages against a query with full cross-attention and you are looking at tens of millions of forward passes. Reimers and Gurevych measured this directly with BERT: about 65 hours on a single V100 GPU to find the most similar pair in a 10,000-sentence set. The same search with a pre-indexed bi-encoder took about five seconds. Neither number is a flaw. Cross-encoder vs bi-encoder is not a quality contest; it is two different jobs wearing the same “encoder” name.
A bi-encoder embeds the query and each document separately into fixed vectors, compared later with cosine similarity, which is what makes a vector index searchable at scale. A cross-encoder feeds the query and one document into the model together, letting every token attend to every other token, which is more accurate but cannot be precomputed. RAG pipelines use a bi-encoder to retrieve broadly and a cross-encoder to rerank a short list.
Cross-encoder vs bi-encoder: why does retrieval need both?
The cross-encoder vs bi-encoder choice is really a choice about when the comparison happens. A bi-encoder runs the query and each document through the network independently, so every document in a corpus can be embedded once, stored, and reused for every future query. That is what makes an agentic RAG loop capable of hitting a million-document index in milliseconds: the expensive part happened at indexing time, not query time.
A cross-encoder cannot do that, because it needs both texts present before it produces a score. The self-attention that makes it accurate is exactly what makes it unindexable: there is no per-document vector to cache, only a score for one specific pairing. Run it against every document in a large corpus and the cost scales linearly with corpus size, for every single query.
| Property | Bi-encoder | Cross-encoder |
|---|---|---|
| How it scores | Cosine similarity of two separate vectors. | Single joint forward pass, one score. |
| Can it be indexed | Yes; documents embedded once, reused always. | No; requires the query at scoring time. |
| Cost per query | One embedding, then a vector lookup. | One forward pass per candidate document. |
| Typical accuracy | Good enough to shortlist candidates. | Higher; sees direct query-document interaction. |
How does a cross-encoder actually rerank a shortlist?
In practice the two run back to back. The bi-encoder pulls a wide net, commonly the top 50 to 100 candidates, and the cross-encoder rescoring only that shortlist keeps its linear cost small enough to matter. The Sentence Transformers CrossEncoder class exposes exactly this pattern: pass a query and a list of candidate passages, get back an ordered score per pair, using a pretrained model such as cross-encoder/ms-marco-MiniLM-L6-v2.
1from sentence_transformers import SentenceTransformer, CrossEncoder, util 2 3bi_encoder = SentenceTransformer("multi-qa-MiniLM-L6-cos-v1") 4cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2") 5 6# Stage 1: bi-encoder retrieval, precomputed corpus vectors 7corpus_vecs = bi_encoder.encode(corpus, convert_to_tensor=True) 8query_vec = bi_encoder.encode(query, convert_to_tensor=True) 9hits = util.semantic_search(query_vec, corpus_vecs, top_k=100)[0] 10 11# Stage 2: cross-encoder reranks only those 100 candidates 12shortlist = [corpus[h["corpus_id"]] for h in hits] 13ranked = cross_encoder.rank(query, shortlist) 14# => [{'corpus_id': 42, 'score': 8.61}, {'corpus_id': 7, 'score': 3.02}, ...]
Where does each one belong in the pipeline?
Use a bi-encoder anywhere the corpus is large and static relative to query volume: first-pass retrieval, deduplication, clustering. Reach for a cross-encoder only on a shortlist you have already narrowed, typically the top few dozen candidates a bi-encoder retrieval stage already surfaced. Skipping the bi-encoder and cross-encoding the full corpus does not buy better recall; it just moves the 65-hour number onto your query path.
The MS MARCO passage ranking dataset, the standard cross-encoder training set, holds over 8.8 million passages behind roughly 500,000 real search queries. No production system cross-encodes against that whole collection per query. Every public reranker was trained expecting a shortlist, not a corpus.
- A bi-encoder embeds query and document separately, which is what allows the document vectors to be precomputed and indexed.
- A cross-encoder scores a query-document pair jointly through full attention, which is more accurate but cannot be cached per document.
- Reimers and Gurevych’s benchmark on 10,000 sentences puts the gap at roughly 65 hours for a cross-encoder against about 5 seconds for a bi-encoder.
- Standard practice retrieves a shortlist with a bi-encoder, then reranks only that shortlist with a cross-encoder.
- Cross-encoding an entire corpus is not a more thorough search; it is the retrieval step and the reranking step collapsed into the slowest possible version of both.
Conclusion
Pick the bi-encoder for anything that has to scale with corpus size, and the cross-encoder for anything that has to scale with precision on a short list. Getting that split wrong shows up as either a slow endpoint or a search that never quite returns the right passage. For the retrieval stage this reranker sits downstream of, see RAG architecture: a complete guide.
Frequently Asked Questions
What is the difference between a cross-encoder and a bi-encoder?
A bi-encoder embeds a query and a document separately into fixed-size vectors, which are compared afterward with cosine similarity. A cross-encoder feeds the query and document into the model together, letting every token attend to every other token, and outputs a single relevance score. Bi-encoders can be indexed; cross-encoders cannot.
Why can't a cross-encoder be used for first-pass retrieval?
A cross-encoder requires the query to be present before it can score a document, so nothing about a document can be precomputed or stored in an index. Scoring an entire corpus this way means one forward pass per document per query, which scales linearly with corpus size and becomes far too slow for real-time search at any meaningful scale.
How much slower is a cross-encoder than a bi-encoder?
Reimers and Gurevych benchmarked this directly in the Sentence-BERT paper. Finding the most similar pair in 10,000 sentences took about 65 hours with a BERT cross-encoder on a single GPU, versus about 5 seconds with a bi-encoder over precomputed embeddings, while reaching comparable accuracy on the underlying task.
How many candidates should a cross-encoder rerank?
Most production pipelines rerank somewhere between 20 and 100 candidates, whatever a bi-encoder or keyword search already retrieved as the shortlist. Pretrained rerankers like the MS MARCO cross-encoder models are trained and evaluated on exactly this shortlist-reranking setup, not on scoring entire corpora.
Do RAG pipelines need both a bi-encoder and a cross-encoder?
Most do, though not always. A bi-encoder alone handles retrieval cheaply but can rank a genuinely more relevant passage below a superficially similar one. Adding a cross-encoder as a second-stage reranker over the bi-encoder’s shortlist typically improves precision at the top of the results with only a small added latency cost.