What Reranking Actually Is
Reranking is a second-pass scoring step that runs after your initial vector search. The first pass retrieves a broad set of candidate chunks, typically 20 to 100, using approximate nearest neighbor matching on embeddings. The reranker then reads each candidate against the actual query and produces a refined relevance score. The top results from this second pass are what you feed to the language model.
This is not the same as hybrid search, query rewriting, or cross-encoder embeddings used at index time. Those are alternative retrieval strategies. Reranking is downstream of all of them and works as a filter and reorder step.
The reason rerankers work is simple. Bi-encoders, the standard embedding models, encode the query and the document separately. They never see them together. This makes them fast but lossy. A cross-encoder reranker takes the query and document as a single input and produces a token-level attention pattern that captures real semantic relationship. It is slower, sometimes by an order of magnitude, but it catches relevance that the initial retrieval missed.
Most teams running RAG systems hit a wall at around 60 to 75 percent retrieval accuracy on domain-specific content. The embeddings are doing their job, but the top-k results are noisy. A reranker usually pushes that number into the 80 to 90 percent range without changing the underlying index.
Setup and Authentication
For a practical walkthrough, Cohere’s rerank endpoint is the most accessible option for most teams. It exposes a single API call, has a free tier suitable for development, and returns clean scores you can sort on.
The setup steps:
Create a Cohere account at cohere.com and grab an API key from the dashboard. The free tier gives you a meaningful number of calls per month, enough for prototyping and small workloads.
Install the client. From a terminal with Python 3.9 or later, run pip install cohere.
Set your API key. Either export it as an environment variable (export COHERE_API_KEY=your_key_here) or pass it directly when you instantiate the client. The environment variable approach is the safer default for any code that touches production.
Verify connectivity with a small script. The library exposes a health endpoint you can hit, or you can just call the rerank function with a known input and confirm a response.
If you are on a managed vector database like Pinecone, Weaviate, or Qdrant, several of them ship with built-in reranker hooks. Pinecone’s inference API, for example, exposes rerank as a sidecar operation you can call after a search. The setup pattern is the same: API key, client instantiation, then a call that takes documents and a query.
For self-hosted setups, sentence-transformers has cross-encoder models you can run locally. The typical choice is a BGE reranker or a MiniLM cross-encoder. These need a GPU for reasonable latency but work fine on CPU for batch jobs.
First Working Example
Below is a minimal end-to-end pattern. It assumes you already have a vector store returning top 20 candidates. The reranker trims that to the top 5.
The Python version, using Cohere:
import cohere import os
co = cohere.Client(os.environ[“COHERE_API_KEY”])
documents = [ “Quarterly revenue grew by 14 percent year over year, driven by enterprise expansion.”, “The CFO noted margin compression in the consumer segment during the earnings call.”, “Customer acquisition cost rose sharply in Q3 as marketing spend was pulled forward.”, “The board approved a new buyback program of up to 500 million dollars.”, “Operating cash flow improved on stronger working capital management.” ]
query = “Why did margins change last quarter?”
results = co.rerank( model=“rerank-english-v3.0”, query=query, documents=documents, top_n=3 )
for r in results.results: print(f”Score: {r.relevance_score:.4f} | Index: {r.index} | {documents[r.index]}”)
Running this, the second and third documents will typically score highest, with the first close behind. The fourth and fifth, while topically related, will score lower because they do not directly address the margin question.
For a local cross-encoder using sentence-transformers:
from sentence_transformers import CrossEncoder
model = CrossEncoder(“BAAI/bge-reranker-v2-m3”) pairs = [[query, doc] for doc in documents] scores = model.predict(pairs)
ranked = sorted(zip(scores, documents), reverse=True) for score, doc in ranked[:3]: print(f”Score: {score:.4f} | {doc}”)
The scores are not on the same scale as Cohere’s, so do not compare them numerically across vendors. Use them only for ranking within a single call.
To wire this into a real RAG pipeline, the structure is:
- Embed the query and retrieve the top 20 to 50 chunks from your vector store.
- Pass those chunks plus the query to the reranker.
- Take the top 5, or whatever your final context window allows, and feed them to the language model as context.
The total added latency is usually 100 to 300 milliseconds for hosted rerankers and 50 to 200 milliseconds for local models on modern hardware. That is well within the budget for most chat-style applications.
Key Settings That Matter
The dial most people ignore is the candidate count you send to the reranker. Sending only 5 is a wasted call because there is nothing meaningful to reorder. Sending 200 is wasted compute. The sweet spot for most workloads is between 20 and 50 candidates. Below 20, you are often not giving the reranker enough room to find the truly relevant chunks. Above 50, returns diminish fast and latency climbs.
The next dial is the score threshold. Some rerankers, including Cohere’s, return normalized scores you can use to filter. A score below roughly 0.3 on Cohere’s scale usually indicates weak relevance. You can drop those documents entirely rather than passing low-quality context to the model. The exact threshold depends on your content and query distribution, so test on a held-out set.
Top_n is the third dial. This is the number of documents you keep after reranking. For most LLM context windows of 8k to 16k tokens, 3 to 8 documents is typical. Going higher dilutes the model’s attention and tends to hurt answer quality even if the raw recall looks better.
Model choice matters. The current version of Cohere’s rerank-english-v3.0 is a strong general-purpose option. For multilingual content, rerank-multilingual-v3.0 is the equivalent. The BGE reranker family from BAAI is the most common local choice, with v2-m3 being a reasonable default. Larger rerankers are more accurate but slower. For most workloads, the small or medium tier is the right starting point.
Max tokens per document is a setting many teams miss. If a chunk is very long, the reranker truncates it. If your chunks are 800 tokens, you are probably truncating useful context. Trim your chunks to 200 to 500 tokens before indexing, and most of this concern goes away.
Finally, batch size. If you are running a local reranker, the batch size controls throughput. A batch size of 16 to 32 on a single modern GPU is usually the throughput sweet spot. Going higher runs out of memory, going lower underutilizes the hardware.
Where It Shines
Reranking is most effective on queries where semantic similarity is not enough. These include cases where the user uses different vocabulary from the source documents, where multiple documents cover overlapping topics, and where the answer hinges on a subtle distinction.
Specific use cases that show measurable improvement:
Domain-specific retrieval where the embedding model was trained on general web text. Legal, medical, financial, and engineering content all benefit heavily. Retrieval accuracy can improve by 15 to 30 percent in these settings.
Multi-hop questions where the answer requires combining pieces from several documents. The reranker helps surface the bridging document that connects the two halves of the question.
Queries with negation or exclusion. Phrases like “excluding acquisitions” or “before the policy change” are notoriously hard for bi-encoders. Cross-encoders handle them much better.
Customer support and help-desk search, where users describe problems in their own words and the canonical documentation uses internal jargon.
Long-tail queries where there is no obvious keyword match. The reranker picks up on semantic cues the embedding model flattens.
Where It Fails
Reranking is not free, and there are situations where it does not help or actively hurts.
If your initial retrieval is already very poor, sending bad candidates to the reranker wastes time. The reranker can only reorder what it sees. If the right document is not in the top 20, no reranker will find it. Fix retrieval first.
For very short queries, one or two words, reranking adds little. The cross-encoder does not have enough signal to differentiate candidates, and you pay latency for no gain.
Multilingual or code-switched content can confuse rerankers not trained for it. If your content mixes English and another language heavily, test explicitly.
Cost scales linearly with the number of documents and the document length. At high QPS, hosted rerankers can become a meaningful fraction of your infrastructure bill. Local models have a fixed GPU cost, so they are cheaper at scale but require more engineering.
Rerankers do not understand freshness or recency. If your retrieval problem is “what changed in the last 24 hours”, reranking does not help. You need a different solution.
Finally, reranking does not fix a bad prompt or a bad language model. It only improves what reaches the model. If the model is hallucinating because the prompt is unclear, reranking will not save you.
Practical Workflow Pattern
The pattern that works in production is straightforward and does not require exotic infrastructure.
Step one, build your vector index with clean chunks between 200 and 500 tokens