What Semantic Search Actually Is

Strip away the marketing and semantic search is a retrieval method that matches queries to documents based on meaning rather than exact keyword overlap. Under the hood it does this by converting text into high-dimensional vectors called embeddings, then comparing those vectors using a distance metric like cosine similarity.

Traditional keyword search treats “refund policy” and “money back guarantee” as unrelated strings. Semantic search recognises they sit close together in vector space because a model trained on large amounts of text has learned that humans use them interchangeably. The same applies to “Postgres performance slow” and “my database queries are taking forever”. A keyword engine sees no overlap. A semantic engine retrieves the right document.

The moving parts are straightforward. You need an embedding model to turn text into vectors, a vector store to hold those vectors and run similarity lookups, and a chunking strategy to break long documents into pieces small enough to embed meaningfully. Once those three pieces are wired together you have a working semantic search system.

The reason this matters now is that embedding models have become cheap and good. A few years ago you needed a custom-trained model and serious infrastructure. Today you can call an embedding API, get back a 1536-dimensional vector for a few cents per million tokens, and store millions of those vectors in an open-source database on a laptop. The barrier to a production-quality semantic search system has dropped from months of engineering work to an afternoon.

Setup and Authentication

For this walkthrough we will use three components: an embedding model, a vector store, and a small set of Python libraries to glue them together.

For the embedding model we will use the OpenAI text-embedding-3-small API. It is inexpensive, well-documented, and produces vectors that work well across most retrieval tasks. You will need an API key from platform.openai.com. Set it as an environment variable so you never hardcode it into a script.

For the vector store we will use Chroma, an open-source embedding database that runs in memory or persists to disk. It is a sensible default because it requires no external server, has a clean Python API, and handles the persistence and indexing for you.

Install the dependencies. From a fresh Python environment run pip install openai chromadb tiktoken. The tiktoken package is optional but useful if you want to count tokens before sending text to the embedding API, which helps you stay within model limits and control cost.

Set your API key in the shell before running any script. On macOS or Linux that looks like export OPENAI_API_KEY=sk-your-key-here. On Windows use set OPENAI_API_KEY=sk-your-key-here or the equivalent PowerShell syntax. If you prefer a .env file, the python-dotenv package will load it automatically when you call load_dotenv at the top of your script.

That is the entire setup. No Docker, no managed cluster, no service accounts. You should be able to run a working semantic search against a folder of documents inside ten minutes of finishing this section.

First Working Example

Let us build a minimal but real semantic search over a small collection of text documents. The script below indexes five short documents, embeds them, stores them in Chroma, then runs a query and prints the most relevant results.

Start by creating a working directory and a file called search.py. Inside that file, import the libraries and set up the OpenAI client. The client picks up your API key from the environment variable automatically, so you do not need to pass it explicitly.

Next, create a Chroma client. Calling chromadb.Client() gives you an in-memory store that disappears when the script exits. For persistence across runs use chromadb.PersistentClient with a path argument pointing to a local folder. For this example the in-memory version is fine.

Create a collection. A collection is roughly equivalent to a table in a relational database, except every row carries a vector alongside its metadata. Give it a name like “knowledge_base”.

Now define your documents. In a real system these would come from files on disk, a database, or an API. For the example we will hardcode five short strings covering topics like shipping, returns, account management, pricing, and technical support.

To embed and store them, iterate over the list and call collection.add on each one, passing the document text as the documents argument and a unique id. Chroma will call the embedding function automatically if you pass one in when creating the collection. The simplest path is to use the built-in OpenAI embedding function by importing OpenAIEmbeddingFunction from chromadb.utils.embedding_functions and passing it your API key and model name.

Once the documents are indexed, run a query. Call collection.query with a query_texts argument containing your search string and an n_results argument set to however many matches you want returned. Chroma returns a dictionary with keys for ids, documents, and distances. The distances are cosine distances, so smaller values mean closer matches.

A working query like “how do I get my money back” should return the returns policy document at the top of the results, even though that exact phrase does not appear in the document text. That is the semantic part doing its job.

The full script is roughly thirty lines including imports. Run it with python search.py and you should see the matching documents printed in order of relevance.

Key Settings That Matter

Most tutorials stop at the working example and skip the configuration that separates a toy system from a useful one. These are the dials worth understanding.

Chunk size and overlap are the first thing to tune. Embedding models have a context window, typically a few thousand tokens, and longer inputs get truncated or produce worse vectors. For documents longer than the limit you must split them. A common starting point is chunks of 500 tokens with 50 tokens of overlap between consecutive chunks. The overlap prevents information loss at chunk boundaries. Smaller chunks give more precise retrieval but lose surrounding context. Larger chunks preserve context but dilute the signal.

The embedding model itself is the second dial. text-embedding-3-small is fast and cheap. text-embedding-3-large is more accurate but costs more and produces larger vectors. For most internal knowledge bases the small model is sufficient. If you are building a customer-facing system where retrieval quality directly affects revenue, test the large model and compare.

Distance metric is the third dial. Chroma defaults to squared L2 distance. For most embedding models cosine similarity is the right choice because it ignores vector magnitude and focuses on direction. You can switch to cosine by passing the right configuration when creating the collection.

Metadata filtering is the fourth dial and the one most people ignore. Every document you store can carry metadata fields like source, date, author, or category. You can then query with a where filter to restrict retrieval to a subset. For example, you might want to search only documents created in the last 30 days, or only documents tagged as “engineering”. Combining semantic similarity with metadata filters gives you much more control than pure vector search.

The number of results returned, often called top-k, is the fifth dial. Returning too few results hurts recall. Returning too many dilutes the signal and slows downstream processing. A typical starting point is 5 to 10 results for a chat-style interface and 20 to 50 for a document review workflow where a human will look at the output.

Finally, the embedding function you choose for Chroma must match the one you use elsewhere. If you embed documents with one model and queries with another, the vectors live in incompatible spaces and retrieval fails silently with poor results. Pick a model and stick with it.

Where It Shines

Semantic search genuinely excels at a few specific use cases that are painful with keyword search.

Internal knowledge bases are the clearest win. Companies accumulate years of Slack threads, Notion pages, Confluence docs, and meeting notes. People know the answer to their question exists somewhere but cannot remember the exact words. Semantic search finds it anyway. The retrieval quality improvement over keyword search is usually dramatic, often the difference between “this tool never finds anything” and “this tool actually works”.

Customer support is another strong fit. Support tickets are written in the language of the customer, while the knowledge base is written in the language of the company. A customer asking “my package never arrived” needs to match a help article titled “delivery exceptions and lost shipments”. Semantic search handles this translation automatically.

Code search is a less obvious but powerful application. Embedding function names, docstrings, and comments lets you find code by describing what it does rather than remembering exact identifiers. Tools like Cursor and Continue use this approach for parts of their retrieval pipeline.

Long-document question answering, where you embed an entire book or report and then ask questions about it, works well once you get the chunking right. The pattern is to retrieve the top-k most relevant chunks and pass them to a language model as context for answering the question.

Multilingual search is a quiet superpower. Many modern embedding models are trained on dozens of languages and produce vectors that live in a shared space. You can index English documents and query them in Spanish, or vice versa, with no extra configuration.

Where It Fails

Honest accounting matters here because semantic search has real limitations.

It is bad at exact matching. If a user searches for “order #48291”, semantic search will not necessarily return that specific order. You need a hybrid approach that combines vector retrieval with traditional keyword filtering for identifiers, codes, and proper nouns.

It is expensive at scale relative to keyword search. Embedding a million documents costs real money and takes time. Querying is cheap, but the indexing step is not. For very large corpora you need to think about incremental indexing, storage costs, and how often you re-embed.

It is opaque. When semantic search returns a wrong result, it is hard to debug why. The vector space has thousands of dimensions and the model is a black box. You can inspect distances and metadata, but you cannot easily explain to a stakeholder why one document ranked above another.

It degrades with poor data. If your source documents are noisy, outdated, or contradictory, semantic search will faithfully retrieve the noise. Garbage in, garbage out applies with extra force because users trust the results more than they trust keyword search.

It does not handle temporal queries well by default. Asking “what changed in the last week” requires metadata filtering on dates, not just semantic similarity. You need to combine the two.

It also struggles with very short queries that lack context. A single word like “pricing” returns a broad mix of results. You can mitigate this by requiring minimum query length, prompting the user for more context, or augmenting short queries with related terms before embedding.

Practical Workflow Pattern

Here is how to slot semantic search into a real work setup rather than leaving it as a demo script.

Start with a single use case and a small corpus. Pick a concrete problem like “find the right policy doc when a customer asks X” or “let the team search our meeting notes”. Do not try to build a general-purpose search engine for the whole company on day one.

Build the indexing pipeline as a scheduled job. A simple cron job or a GitHub Action that runs nightly is enough. It reads new or modified documents, chunks them, embeds them, and upserts them into the vector store. Treat the vector store like any other database with backups, monitoring, and a clear schema.

Build the query interface as a thin layer over the vector store. A FastAPI endpoint that accepts a query string, runs the search, and returns the top-k results is a good starting point. From there you can wrap it in a chat interface, a Slack bot, or a browser extension depending on where your users actually work.

Add evaluation early. Create a small set of test queries with known correct answers and measure retrieval quality over time. Tools like ragas or simple precision-at-k metrics will catch regressions when you change the embedding model, the chunking strategy, or the source data.

Plan for hybrid retrieval from the start. Combine semantic search with a keyword filter for identifiers, dates, and exact phrases. Most production systems that look “purely semantic” are actually hybrid under the hood.

Finally, set a budget and monitor usage. Embedding API costs are small per query but add up across a team. Track tokens embedded per day, average query latency, and storage growth. These three numbers tell you almost everything you need to know about whether the system is healthy.

That last point about monitoring is the one most teams skip and regret later. A semantic search system that worked great in a demo can quietly rot as the source data changes, the embedding model gets updated, or the chunking strategy stops matching new document formats. Treat it as a living system, not a one-time build.

The practical next step is the free Working With Claude field guide. Thirty-two pages covering the ecosystem, Claude Code, and how to govern a rollout properly. Get your copy.