What LlamaIndex Actually Is
LlamaIndex is a data framework for LLM applications. It is not an LLM, not a vector database, and not a chatbot builder. Its job is to sit between your private data and a language model so the model can answer questions grounded in that data.
The core primitives you will work with are Documents, Nodes, Indexes, Retrievers, QueryEngines, and ResponseSynthesizers. A Document is raw text plus metadata loaded from a file, URL, or API. A Node is a chunk of a Document, the unit that actually gets embedded and retrieved. An Index is a data structure over those Nodes, the most common being VectorStoreIndex. A Retriever decides which Nodes to return for a given query. A QueryEngine combines a Retriever with a ResponseSynthesizer, which formats the retrieved context into a final answer.
In the current version, LlamaIndex also ships an event-driven Workflows layer that replaces older callback patterns. Workflows let you express multi-step RAG pipelines as Python functions that emit and receive events, which is closer to how production teams actually want to model async work.
Compared to LangChain, LlamaIndex is more opinionated about the data layer and less about chains of LLM calls. If your problem is connect an LLM to a pile of documents and let it answer questions, LlamaIndex tends to be the shorter path. If your problem is wire together a dozen LLM calls with branching logic, the comparison is closer.
Setup and Authentication
Install the base package with pip install llama-index. From there you add extras based on which integrations you need. The common ones are llama-index-llms-openai, llama-index-llms-anthropic, llama-index-embeddings-openai, llama-index-vector-stores-chroma, llama-index-vector-stores-pinecone, and llama-index-parse-pdf if you want the hosted PDF parser.
Authentication is handled through standard environment variables. For OpenAI you set OPENAI_API_KEY in your shell or a .env file. For Anthropic you set ANTHROPIC_API_KEY. For hosted services like LlamaParse you set LLAMA_CLOUD_API_KEY. LlamaIndex reads these through its Settings object, which in the current version has replaced the older ServiceContext.
A clean starting point is to create a .env file with your keys, install python-dotenv, and call load_dotenv() at the top of your script. Then configure your default LLM and embedding model once with Settings.llm and Settings.embed_model so every downstream component picks them up without you passing them around explicitly.
First Working Example
Here is a minimal, runnable RAG pipeline. Save it as rag_demo.py and run it with python rag_demo.py.
First, install the dependencies:
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenvThen create the script:
from dotenv import load_dotenv
from llama_index.core import (
SimpleDirectoryReader,
VectorStoreIndex,
StorageContext,
load_index_from_storage,
Settings,
)
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
load_dotenv()
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What does this corpus cover?")
print(response)The load step uses SimpleDirectoryReader which walks a folder and ingests any supported file types including .txt, .md, .pdf, .csv, and .docx. Pass it a path and it returns a list of Document objects.
The index step calls VectorStoreIndex.from_documents(documents). This chunks each Document into Nodes using the default NodeParser, embeds each Node using your configured embedding model, and stores everything in an in-memory vector store by default.
The query step calls index.as_query_engine() to get a QueryEngine, then calls .query() on it. The retriever fetches the top-k most similar Nodes, the response synthesizer feeds them to the LLM as context, and the LLM produces a final answer.
To persist the index between runs--- title: “LlamaIndex RAG Tutorial Step by Step for Production Use” description: “A hands-on LlamaIndex RAG tutorial step by step covering setup, indexing, retrieval settings, and a runnable first example you can extend.” publishDate: “2026-06-27” author: “Sam McKay” difficulty: “intermediate” service: “general” tags:
- ai-tools
- tutorial draft: false
What LlamaIndex Actually Is
LlamaIndex is a data framework for LLM applications. It is not an LLM, not a vector database, and not a chatbot builder. Its job is to sit between your private data and a language model so the model can answer questions grounded in that data.
The core primitives you will work with are Documents, Nodes, Indexes, Retrievers, QueryEngines, and ResponseSynthesizers. A Document is raw text plus metadata loaded from a file, URL, or API. A Node is a chunk of a Document, the unit that actually gets embedded and retrieved. An Index is a data structure over those Nodes, the most common being VectorStoreIndex. A Retriever decides which Nodes to return for a given query. A QueryEngine combines a Retriever with a ResponseSynthesizer, which formats the retrieved context into a final answer.
In the current version, LlamaIndex also ships an event-driven Workflows layer that replaces older callback patterns. Workflows let you express multi-step RAG pipelines as Python functions that emit and receive events, which is closer to how production teams actually want to model async work.
Compared to LangChain, LlamaIndex is more opinionated about the data layer and less about chains of LLM calls. If your problem is connect an LLM to a pile of documents and let it answer questions, LlamaIndex tends to be the shorter path. If your problem is wire together a dozen LLM calls with branching logic, the comparison is closer.
Setup and Authentication
Install the base package with pip install llama-index. From there you add extras based on which integrations you need. The common ones are llama-index-llms-openai, llama-index-llms-anthropic, llama-index-embeddings-openai, llama-index-vector-stores-chroma, llama-index-vector-stores-pinecone, and llama-index-parse-pdf if you want the hosted PDF parser.
Authentication is handled through standard environment variables. For OpenAI you set OPENAI_API_KEY in your shell or a .env file. For Anthropic you set ANTHROPIC_API_KEY. For hosted services like LlamaParse you set LLAMA_CLOUD_API_KEY. LlamaIndex reads these through its Settings object, which in the current version has replaced the older ServiceContext.
A clean starting point is to create a .env file with your keys, install python-dotenv, and call load_dotenv() at the top of your script. Then configure your default LLM and embedding model once with Settings.llm and Settings.embed_model so every downstream component picks them up without you passing them around explicitly.
First Working Example
Here is a minimal, runnable RAG pipeline. Save it as rag_demo.py and run it with python rag_demo.py.
First, install the dependencies:
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenvThen create the script:
from dotenv import load_dotenv
from llama_index.core import (
SimpleDirectoryReader,
VectorStoreIndex,
StorageContext,
load_index_from_storage,
Settings,
)
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
load_dotenv()
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What does this corpus cover?")
print(response)The load step uses SimpleDirectoryReader which walks a folder and ingests any supported file types including .txt, .md, .pdf, .csv, and .docx. Pass it a path and it returns a list of Document objects.
The index step calls VectorStoreIndex.from_documents(documents). This chunks each Document into Nodes using the default NodeParser, embeds each Node using your configured embedding model, and stores everything in an in-memory vector store by default.
The query step calls index.as_query_engine() to get a QueryEngine, then calls .query() on it. The retriever fetches the top-k most similar Nodes, the response synthesizer feeds them to the LLM as context, and the LLM produces a final answer.
To persist the index between runs