Modal is one of those tools that gets explained badly in most articles because people try to pitch it as a competitor to AWS Lambda, Kubernetes, or Heroku. It is not really any of those things. Modal is a Python-first serverless runtime built specifically for compute-heavy jobs. Think model inference, batch embeddings, fine-tuning runs, scheduled retraining jobs, and ad-hoc GPU experiments that you do not want to provision a VM for.

Under the hood Modal runs your function inside a container that it builds from an image you define in code. That container can request CPUs, RAM, and a wide range of GPUs at call time. You write a Python file, decorate a function, and run modal run. The platform figures out the container, the image build, the scheduling, and the scaling. You pay for the seconds the function actually runs.

The mental model is closer to “AWS Lambda but with GPUs, real filesystems, and no 15-minute timeout” than to a traditional MLOps platform. There is no cluster to manage, no Kubernetes YAML, no Docker registry to babysit. You do however get access to things Lambda cannot do well, like long-running jobs, large container images, persistent volumes, and webhooks with proper streaming.

This guide walks through the parts that matter when you go from zero to a working Modal pipeline. The goal is that by the end you can confidently run a GPU job, tune the settings that affect cost and latency, and know where Modal is the right call versus where you should reach for something else.

What Modal Actually Is

Strip away the marketing and Modal is three things glued together. A container build system that takes a Python file and produces a layered image. A job scheduler that places function calls onto warm or cold containers with the resources you asked for. A small client SDK and CLI that talks to the control plane to submit those calls.

When you decorate a function with @app.function you are doing two things. You are telling Modal what the function needs at runtime, like a GPU type, a memory ceiling, or a set of secrets. You are also telling Modal what the function depends on, because the decorator can reference an image object that includes pip packages, system packages, local files, and even the output of other functions.

The interesting part is the image system. Modal images are defined as Python objects that compose. You start from a base image, often modal.Image.debian_slim(), and chain pip_install, apt_install, run_commands, and add_local_dir. The image is built once, content-addressed, and cached. If you change only one line of one function, Modal rebuilds only the affected layer. For ML workloads that means a multi-gigabyte PyTorch image gets reused across runs and you only pay the cold build cost once.

GPU support is the headline feature. Modal exposes the standard cloud GPU tiers you’d expect at this price band, from T4s up through H100s. You select the GPU inside the decorator. The scheduler places your function on a host that has the requested accelerator. If you ask for an H100 and none are free, your call queues. There is no automatic fallback to a weaker GPU, which is a design choice that prevents silent downgrades but can bite you if you set a hard requirement on a scarce tier.

Setup and Authentication

Modal installs as a standard Python package and authenticates with a token tied to your account. The current recommended flow is to install the CLI and run a single login command.

Start by creating a virtual environment if you care about isolation. A typical pattern is python -m venv .venv then source .venv/bin/activate. Install the client with pip install modal. That gives you the modal CLI and the Python SDK in one package.

Authentication is the part that trips people up the first time. Run modal token new in your terminal. This opens a browser window, asks you to sign in or create an account, and writes a token to ~/.modal.toml. The token is what the CLI and SDK use to authenticate API calls. Treat this file like an SSH key. Anyone with the token can run jobs under your account and your account is billed for the GPU seconds.

For CI environments you cannot open a browser. In that case you generate a token from the dashboard, then export it as MODAL_TOKEN_ID and MODAL_TOKEN_SECRET environment variables. The CLI picks them up automatically and skips the browser flow. The same two variables work in any container that needs to deploy or run Modal functions, including GitHub Actions runners, Airflow workers, and Lambda functions calling other Modal functions.

You can verify everything is wired up with modal profile list. It should show your account, the workspace name, and the token fingerprint. If you see a “no profile configured” error, the token file is missing or the env vars are not set.

First Working Example

The fastest way to see Modal work is to run a tiny CPU job first, then graduate to a GPU. The example below runs a function that generates a synthetic dataset, fits a quick model, and returns a metric. It is intentionally small so the focus is on the platform mechanics rather than the math.

Define a file called hello_modal.py. The first line imports the SDK. Then you create an App object which is the deployment unit. The @app.function decorator marks a regular Python function as something Modal can run remotely. When you execute modal run hello_modal.py the CLI reads this file, uploads the source, builds the image, and invokes the function on a fresh container.

import modal

app = modal.App("hello-modal")

@app.function()
def square(x: int) -> int:
    return x * x

@app.local_entrypoint()
def main():
    print("3 squared is", square.remote(3))

The @app.local_entrypoint decorator marks a function that runs on your laptop and orchestrates calls to remote functions. It is the equivalent of a script main for Modal. Inside it you call .remote() on a function to submit it for execution. The result comes back as a Python value because Modal serializes the return.

To run it, hit modal run hello_modal.py. First run will take a while as Modal builds the image, which is the standard Debian slim plus the Modal SDK. Subsequent runs are quick because the image is cached.

Now the GPU version. Replace the contents of the file with a function that loads a small model and runs inference. The pattern is to import the heavy libraries inside the function body, not at the top of the file. That way the import only happens on the remote container, not on your laptop, and you do not need PyTorch installed locally to deploy a PyTorch function.

import modal

app = modal.App("gpu-inference")

image = (
    modal.Image.debian_slim()
    .pip_install("torch", "transformers")
)

@app.function(gpu="T4", image=image)
def classify(text: str) -> str:
    from transformers import pipeline
    clf = pipeline("sentiment-analysis")
    return clf(text)[0]["label"]

@app.local_entrypoint()
def main():
    label = classify.remote("Modal is genuinely useful")
    print("Sentiment:", label)

Run with modal run hello_modal.py. Modal will build the image, which downloads PyTorch and transformers. That first build can take several minutes. After that the image is cached and subsequent deploys are fast. The function will run on a T4. The first call will be slow because the container has to start and the model has to download. The second call inside the same modal run will be fast because the container is warm.

To expose the function as an HTTP endpoint, swap the @app.function for @app.webhook and add a method. Modal handles the routing and gives you a URL. You can curl it from anywhere.

Key Settings That Matter

The defaults Modal ships with are sensible, but a few knobs are worth understanding because they drive cost and latency more than anything else in the system.

The first is gpu. You can pass a string like "T4", "A10G", "A100-40GB", or "H100", or a count like gpu="A10G:2" to request multiple accelerators on one host. Cost scales roughly linearly with the GPU tier and the count, and a single H100 second costs more than ten T4 seconds in the pricing you’d expect at this tier. Pick the smallest GPU that fits your model and batch size.

The second is timeout. Default is something in the range of a few minutes, but you can raise it to several hours for long jobs like training or bulk inference. If your function times out the call is killed and you are billed for the time it ran. Set it just above your expected duration to avoid paying for runaway processes.

The third is container_idle_timeout. This controls how long a warm container sticks around between calls. A higher value means fewer cold starts, which is great for latency-sensitive endpoints. A lower value frees GPU capacity faster and reduces cost when traffic is bursty. For production endpoints a value in the range of several minutes is typical. For batch jobs you can set it to zero to always start cold.

The fourth is memory and cpu. Most ML jobs are memory bound, not CPU bound, so bumping memory often matters more than bumping cores. If you see OOM errors the fix is usually raising memory, not switching GPUs.

The fifth is secrets. You can attach named secrets from the Modal dashboard, and they get injected as environment variables inside the function. Use this for Hugging Face tokens, OpenAI keys, or any private model registry credentials. Never hardcode secrets in source.

The sixth is volumes. Modal supports persistent block storage that you mount into a container. The common pattern is to mount a volume at a path, write model weights or data shards to it from one function, and read them from another. This avoids re-downloading multi-gigabyte model files on every cold start.

The seventh is schedule. You can attach a cron-style schedule to a function and Modal will invoke it on the cadence you specify. This is the right way to run nightly retraining or hourly batch jobs. The scheduled function runs on whatever GPU and image you configured, so heavy retraining jobs do not need a separate runner.

Where It Shines

Modal is genuinely strong at three things. First, ad-hoc GPU work for individuals and small teams. If you are a developer who needs an A100 for twenty minutes a day to run experiments, Modal is dramatically cheaper and faster than renting a reserved instance. The same applies to founders who want to demo an AI feature without committing to a cloud account and a billing relationship with AWS.

Second, batch inference at moderate scale. Functions that fan out across hundreds or thousands of inputs, like embedding a corpus or running a vision model over a directory of images, fit Modal’s model perfectly. You can call .map() over a list and Modal will parallelize the calls across many containers automatically. The throughput scales with concurrency, and you only pay for the compute you actually use.

Third, internal tools and webhooks backed by models. Spinning up a @app.webhook that wraps a model and gives you a public URL is a one-liner. The endpoint supports streaming, custom headers, and integrates with Modal’s auth tokens if you need to lock it down. This is the fastest path from “I have a model in a notebook” to “I have an API my team can call.”

In all three of these cases the value proposition is the same. You skip the infrastructure layer. No Dockerfile, no ECR push, no IAM role, no auto-scaling group. The Python file is the deployment artifact.

Where It Fails

Modal is not a great fit when you need a long-lived always-on service. The platform is optimized for functions that get called, run, and exit. While you can run a webhook indefinitely, the underlying container can be recycled and there is no guaranteed uptime SLA in the way a managed service gives you. If you need 24/7 availability for a customer-facing product you are probably better off using Modal for batch and precompute, then serving results from a regular web host.

It is also a poor choice for stateful, multi-step pipelines that need to talk to each other with low latency. Functions communicate by returning values, and while that works fine for most ML patterns, it is not a replacement for a workflow engine if you have complex DAGs with branching and retries.

Observability is decent but not enterprise. You get logs, function-level metrics, and a simple dashboard. You do not get distributed tracing across calls, fine-grained cost attribution by team, or the kind of audit logs a regulated environment expects. For an internal tool this is fine. For a production system with compliance review you will likely export logs to your own stack.

Finally, vendor lock-in is real. The image definition, the decorator API, and the volume system are all Modal-specific. Migrating off means rewriting the deployment layer. The good news is that the actual business logic inside the function is plain Python and ports without changes. The bad news is that the thin orchestration layer around it is bespoke.

Practical Workflow Pattern

The pattern I have seen work well for small teams is to keep two Modal apps per project. One app handles ingestion and batch jobs, scheduled or triggered from external events. The other app exposes the inference endpoints used by the product. Both apps share a Modal volume where intermediate artifacts live.

Local development follows a tight loop. Write the function in a regular Python file. Run modal run to test it on real infrastructure. Modal gives you a live log stream while the function runs, which is closer to running it locally than deploying to a staging environment. Once it works, you add a webhook decorator if it needs to be callable over HTTP, or a schedule if it needs to run on a timer. You commit the file to git and that is the deployment.

For production you will probably want CI that runs modal deploy on merge to main. The deploy command promotes the app to a stable version. You can pin your product code to a specific version of the deployed app rather than always hitting the latest, which gives you a rollback path.

The last habit worth building is cost monitoring. Modal gives you usage breakdowns by function in the dashboard. Check it weekly at first. Functions that run longer than expected or that get called more than expected will show up there. Setting alerts on monthly spend is the simplest way to avoid the surprise bill that comes from a runaway scheduled job.

Modal will not replace your cloud platform. It is a specialized tool that removes a specific category of pain, the “I just need a GPU to run this for a few minutes” pain. Used in that lane it is one of the most productive pieces of infrastructure a developer can adopt in 2026.

If you want the playbook other teams are using with Claude and Codex right now, grab the free Working With Claude field guide. Download it here.