Function calling was supposed to be the moment LLMs became useful for real work. Instead, it turned into a months-long debugging exercise for most teams who tried it at scale. After spending time in r/LocalLLaMA threads, HN discussions, and a handful of practitioner Discord servers, here is what the actual experience looks like once you move past the demo videos.

The pitch was simple. Define a JSON schema, hand it to the model, and the LLM returns structured arguments you can route to your code. No more regex parsing of freeform text. No more brittle prompt engineering to extract entities. The OpenAI function calling launch in mid-2023 set the template, and every major provider followed with their own variant. Anthropic added tool use, Google shipped function calling in Gemini, and the open source community built wrappers around smaller models to mimic the behavior.

What the marketing did not show was the failure modes.

The expectation gap

Most engineers came in expecting function calling to behave like a typed API. You declare a function signature, the model fills in the parameters, and your code runs. The reality, as multiple HN commenters have noted, is closer to an intern who reads the docs but sometimes invents parameter names that do not exist.

A thread on r/LocalLLaMA from late 2024 captured the sentiment well. The original poster had built a customer support agent with six tools and watched it hallucinate tool names that were not in the schema. The fix involved adding more explicit descriptions, constraining the temperature, and adding a fallback parser. The thread accumulated 200+ comments, mostly from developers sharing similar war stories.

The gap between expectation and reality usually shows up in three places. Parameter hallucination, schema misunderstanding, and tool selection errors. Each one has a different mitigation, and none of them are obvious from the documentation.

Where it actually delivers

When function calling works, it works well. The strongest use cases share a few characteristics.

Retrieval augmented generation over structured APIs is the obvious winner. If you have a well-documented REST API and the model can pick the right endpoint, function calling turns natural language into API calls with reasonable accuracy. Practitioners on the Latent Space podcast reported 85-95% accuracy on single-tool selection for clearly scoped tasks like look up order status or check inventory.

Latency has improved significantly. Early implementations added 1.5-3 seconds to a request. Current production deployments on GPT-4o and Claude 3.5 Sonnet typically see 400-900ms for the function call decision, with the full round trip including tool execution running 1-2 seconds for simple cases. Streaming the function call arguments separately from the natural language response helps mask this.

Cost is manageable for most workloads. A typical function-calling turn on GPT-4o costs around $0.005-0.015 depending on schema complexity and context length. For an agent doing 5-10 tool calls per user request, you are looking at $0.03-0.10 per conversation. Teams running thousands of conversations daily report this as a meaningful but not crushing line item.

The tasks where function calling genuinely shines include database queries through text-to-SQL wrappers, calendar and scheduling operations, CRM updates from natural language, document classification with structured outputs, and form filling from conversational input. These all share a pattern. The tool surface is small, the schemas are well-defined, and the cost of an occasional error is recoverable.

Where it falls apart

The failure modes cluster into predictable categories. Once you know what to look for, you can spot them in production logs within a week.

Parameter hallucination is the most common. The model invents a parameter that is not in the schema, passes a string where an integer is expected, or fills in a plausible-looking but wrong value. A practitioner blog post by Simon Willison documented this in detail, noting that even GPT-4 would occasionally return dates in the wrong format or invent user IDs that did not exist.

Tool selection errors compound with scale. If you have 3-5 tools, accuracy stays high. Push it to 15-20 tools and the model starts confusing similar functions. The HN thread on function calling at scale had multiple reports of teams needing to split their tool surface into multiple agents or use a router pattern to narrow the choice.

Multi-turn conversations are where things get ugly. Each function call introduces a chance for the model to drift from the original intent. A team I spoke with had a 92% success rate on single-turn tool use but dropped to 67% on the third turn of a multi-step workflow. The fix involved aggressive context pruning and explicit state tracking outside the model.

JSON schema violations are rarer but more frustrating. The model returns syntactically invalid JSON, escapes strings incorrectly, or wraps the response in markdown fences despite instructions not to. The instructor library and similar wrappers handle this with retry logic, but it adds latency and cost.

Cost surprises show up in two places. First, long schemas eat tokens. A complex tool definition with nested objects can consume 500-1000 tokens of context, and that gets billed on every request. Second, retry loops from validation failures can multiply your costs 3-5x if you do not set hard limits.

Onboarding friction is real but underdiscussed. New engineers on a team using function calling need to understand prompt engineering, JSON schema design, error handling patterns, and the specific quirks of whichever provider you are using. There is no standard abstraction yet, and switching providers usually means rewriting parts of your tool definitions.

Who it fits best

Function calling makes sense for teams with specific characteristics.

Team size matters more than people admit. Solo developers and small teams of 2-5 engineers can ship function-calling features in days. Larger teams of 20+ often struggle with coordination, schema versioning, and the lack of standardized tooling. The sweet spot seems to be 3-15 engineers with at least one person who has shipped LLM features before.

Use case fit is critical. If your task can be described as convert natural language to a structured API call, function calling is probably the right tool. If your task involves open-ended reasoning, creative generation, or ambiguous user intent, you will spend more time fighting the tool than using it.

Stack context matters. Teams already using Python with Pydantic have a much easier time than teams on Node or Go, because the tooling ecosystem is more mature. The instructor library, outlines, and similar projects give Python developers a significant head start.

If you are building a customer-facing product where reliability matters more than flexibility, function calling might not be the right choice yet. The error rates are still too high for fully autonomous operation in most domains. A human-in-the-loop pattern or a constrained tool surface usually works better.

What teams pair it with

The ecosystem around function calling has matured significantly. Most production deployments use a combination of tools.

Pydantic for schema definition is nearly universal in the Python world. You define your tool inputs as Pydantic models, and the validation is built in. The instructor library wraps this pattern and adds retry logic, response models, and streaming support.

LangChain and LlamaIndex provide higher-level abstractions, though they get mixed reviews. Practitioners on r/MachineLearning often complain about LangChain complexity and abstraction overhead, but acknowledge it is useful for prototyping. For production, many teams end up writing thinner wrappers over the raw API.

For local models, Ollama and vLLM both support function calling, but with caveats. Smaller models in the 7B-13B parameter range have noticeably worse tool selection accuracy. The community has settled on 70B+ models for serious function calling workloads, which means you need serious hardware.

Observability is non-optional. LangSmith, Helicone, and custom logging solutions are table stakes for any production deployment. You need to see which tools the model is selecting, what arguments it is passing, and where it is failing. Without this, debugging is guesswork.

What teams replace it with

Not every team sticks with function calling. The alternatives fall into a few categories.

For simple structured outputs, JSON mode or constrained decoding is often enough. If you just need the model to return a specific JSON shape and you are not actually calling external tools, you do not need function calling. Libraries like outlines and guidance give you guarantees about the output structure without the overhead.

For complex agentic workflows, some teams have moved to code generation. Instead of defining tools as JSON schemas, you let the model write Python or another language that gets executed in a sandbox. This is more flexible but introduces security concerns and execution latency.

For high-reliability use cases, fine-tuned models sometimes work better. If your tool surface is stable and well-defined, a fine-tuned 7B model can match GPT-4 accuracy for your specific tools while running faster and cheaper. The tradeoff is the upfront training cost and ongoing maintenance.

A few teams have gone back to prompt engineering for simple cases. If you only have one or two tools and the schema is straightforward, a well-crafted prompt with examples can match function calling accuracy without the overhead. It is not glamorous, but it works.

The honest take

Function calling is a real tool that solves real problems. It is also more fragile, more expensive, and more complex than the documentation suggests. The teams getting the most out of it are the ones who treat it as one component in a larger system, not a magic solution.

If you are evaluating it for a new project, start small. Pick one tool, one use case, and measure the failure modes. Build the validation and retry logic before you build the features. And budget for observability from day one, because you will need it.

The technology will keep improving. Models are getting better at tool selection, schemas are getting more expressive, and the tooling ecosystem is maturing. But the gap between works in a demo and works in production is still measured in months of engineering time, and that is not changing as fast as the launch announcements suggest.

Want the practical version of this? The free Working With Claude field guide covers the full Claude ecosystem, Claude Code, and how to roll it out across a real business. Download it here.