Start from the workload, not the technology

The question of building a custom AI agent vs calling the ChatGPT API gets answered backwards by most teams. They pick the technology first, then discover the workload does not fit. The right starting point is: how does your AI feature actually behave under real traffic, real edge cases, and real cost pressure.

When the ChatGPT API is the right call

Use OpenAI, Anthropic, or Google's API directly when your feature volume is unpredictable, when you need general reasoning that benefits from a frontier model, when your team has zero ML infrastructure, or when the feature is a small slice of a broader product. APIs win on time-to-first-version. We have shipped useful AI features in two weeks by wrapping the OpenAI Chat Completions API with a thin prompt template and structured output validation. Hard to beat that for a first move.

When a custom AI agent earns the investment

A custom AI agent (your own orchestration layer over multiple models, with tools, memory, and structured workflows) makes sense when one of three conditions is true. First, when token cost at your usage volume crosses $5,000-$10,000 per month and trends up — at that point your own routing, caching, and smaller fine-tuned models save real money. Second, when latency requirements drop below 500 ms — round-trips to a hosted API rarely beat that consistently. Third, when the workflow requires multi-step reasoning with tool use, where you need observability into which step failed and why.

The agent architecture you actually need

A production AI agent is not just a chat loop. It is a planner that decomposes the user request, a tool layer that performs typed calls (search, database, calculation), a memory system that holds short-term state per conversation and longer-term state per user or organization, and an evaluation layer that catches regressions before they reach users. We build agents on top of frameworks like LangGraph or our own typed orchestration in TypeScript. The orchestration layer is the part that earns its keep over months of iteration.

Cost economics at scale

For 10,000 conversations per month with average 2,000 tokens, a GPT-4-class API call costs roughly $200-$400. That same workload, served by a fine-tuned 8B model on dedicated infrastructure, costs roughly $80-$150 including infrastructure. The break-even depends on your traffic and how much you can compress. We do not see the math favor self-hosting until you cross around $3,000-$5,000 monthly API spend, which lines up with 100,000+ requests per month for most workloads.

Observability is non-negotiable

Whichever route you choose, instrument from day one. Log the full prompt and response (with PII redaction). Track which model version answered which request. Track tool-call success and failure. Without this you cannot debug regressions, you cannot evaluate model swaps, and you cannot answer compliance questions. We use a combination of OpenTelemetry traces and a dedicated evaluation harness that runs after every prompt change.

Hybrid is usually the right answer

At enough scale, most teams end up running a hybrid: a frontier API for the hard 10% of queries and a self-hosted smaller model for the easy 90%. The routing layer is straightforward to build once you have evaluations in place. The cost savings show up quickly and the architecture stays maintainable.

Our default recommendation

For a first AI feature with unclear demand, start with the OpenAI or Anthropic API and a thin custom orchestration layer. Build the evaluation harness in parallel. Once you cross $2,000 in monthly API spend or hit latency limits, add a self-hosted route for the simple cases and keep the API call for the hard cases. Skip the urge to build everything custom on day one. The infrastructure cost of running your own model serving stack is real and easy to underestimate.