Generative AI features fail for one reason more than any other: they make your app feel slow.
When a user clicks a button and waits 4 seconds for a response, they leave. That’s not a product problem — it’s an architecture problem. The AI response time doesn’t have to dictate the perceived experience.
Here’s how you build LLM features that feel fast.
The Core Problem: LLMs Are Slow
Language models don’t return responses instantly. GPT-4o returns the first token in roughly 400–600ms and streams the rest. For content generation tasks, a full response can take 8–15 seconds.
That’s too long to show a blank screen or a spinner. Users stop waiting after 2–3 seconds.
The solution is to separate perceived latency from actual latency. Your job is to make the wait feel short, even if it isn’t.
Technique 1: Stream from the Edge
Instead of waiting for a complete LLM response and then sending it all at once, stream tokens to the browser as they arrive. Next.js Route Handlers make this straightforward using the Vercel AI SDK’s streaming utilities.
Text appears word-by-word as the model generates it. The first visible output reaches the user in under 600ms. The experience feels like watching someone type — not waiting for a page to load.
Technique 2: Optimistic UI State
For actions that use AI in the background — generating a title, summarizing text, writing a reply — show an immediate response in the UI before the LLM finishes.
Show a pulsing cursor or a subtle skeleton loader while generating. The user sees immediate feedback. The complete response fills in naturally.
Technique 3: Run AI at the Edge
Cold start latency kills AI features in serverless environments. A Lambda function that hasn’t been called in 10 minutes takes 1–3 seconds just to boot up — before the LLM even starts processing.
Deploy your AI route handlers on Edge runtime. Edge functions start in under 5ms globally. Your LLM calls start immediately. Combined with streaming, users see the first token in under 700ms from anywhere in the world.
Technique 4: Cache What You Can
Not every LLM call needs to be fresh. Product descriptions, FAQ answers, content summaries — these don’t change every minute.
Cache LLM outputs in Redis or KV storage with a short TTL. Use a composite cache key that includes the content’s last-modified timestamp. This means the cache automatically invalidates when content changes. You only hit the LLM API when content is genuinely new.
Controlling API Costs
LLM APIs charge per token. On a high-traffic site, uncached calls add up fast. Three practical controls:
- Rate limit per user: Track calls with a sliding window counter in Redis. Block excessive requests early.
- Set max_tokens: Don’t let the model generate 4,000 tokens when you need 200.
- Use smaller models for simple tasks: GPT-4o-mini costs 15x less than GPT-4o. For summarization and classification, it works just as well.
The Mental Model
Building AI features into production web apps means solving two separate problems:
- The AI problem: Getting the right output from the model
- The UX problem: Making the wait feel short regardless of model speed
Solve both independently. Stream output from the edge. Use optimistic UI. Cache aggressively. Rate limit cleanly.
Your users don’t care how impressive the model is. They care if the feature feels fast.