AI Gateway

An ecosystem with 14+ products calling multiple AI providers needs centralized management — not scattered API calls.

• Cost control across 9+ providers with different pricing models, token counting, and billing units
• Intelligent routing — choosing the cheapest, fastest, or highest-quality model depending on the request context
• API key resolution across organizations, users, and system tiers — with fallback to free models when all else fails
• Real-time monitoring: alerting when spend spikes, error pattern detection, and model health tracking

A gateway layer that intercepts every AI call in the ecosystem:

• Smart Router with 4 strategies — cheapest, fastest, quality, or balanced (scoring: quality×2 + speed - log₁₀(cost)×2). Each request routes to the optimal model based on the chosen strategy
• 6-level API key resolution — user → org → system-tier → system-all → env → free Workers AI. The first valid key wins; if all fail, the call falls back to Cloudflare Workers AI at zero cost
• Provider-specific adapters — each provider has its own adapter normalizing requests/responses to a common format. Custom providers (LMStudio, Ollama, vLLM) use an openai_compatible adapter
• Neuron budget management — Workers AI's free tier gives 10K neurons/day. The gateway tracks consumption and selects models that fit the remaining budget

Full request lifecycle managed end-to-end:

• Pipeline — Request → Tier Verification (DB) → Rate Limiting (tier-based, fail-open) → Smart Routing → API Key Resolution (6 levels) → Provider Adapter → callAI() wrapper → Response Normalization (all providers → OpenAI format)
• 9 modalities — LLM chat, Vision/OCR, Image Generation (TTI), TTS, STT, Video Generation, Embeddings, Gemini Live (bidirectional voice), ElevenLabs Conversational AI
• Error handling — 9 error types detected (rate_limit, auth, timeout, model_not_found, content_filter, quota_exceeded...), model health auto-updated, category-specific fallback chains
• Alerting — $1/hr warning, $5/hr critical thresholds. 7 SQL views for cost analytics, per-model usage, and error patterns
• Rate limiting tiers — FREE: 10/day, PERSONAL: 100/day, PROFESSIONAL: 1000/day, BUSINESS: unlimited pay-as-you-go