Six months ago, we added AI image generation to our e-commerce platform. We expected moderate usage—maybe 50,000 images per month.

Within three weeks, we were generating 500,000 images monthly. After two months, we crossed 2 million. Today, we handle 3+ million generations monthly with 99.7% uptime and P95 latency under 800ms.

This isn't a story about heroic firefighting or midnight debugging sessions (though we had those). It's about architecture decisions that let a small team scale AI generation to production workloads without drowning in operational complexity.

Let me show you the patterns that worked.

The Challenge: AI APIs Don't Scale Like Traditional APIs

Before diving into solutions, understand why AI generation creates unique scaling challenges:

Traditional REST API:

• Response time: 10-100ms

• Predictable costs: Fractions of a cent per request

• Resource usage: Minimal per request

• Error rate: <0.1% under normal conditions

AI Generation API:

• Response time: 2-30 seconds (200-3000x slower)

• Variable costs: $0.005-0.10 per request (1000x higher)

• Resource intensive: Significant compute per request

• Error rate: 2-5% (due to provider issues, timeouts, quality failures)

These differences cascade through your entire architecture. Patterns that work for traditional APIs break under AI workloads.

According to Google Cloud's AI infrastructure guide, teams building production AI services report spending 3-5x more time on infrastructure and operations compared to traditional API services.

Our Scale and Requirements

To ground this discussion in reality, here's what we're actually running:

Monthly volume: 3.2M image generations
Peak throughput: 180 requests/second
P50 latency: 280ms (cached), 4.2s (generated)
P95 latency: 320ms (cached), 8.5s (generated)
P99 latency: 450ms (cached), 12.8s (generated)
Cache hit rate: 68%
Error rate: 1.2%
Monthly cost: \(4,800 (generation) + \)380 (infrastructure)
Team size: 2 engineers (part-time maintenance)

Key insight: 68% cache hit rate means only 1M requests actually generate new images. Caching isn't optional at scale—it's what makes the economics work.

Architecture Overview

Here's the system we built:

┌─────────────┐
│   Clients   │
└──────┬──────┘
       │
       ▼
┌─────────────────┐
│  Load Balancer  │
│    (nginx)      │
└──────┬──────────┘
       │
       ▼
┌─────────────────────────────────┐
│   API Servers (3 instances)     │
│   - Request validation          │
│   - Auth & rate limiting        │
│   - Response formatting         │
└──────┬──────────────────────────┘
       │
       ├──────────────────┬───────────────────┐
       ▼                  ▼                   ▼
┌─────────────┐   ┌──────────────┐   ┌──────────────┐
│ L1 Cache    │   │  L2 Cache    │   │  Job Queue   │
│ (In-memory) │   │  (Redis)     │   │  (Bull)      │
└─────────────┘   └──────┬───────┘   └──────┬───────┘
                          │                  │
                          │                  ▼
                          │          ┌───────────────┐
                          │          │   Workers     │
                          │          │ (10 instances)│
                          │          └───────┬───────┘
                          │                  │
                          ▼                  ▼
                  ┌──────────────────────────────┐
                  │      AI Provider             │
                  │     (WaveSpeedAI)            │
                  └──────────────────────────────┘

Let's break down each layer.

Layer 1: Multi-Tier Caching

Caching at scale requires more than Redis. We use three tiers:

L1: In-Memory LRU Cache (Per Server)

const LRU = require('lru-cache');

class L1Cache {
  constructor() {
    this.cache = new LRU({
      max: 1000, // Store 1000 most recent
      maxAge: 1000 * 60 * 5, // 5 minute TTL
      updateAgeOnGet: true
    });
  }

  get(key) {
    const value = this.cache.get(key);
    if (value) {
      this.recordHit('l1');
      return value;
    }
    this.recordMiss('l1');
    return null;
  }

  set(key, value) {
    this.cache.set(key, value);
  }

  // Track statistics
  recordHit(tier) {
    metrics.cacheHits.inc({ tier });
  }

  recordMiss(tier) {
    metrics.cacheMisses.inc({ tier });
  }
}

Why it works: Hot requests (repeated in seconds) never leave the server. This handles traffic spikes without Redis load.

L1 stats in production:

• Hit rate: 12-15% of total requests

• Response time: <5ms

• Cost: $0 (in-memory)

L2: Redis Cache (Shared)

const Redis = require('ioredis');
const crypto = require('crypto');

class L2Cache {
  constructor(redisUrls) {
    // Redis cluster for high availability
    this.redis = new Redis.Cluster(redisUrls, {
      redisOptions: {
        password: process.env.REDIS_PASSWORD
      }
    });
  }

  getCacheKey(params) {
    const normalized = {
      prompt: params.prompt.toLowerCase().trim(),
      model: params.model,
      width: params.width,
      height: params.height
    };
    
    return `img:v2:${crypto
      .createHash('sha256')
      .update(JSON.stringify(normalized))
      .digest('hex')}`;
  }

  async get(params) {
    const key = this.getCacheKey(params);
    const value = await this.redis.get(key);
    
    if (value) {
      this.recordHit('l2');
      return JSON.parse(value);
    }
    
    this.recordMiss('l2');
    return null;
  }

  async set(params, value, ttl = 86400) {
    const key = this.getCacheKey(params);
    
    await this.redis.setex(
      key,
      ttl,
      JSON.stringify(value)
    );
  }

  // Batch operations for efficiency
  async mget(paramsArray) {
    const keys = paramsArray.map(p => this.getCacheKey(p));
    const values = await this.redis.mget(keys);
    
    return values.map((v, i) => {
      if (v) {
        this.recordHit('l2');
        return JSON.parse(v);
      }
      this.recordMiss('l2');
      return null;
    });
  }
}

Why it works: Shared across all servers, captures 50-55% of traffic. Redis cluster provides HA.

L2 stats in production:

• Hit rate: 53% of non-L1 requests

• Response time: 8-15ms

• Cost: $180/month (managed Redis)

L3: Intelligent Cache Warming

Proactively cache predicted requests before they arrive:

class CacheWarmer {
  constructor(cache, generator) {
    this.cache = cache;
    this.generator = generator;
    this.analytics = new AnalyticsService();
  }

  async warmPopularContent() {
    // Analyze past 7 days of requests
    const popular = await this.analytics.query(`
      SELECT 
        prompt, 
        model, 
        width, 
        height,
        COUNT(*) as frequency
      FROM generation_requests
      WHERE timestamp > NOW() - INTERVAL '7 days'
      GROUP BY prompt, model, width, height
      HAVING COUNT(*) > 10
      ORDER BY frequency DESC
      LIMIT 500
    `);

    console.log(`Warming ${popular.length} popular patterns`);

    // Generate in background with low priority
    for (const pattern of popular) {
      // Check if already cached
      const cached = await this.cache.get(pattern);
      if (cached) continue;

      // Queue low-priority generation
      await this.generator.enqueue({
        params: pattern,
        priority: 'low',
        skipIfExists: true
      });

      // Rate limit to avoid overwhelming system
      await this.sleep(100);
    }
  }

  // Run periodically
  startWarming() {
    // Run every 6 hours
    setInterval(() => {
      this.warmPopularContent().catch(err => {
        console.error('Cache warming failed:', err);
      });
    }, 6 * 60 * 60 * 1000);

    // Run once on startup
    this.warmPopularContent();
  }

  sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

Impact: Raises effective cache hit rate by 8-12% for predictable workloads.

Layer 2: Request Deduplication

When multiple users request identical content simultaneously, generate once:

class RequestDeduplicator {
  constructor() {
    this.pending = new Map(); // In-flight requests
  }

  async deduplicate(key, generatorFn) {
    // Check if request already in flight
    if (this.pending.has(key)) {
      console.log(`Deduplicating request: ${key}`);
      
      // Wait for the in-flight request
      const existingPromise = this.pending.get(key);
      return await existingPromise;
    }

    // Start new request
    const promise = (async () => {
      try {
        const result = await generatorFn();
        return result;
      } finally {
        // Clean up when done
        this.pending.delete(key);
      }
    })();

    this.pending.set(key, promise);
    return await promise;
  }

  // Get statistics
  getStats() {
    return {
      inflightRequests: this.pending.size
    };
  }
}

// Usage in request handler
const deduplicator = new RequestDeduplicator();

app.post('/api/generate', async (req, res) => {
  const params = req.body;
  const key = getCacheKey(params);

  const result = await deduplicator.deduplicate(key, async () => {
    // This function only runs once even if 10 requests arrive simultaneously
    return await generator.generate(params);
  });

  res.json(result);
});

Impact: During traffic spikes, reduces actual generations by 15-20% by serving multiple clients from single generation.

Request deduplication: Multiple simultaneous requests for identical content generate once, serve many

Layer 3: Intelligent Job Queue

Not all generations are equally urgent. Prioritize accordingly:

const Bull = require('bull');

class PrioritizedQueue {
  constructor(redisUrl) {
    this.queue = new Bull('generations', redisUrl, {
      settings: {
        maxStalledCount: 3,
        stalledInterval: 30000
      }
    });

    this.setupProcessors();
    this.setupEventHandlers();
  }

  setupProcessors() {
    // High priority: User-facing requests
    this.queue.process('high-priority', 10, async (job) => {
      return await this.processGeneration(job);
    });

    // Normal priority: Background tasks
    this.queue.process('normal-priority', 5, async (job) => {
      return await this.processGeneration(job);
    });

    // Low priority: Cache warming, experiments
    this.queue.process('low-priority', 2, async (job) => {
      return await this.processGeneration(job);
    });
  }

  async processGeneration(job) {
    const { params, userId, priority } = job.data;
    
    // Track start time
    const startTime = Date.now();

    try {
      // Update progress
      await job.progress(10);

      // Generate with retries
      const result = await this.generateWithRetry(params, 3);

      await job.progress(90);

      // Track metrics
      const duration = (Date.now() - startTime) / 1000;
      this.recordMetrics({
        userId,
        priority,
        duration,
        cost: result.cost,
        model: result.model
      });

      await job.progress(100);

      return result;

    } catch (error) {
      console.error(`Job ${job.id} failed:`, error);
      throw error; // Bull will handle retries
    }
  }

  async generateWithRetry(params, maxAttempts) {
    for (let attempt = 1; attempt <= maxAttempts; attempt++) {
      try {
        return await generator.generate(params);
      } catch (error) {
        if (attempt === maxAttempts) throw error;
        
        // Exponential backoff
        await this.sleep(Math.pow(2, attempt) * 1000);
      }
    }
  }

  // Add job with priority
  async enqueue(params, options = {}) {
    const priority = options.priority || 'normal';
    
    return await this.queue.add(
      `${priority}-priority`,
      {
        params,
        userId: options.userId,
        priority
      },
      {
        priority: this.getPriorityValue(priority),
        attempts: 3,
        backoff: {
          type: 'exponential',
          delay: 5000
        },
        removeOnComplete: 100, // Keep last 100
        removeOnFail: 50
      }
    );
  }

  getPriorityValue(priority) {
    const values = {
      'high': 1,
      'normal': 5,
      'low': 10
    };
    return values[priority] || 5;
  }

  setupEventHandlers() {
    this.queue.on('failed', (job, error) => {
      console.error(`Job ${job.id} failed permanently:`, error.message);
      
      // Alert on high-priority failures
      if (job.data.priority === 'high') {
        this.alertOnError(job, error);
      }
    });

    this.queue.on('stalled', (job) => {
      console.warn(`Job ${job.id} stalled, will retry`);
    });
  }
}

Impact: User-facing requests processed within 5 seconds even when queue depth is 1000+.

Layer 4: Provider Abstraction with Failover

Don't couple to a single provider. Build flexibility:

class MultiProviderGenerator {
  constructor() {
    this.providers = this.initializeProviders();
    this.circuitBreakers = new Map();
  }

  initializeProviders() {
    return [
      {
        name: 'wavespeed-primary',
        client: new WaveSpeedClient({
          apiKey: process.env.WAVESPEED_KEY,
          baseUrl: 'https://api.wavespeed.ai/v1'
        }),
        priority: 1,
        costMultiplier: 1.0
      },
      {
        name: 'wavespeed-backup',
        client: new WaveSpeedClient({
          apiKey: process.env.WAVESPEED_BACKUP_KEY,
          baseUrl: 'https://backup.wavespeed.ai/v1'
        }),
        priority: 2,
        costMultiplier: 1.1
      }
    ];
  }

  async generate(params) {
    const availableProviders = this.providers
      .filter(p => !this.isCircuitOpen(p.name))
      .sort((a, b) => a.priority - b.priority);

    if (availableProviders.length === 0) {
      throw new Error('All providers unavailable');
    }

    for (const provider of availableProviders) {
      try {
        const startTime = Date.now();
        
        const result = await provider.client.generate(params);
        
        const duration = (Date.now() - startTime) / 1000;

        // Record success
        this.recordSuccess(provider.name, duration);

        return {
          ...result,
          provider: provider.name,
          cost: result.cost * provider.costMultiplier
        };

      } catch (error) {
        console.error(`Provider ${provider.name} failed:`, error.message);
        
        this.recordFailure(provider.name, error);

        // Continue to next provider
        if (provider !== availableProviders[availableProviders.length - 1]) {
          continue;
        }

        // All providers failed
        throw error;
      }
    }
  }

  // Circuit breaker pattern
  isCircuitOpen(providerName) {
    const breaker = this.circuitBreakers.get(providerName);
    if (!breaker) return false;

    // Open if error rate > 50% over last 100 requests
    const errorRate = breaker.errors / breaker.total;
    if (errorRate > 0.5 && breaker.total > 100) {
      // Auto-close after 5 minutes
      if (Date.now() - breaker.openedAt > 300000) {
        this.circuitBreakers.delete(providerName);
        return false;
      }
      return true;
    }

    return false;
  }

  recordSuccess(providerName, duration) {
    metrics.providerRequests.inc({ 
      provider: providerName, 
      status: 'success' 
    });
    
    metrics.providerDuration.observe({ 
      provider: providerName 
    }, duration);
  }

  recordFailure(providerName, error) {
    metrics.providerRequests.inc({ 
      provider: providerName, 
      status: 'error' 
    });

    const breaker = this.circuitBreakers.get(providerName) || {
      errors: 0,
      total: 0,
      openedAt: null
    };

    breaker.errors++;
    breaker.total++;

    if (breaker.errors / breaker.total > 0.5 && breaker.total > 100) {
      breaker.openedAt = Date.now();
      console.warn(`Circuit breaker opened for ${providerName}`);
    }

    this.circuitBreakers.set(providerName, breaker);
  }
}

Impact: System stays up even when primary provider has issues. We've had zero downtime from provider outages.

Layer 5: Cost Control and Monitoring

At scale, costs can spiral. Build controls:

class CostController {
  constructor(redis) {
    this.redis = redis;
    this.budgets = {
      hourly: 100,  // $100/hour
      daily: 1500,  // $1500/day
      monthly: 40000 // $40000/month
    };
  }

  async checkBudget(estimatedCost) {
    const now = new Date();
    
    // Check hourly budget
    const hourKey = `budget:hour:${now.toISOString().slice(0, 13)}`;
    const hourlySpent = parseFloat(await this.redis.get(hourKey) || '0');
    
    if (hourlySpent + estimatedCost > this.budgets.hourly) {
      throw new BudgetExceededError('Hourly budget exceeded');
    }

    // Check daily budget
    const dayKey = `budget:day:${now.toISOString().slice(0, 10)}`;
    const dailySpent = parseFloat(await this.redis.get(dayKey) || '0');
    
    if (dailySpent + estimatedCost > this.budgets.daily) {
      throw new BudgetExceededError('Daily budget exceeded');
    }

    // Check monthly budget
    const monthKey = `budget:month:${now.toISOString().slice(0, 7)}`;
    const monthlySpent = parseFloat(await this.redis.get(monthKey) || '0');
    
    if (monthlySpent + estimatedCost > this.budgets.monthly) {
      throw new BudgetExceededError('Monthly budget exceeded');
    }

    return true;
  }

  async recordCost(actualCost) {
    const now = new Date();
    
    const hourKey = `budget:hour:${now.toISOString().slice(0, 13)}`;
    const dayKey = `budget:day:${now.toISOString().slice(0, 10)}`;
    const monthKey = `budget:month:${now.toISOString().slice(0, 7)}`;

    await Promise.all([
      this.redis.incrbyfloat(hourKey, actualCost),
      this.redis.incrbyfloat(dayKey, actualCost),
      this.redis.incrbyfloat(monthKey, actualCost),
      
      // Set expiry
      this.redis.expire(hourKey, 7200),  // 2 hours
      this.redis.expire(dayKey, 172800), // 2 days
      this.redis.expire(monthKey, 5184000) // 60 days
    ]);

    // Alert if approaching limits
    const dailySpent = parseFloat(await this.redis.get(dayKey));
    if (dailySpent > this.budgets.daily * 0.8) {
      this.alertApproachingLimit('daily', dailySpent, this.budgets.daily);
    }
  }

  async getCostReport(period = 'day') {
    const now = new Date();
    let key;

    if (period === 'hour') {
      key = `budget:hour:${now.toISOString().slice(0, 13)}`;
    } else if (period === 'day') {
      key = `budget:day:${now.toISOString().slice(0, 10)}`;
    } else {
      key = `budget:month:${now.toISOString().slice(0, 7)}`;
    }

    const spent = parseFloat(await this.redis.get(key) || '0');
    const budget = this.budgets[period === 'hour' ? 'hourly' : period === 'day' ? 'daily' : 'monthly'];

    return {
      period,
      spent,
      budget,
      remaining: budget - spent,
      percentUsed: (spent / budget) * 100
    };
  }
}

Monitoring Stack

Comprehensive monitoring is critical at scale:

const prometheus = require('prom-client');
const register = new prometheus.Registry();

// Metrics we track
const metrics = {
  // Counters
  requestsTotal: new prometheus.Counter({
    name: 'generation_requests_total',
    help: 'Total generation requests',
    labelNames: ['model', 'cached', 'priority'],
    registers: [register]
  }),

  errorsTotal: new prometheus.Counter({
    name: 'generation_errors_total',
    help: 'Total generation errors',
    labelNames: ['model', 'error_type'],
    registers: [register]
  }),

  // Histograms
  requestDuration: new prometheus.Histogram({
    name: 'generation_duration_seconds',
    help: 'Generation duration in seconds',
    labelNames: ['model', 'cached'],
    buckets: [0.1, 0.5, 1, 2, 5, 10, 30],
    registers: [register]
  }),

  // Gauges
  queueDepth: new prometheus.Gauge({
    name: 'generation_queue_depth',
    help: 'Current queue depth',
    labelNames: ['priority'],
    registers: [register]
  }),

  // Cost tracking
  costTotal: new prometheus.Counter({
    name: 'generation_cost_usd_total',
    help: 'Total cost in USD',
    labelNames: ['model'],
    registers: [register]
  })
};

// Expose metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

// Update queue depth every 10 seconds
setInterval(async () => {
  const counts = await queue.getJobCounts();
  metrics.queueDepth.set({ priority: 'high' }, counts.waiting);
}, 10000);

Dashboard we built:

• Request volume and latency (P50, P95, P99)

• Cache hit rates (L1, L2, total)

• Cost tracking (hourly, daily, monthly)

• Error rates by type and provider

• Queue depth and processing times

• Budget utilization and alerts

Real-time monitoring dashboard showing key metrics: latency, cache performance, costs, and queue depth

Performance Optimization Results

Here's what these patterns achieved:

Latency:

• Before optimization: P95 = 25s, P99 = 60s+

• After optimization: P95 = 8.5s, P99 = 12.8s

• Improvement: 66% reduction at P95

Costs:

• Before optimization: $14,200/month

• After optimization: $4,800/month

• Improvement: 66% reduction

Reliability:

• Before: 93% uptime, 8% error rate

• After: 99.7% uptime, 1.2% error rate

• Improvement: 7.2% uptime gain

Cache Performance:

• L1 hit rate: 12-15%

• L2 hit rate: 53%

• Total hit rate: 68%

• Cost savings from caching: $9,100/month

Lessons Learned

After six months running this at scale:

1. Caching is 10x more important than you think

Our initial cache hit rate was 12%. We thought that was reasonable. After optimization, it's 68%. The difference is $9,000/month in costs and dramatically better latency.

2. Multi-tier caching beats single-tier

L1 (in-memory) + L2 (Redis) performs better than either alone. L1 handles spikes, L2 provides consistency across servers.

3. Provider abstraction saves you

We've had three provider outages in six months. Circuit breakers and automatic failover meant zero user-visible downtime.

4. Monitoring is mandatory at scale

You can't optimize what you don't measure. Comprehensive metrics let us identify bottlenecks and track improvements.

5. Cost control prevents surprises

Budget alerts saved us from a $12,000 surprise when a bug caused a generation loop. Hourly budgets catch issues before they become expensive.

Platform Choice Matters

We use WaveSpeedAI as our primary provider. Why?

Unified API: Access to 100+ models through one integration No Cold Starts: Consistently fast performance Built-in Failover: Automatic retry with alternative models Cost Optimization: Test models easily to find optimal cost/quality ratio Excellent Docs: Comprehensive documentation accelerated our development

The model catalog lets us quickly switch between models based on use case—fast models for drafts, premium models for final assets.

According to TechCrunch's analysis of AI infrastructure, aggregation platforms like WaveSpeedAI reduce integration time by 70-85% compared to direct provider integration, while providing better reliability through built-in failover.

Code Repository

Full implementation available on GitHub: ai-generation-at-scale

Includes:

• Complete Express API

• Multi-tier caching implementation

• Job queue with priorities

• Monitoring and metrics

• Cost control system

• Docker compose for local development

Conclusion

Scaling AI generation to millions of requests requires different patterns than traditional APIs:

✅ Multi-tier caching (L1 + L2 + predictive warming)
✅ Request deduplication (generate once, serve many)
✅ Priority-based queuing (user-facing requests first)
✅ Provider abstraction (circuit breakers, automatic failover)
✅ Cost controls (budget limits, alerts, tracking)
✅ Comprehensive monitoring (latency, costs, errors, queues)

These patterns let our small team handle 3M+ requests monthly with 99.7% uptime and controlled costs.

The architecture scales horizontally—adding servers increases throughput linearly. We've tested to 500 req/sec with no degradation.

If you're building AI generation features at scale, these patterns will save you months of trial and error.

Questions? Drop them in the comments!