Prompt and context
This is a product judgment question. A developer API must protect shared resources without making valid integrations fail through opaque 429 responses. Decide what to limit, which identity owns the limit, how tiers differ, how clients discover boundaries early, and how to prove the policy improves reliability without blocking valuable work.
Assume free, professional, and enterprise tiers; request cost varies with request count, concurrency, input size, and downstream compute; a burst must not exhaust a shared dependency; and the policy can be rolled out and rolled back. Example values such as 60 RPM, 10,000 calls per month, or an 80% warning are placeholders to replace with real data, not industry standards.
What the interviewer is testing
The interviewer wants product strategy tied to technical constraints:
- defining rate, usage quota, concurrency, and cost budget as different promises;
- starting from the developer’s job to be done rather than only from a plan price;
- explaining the fairness and abuse trade-offs of limiting by organization, project, API key, user, or IP;
- providing predictable errors, headers, dashboards, alerts, and a path to request more capacity;
- measuring success rate, retry amplification, resource cost, retention, and support burden as guardrails;
- planning a canary, exceptions, appeals, rollback, and policy-change communication.
A weak answer lists “low limits for free and high limits for enterprise.” A strong answer states what each limit protects, what developers experience, and how side effects will be verified.
Clarifying questions to ask
Ask constraints that change the policy:
- What resource is protected? Gateway CPU, database connections, model tokens, a third-party bill, or fair share for one tenant? Each may need a different dimension.
- What shape is the workload? Steady requests, short bursts, batch jobs, and long-lived connections cannot share one per-minute number.
- Is the boundary hard or a soft budget? A hard boundary protects safety; a soft budget may queue, degrade, or charge extra, but must give clear feedback.
- Are billing and throttling measured in the same unit? Tokens, calls, and concurrent connections may need separate counters; one blended metric is hard to forecast.
- What is success? Fewer dependency overloads, more first successful calls, better gross margin, or higher high-value developer retention? The ordering changes the policy.
A 30-second answer framework
Open with this:
“I separate four promises: short-window rate limits protect instantaneous capacity, a billing-cycle quota controls budget, a concurrency cap protects occupied execution slots, and a cost budget prevents an unpredictable bill. I would use organization or project as the main identity and pair request and resource dimensions to endpoint cost. Tiers would add budget, burst capacity, concurrency, and support response rather than only raising RPM. Documentation, dashboards, and response headers would show remaining capacity, reset time, and Retry-After. I would shadow-evaluate the policy first, canary it to a small set of tenants, and watch success rate, retry amplification, cost, upgrades, and retention before expanding or rolling back.”
This states units, developer experience, and the measurement loop before technical or commercial follow-ups.
Step-by-step deep dive
Turn boundaries into a product contract. A rate limit asks how many requests fit in a short window. A usage quota asks how much resource fits in a billing period. A concurrency cap asks how many execution slots can be occupied at once. Keeping them separate lets a customer tell whether a 429 means a burst, an exhausted period budget, or full concurrency. Long jobs may also need a maximum runtime or queue budget.
Derive units from workload, not plan names. Cheap reads can use request counts. Expensive operations may need input bytes, output tokens, compute seconds, or database scan units. Map the critical customer workflow first, then choose endpoint-specific units. Do not promise one RPM that represents every endpoint’s cost.
Choose identity and fairness. Organization or project is usually a better paid-API identity than IP: NAT combines customers, while rotating keys can bypass a key-only limit. Use organization budgets with project concurrency, and use IP as an abuse guardrail. Enterprise exceptions must be auditable; a sales promise must not bypass platform policy.
Design tier differences. The free tier should complete a small end-to-end trial. The professional tier can add period budget, burst capacity, and concurrency. Enterprise can add reserved capacity, compliance controls, or dedicated support. Every tier must state the over-limit behavior: reject, queue, degrade, or charge by usage. Raising a number without improving predictability usually increases support load.
Make client behavior predictable. Documentation, dashboards, and response headers show remaining capacity, reset time, and request ID. On 429, clients can honor Retry-After; exponential backoff with a retry cap avoids a retry storm. Do not encourage retries for non-retryable business errors. Return stable error codes and a next action instead of only “Too Many Requests.”
Define launch and guardrail metrics. Primary measures can include first successful call rate, valid-request 429 rate, share of customers reaching 80% of period budget, unit-request margin, support contacts, upgrade conversion, and 30-day retention. Guardrails include retry amplification, downstream p99, abuse events, and cross-tenant contention. Segment every measure by tier, endpoint, and workload shape so averages do not hide damage.
Canary, exceptions, and rollback. First calculate the new policy in shadow mode for internal projects and a few customers without changing responses. Compare it with the old policy before enforcing it. Offer a migration window, budget alerts, and a temporary capacity process. If valid-call success or retry amplification crosses a threshold, roll back the policy version instead of editing a database row by hand.
Test with explicit assumptions. Suppose a professional project sends 40 steady requests per minute and bursts to 120. A 60-RPM limit serves the steady path but creates repeated 429s during the burst. A sample policy might permit 60 RPM with burst capacity 120 and count the monthly budget separately; this is illustrative. Load tests should cover bursts, concurrency, multiple projects sharing an organization budget, clock boundaries, retrying clients, and policy propagation after an upgrade.
High-quality sample answer
“I would separate the protected resource from the developer promise. Rate limits protect short-window capacity, a period quota controls budget, and a concurrency cap protects occupied execution slots; expensive endpoints add input-size or compute units. I would measure primarily by organization and secondarily by project, using IP only as an abuse guardrail so NAT does not combine valid customers. The free tier must complete a small end-to-end trial, the professional tier adds period budget, burst, and concurrency, and enterprise adds reserved capacity, compliance controls, and an auditable temporary increase. Each tier states whether over-limit work is rejected, queued, degraded, or billed.
Clients see remaining capacity, reset time, and request ID in documentation, dashboards, and headers. A 429 follows Retry-After, and the SDK uses capped exponential backoff to avoid duplicate-request amplification. I would shadow-evaluate the new policy, then canary it to a few projects. Primary measures are first successful calls, valid-call 429 rate, unit-request cost, and upgrade conversion; guardrails are retry amplification, downstream p99, support contacts, and 30-day retention. If valid-call failure crosses the threshold, I roll back the policy version and extend migration rather than opening an unaudited exception for one customer.”
The point is the connection among product units, predictability, technical behavior, and experiment decisions. Replace every number with a real baseline; example quotas are not defaults.
Common mistakes
- Only saying “low free tier, high enterprise tier.” That omits the protected resource and over-limit behavior. Tie every limit to a resource, customer job, and feedback path.
- Treating RPM as total cost. Large requests and long jobs consume more downstream capacity. Add resource or concurrency dimensions.
- Using IP as the only customer identity. NAT, proxies, and key rotation break fairness and auditability. Use organization or project as the primary key and IP as a guardrail.
- Making clients retry blindly. Immediate retries amplify congestion. Honor
Retry-After, use exponential backoff, and cap attempts. - Watching only revenue or 429 rate. Higher revenue can coincide with lower success, while low 429 can mean over-reservation. Track experience, cost, and reliability together.
- Replacing policy with a sales exception. An undocumented back door is hard to revoke and unfair. Use a time-bounded, auditable, reversible capacity process.
Follow-up questions and responses
A customer gets 429s during a burst while much monthly quota remains. What do you do?
Explain that a period quota and instantaneous rate are different promises. Inspect endpoint cost and the customer workflow, then add a justified burst allowance or a temporary burst request path. Document reset and wait behavior in headers, docs, and the dashboard. Do not raise every customer’s RPM without evidence.
A large customer asks to bypass rate limits. Do you agree?
Ask for workload, dependency cost, and reliability targets. Offer reserved capacity, a dedicated queue, or a time-bounded increase while retaining system-wide safety guardrails. Record approval, price, expiry, and rollback. An unlimited promise transfers risk to other customers and downstream services.
429 rate falls but retention also falls. How do you decide whether the policy failed?
Slice by endpoint, tier, workload, and migration phase. Check whether a wider limit hid cost or latency, then combine first successful calls, support contacts, budget alerts, and churn interviews. If only critical workflows suffer, fix units, documentation, or burst policy before removing all guardrails.
Customers share a budget across projects. How do you stop one project exhausting the organization allowance?
Use two levels: an organization budget plus project concurrency or reservations. Expensive or risky endpoints can require a project reservation. Show both remaining values in the dashboard and identify whether an error is project-level or organization-level so an administrator can adjust budget or priority.