Rate limiting and abuse prevention on a public endpoint

Chapter 6 of 8. A public subscribe endpoint is a free email cannon for anybody who finds it, and per-IP rate limiting bounds the blast radius. Prerequisite: the double-opt-in chapter of this course, where you build the endpoint that mints a token and sends a confirmation mail on every accepted POST.

The subscribe endpoint from that chapter works, and working is the whole problem. It accepts a POST from anybody who finds the URL, and then does every expensive thing a real signup does. Nothing in it ever says no.

PHP newsletter tutorials ship a functioning endpoint and a note telling you to add restrictions before you go to production. The code contains no restrictions anywhere. The note names the risk correctly and hands it straight back to the reader, who is reading a tutorial precisely because they do not yet know what the restrictions should be.

This chapter is the code that discharges the note. It is what actually runs on captainrandom.co.uk. A file-backed sliding-window limiter in about 110 lines of PHP. No Redis, no daemon, because a cPanel shared host gives you neither.

What abuse actually looks like on a public subscribe endpoint

Look at what one accepted POST costs you. It writes a pending row to the subscribers table, mints two 64-hex tokens, and sends an email from your domain through PHP's mail() on the shared host. All three are attacker-triggerable by anyone with curl, and none of them require the attacker to own the address they submit.

Only one of the resulting failure modes is the DoS you probably pictured. Mail amplification does the most damage. A script POSTs a victim's address a thousand times, your domain sends a thousand confirmation emails, and the SPF, DKIM and DMARC alignment you got right earlier in this course now works against you. The mail is provably from you, so the reputation that burns is yours. Underneath that sit the quieter ones. Shared cPanel hosts cap outbound mail per hour, so a flood silently kills every legitimate confirmation email for the rest of the window. Unbounded pending rows accumulate unless something purges them, and each one holds an email address you are now the data controller for.

The OWASP Denial of Service Cheat Sheet gives an ordering principle before it gives any technique. Perform the cheap validation first, so expensive work is never spent on hostile input. In this endpoint the expensive work is the DB round trip and the mail() call. Everything that can reject a request must sit in front of them.

So subscribe.php writes its pipeline out as a docblock, with the rate limit at stage three of eight:

/**
 * POST /api/subscribe/  — public, unauthenticated.
 *
 * Pipeline:
 *   1. Method check         — POST only, else 405
 *   2. Origin check         — Origin header must match SITE_URL, else 403
 *   3. Per-IP rate limit    — Throttle::check(); 429 on exceed
 *   4. Honeypot + time-trap — silently succeed if bot-shaped
 *   5. Email validation     — silently succeed if invalid (no oracle)
 *   6. DB upsert            — pending row created OR token re-issued on
 *                             existing pending row; existing confirmed
 *                             or unsubscribed rows are NOT mutated
 *   7. Confirmation mail    — sent only if a token was created/re-issued
 *   8. Redirect             — always to /newsletter/check-your-inbox/
 *                             regardless of what happened above
 */

Stages one and two are string comparisons. Stage three is one file read under a lock. Nothing touches MySQL until stage six, and nothing sends mail until stage seven. An attacker firing a flood gets stopped for the cost of a stat and a flock.

A per-IP sliding window in PHP with no Redis

Every rate-limiting tutorial assumes Redis or a token-bucket middleware. Shared hosting gives you neither. What it does give you is a writable home directory and flock(), and that is enough.

lib/Throttle.php declares its policy as four constants:

final class Throttle
{
    public const LIMIT_PER_MIN = 20;
    public const LIMIT_PER_DAY = 50;
    public const WINDOW_MIN = 60;
    public const WINDOW_DAY = 86400;

Two windows, both per IP, and both must pass:

$state['min'] = self::trim($state['min'] ?? [], $now, self::WINDOW_MIN);
$state['day'] = self::trim($state['day'] ?? [], $now, self::WINDOW_DAY);

$minCount = count($state['min']);
$dayCount = count($state['day']);
$allowed = $minCount < self::LIMIT_PER_MIN && $dayCount < self::LIMIT_PER_DAY;

The two limits catch different attackers. Twenty per minute stops a burst. A patient script that submits one request every ninety seconds forever never touches the per-minute ceiling. Twenty per minute on its own lets that script send 960 emails a day from your domain. The daily limit of 50 exists to catch exactly that, and a single-window limiter is blind to it entirely.

trim() makes the window slide rather than reset:

private static function trim(array $timestamps, int $now, int $window): array
{
    $cutoff = $now - $window;
    return array_values(array_filter($timestamps, static fn($t) => $t >= $cutoff));
}

State is a list of request timestamps, and every check drops the ones that have aged out. A fixed bucket keyed on the current minute lets an attacker send 20 requests at 12:00:59 and 20 more at 12:01:00, so 40 land in two seconds. A sliding window will not, because the first 20 are still inside the last 60 seconds.

Storage is a flat JSON file per IP, and the filename is the interesting part:

private static function pathFor(string $ip): string
{
    $salt = (string) ($_ENV['HASH_SALT'] ?? '');
    $hash = hash('sha256', $ip . '|' . $salt);
    $dir = ($_ENV['THROTTLE_DIR'] ?? (($_ENV['HOME'] ?? '/tmp') . '/captainrandom-newsletter/throttle'));
    return $dir . '/' . $hash . '.json';
}

The raw IP never reaches disk. The ICO is explicit that IP addresses are online identifiers capable of being personal data, and Recital 30 names them directly. A limiter that writes 192.0.2.44.json into your home directory has built a second, undocumented store of subscriber IPs, sitting outside your privacy policy. Hashing the key with the server-side HASH_SALT gives you the same stable correlation key with nothing readable at rest. The directory is created 0700 and the file is opened c+. Every read-modify-write then runs inside flock($fp, LOCK_EX), with the unlock in a finally, so two concurrent requests from the same IP cannot both read the same count and both decide they are under the limit.

The standard 429 headers, and who is supposed to emit them

Throttle.php never calls header(). It returns an array and nothing else:

/**
 * @return array{allowed:bool, limit:int, remaining:int, reset:int, retryAfter:int}
 */
public static function check(string $ip): array

The caller owns the HTTP shape, and in subscribe.php that is thirteen lines:

$ip = (string) ($_SERVER['REMOTE_ADDR'] ?? '');
$throttle = Throttle::check($ip);
header(sprintf('X-RateLimit-Limit: %d',     $throttle['limit']));
header(sprintf('X-RateLimit-Remaining: %d', $throttle['remaining']));
header(sprintf('X-RateLimit-Reset: %d',     $throttle['reset']));
if (!$throttle['allowed']) {
    header(sprintf('Retry-After: %d', $throttle['retryAfter']));
    logSafely('rate_limited', null, [
        'ip_hash' => hash('sha256', $ip . '|' . ($_ENV['HASH_SALT'] ?? '')),
        'limit'   => $throttle['limit'],
    ]);
    fail(429, 'too many requests');
}

Three things in that block are worth copying and one is worth criticising.

Every request that reaches the limiter gets the X-RateLimit-* headers, not only the blocked ones. A client that first learns its budget at the moment it is refused cannot back off gracefully. X-RateLimit-Reset is Unix epoch seconds, so a client comparing it against its own clock gets a real answer. Retry-After appears on the block path alone. The spec makes it optional even there, but a 429 without it tells the client nothing about when to come back.

The 429 body is the string too many requests, set by fail() alongside Cache-Control: no-store and a plain-text content type. Generic and non-descriptive, which is where the OWASP Error Handling Cheat Sheet lands too: keep the detail server-side. Server-side, that detail goes into the events table as a rate_limited row carrying a salted ip_hash and the limit that was hit, never the raw address.

Now the criticism, because a course shipping its own production code owes you its defects. $throttle['limit'] is always LIMIT_PER_MIN, hard-coded to 20. When the daily ceiling of 50 is what blocked you, the emitted X-RateLimit-Limit: 20 describes the wrong window, and retryAfter is computed from the minute window as max(1, $reset - $now), so it can tell a client to retry in 40 seconds when the real wait is hours. The client is still blocked, so nothing leaks. What breaks is the advice the headers give, and it breaks because the return shape was designed around one window before a second was added underneath it.

The checks a rate limit cannot make: honeypot and time-trap

Twenty per minute is a generous ceiling for a human and a mild inconvenience for a botnet with a thousand addresses. Rate limiting bounds volume per source. It does not tell you whether the source is a person. Two cheap checks in lib/Validator.php do that, and they run at stage four, still ahead of the DB:

public static function honeypotOk(?string $value): bool
{
    return $value === null || $value === '';
}

public static function timeTrapOk(?string $renderedAtRaw, int $now): bool
{
    if ($renderedAtRaw === null || $renderedAtRaw === '' || !ctype_digit($renderedAtRaw)) {
        return false;
    }
    $renderedAtSec = (int) ($renderedAtRaw / 1000);
    if ($renderedAtSec <= 0) {
        return false;
    }
    $elapsed = $now - $renderedAtSec;
    return $elapsed >= self::MIN_RENDER_AGE_SECONDS
        && $elapsed <= self::MAX_RENDER_AGE_SECONDS;
}

The honeypot is a hidden input named website. A human never fills it, so anything caught there is bot-shaped and gets a silent 302 to /newsletter/check-your-inbox/, exactly as if it had succeeded. The time-trap reads a rendered_at field the form ships as milliseconds since epoch, and requires between MIN_RENDER_AGE_SECONDS and MAX_RENDER_AGE_SECONDS to have elapsed since the page rendered. A stale cached form trips the 24-hour ceiling; a submit-in-zero-milliseconds bot trips the floor.

MIN_RENDER_AGE_SECONDS is 1. It was 2, and it rejected the first real human who used the form. Browser autocomplete pre-filled the email, the click came about a second later, and two consecutive bot_detected rows landed in events with reason: time_trap. The whole sequence is in the newsletter hotfixes article, including the second bug the new instrumentation caught within the hour, where React's hydration pass overwrote rendered_at back to its defaultValue of "0".

The constant was only half of it. Silent rejection is correct for a signal a human cannot trip, and dishonest for one they can, so the routing split in two. Honeypot and Origin mismatch stay silent, because nobody legitimate ever trips them. A failed time-trap now redirects to a real /newsletter/please-try-again/ page that explains the autocomplete-and-fast-click case and offers a manual escape hatch.

That leaks which check fired, and it was accepted with the reasoning written into Validator.php so it does not decay. An attacker already has to beat the honeypot, the Origin check, the per-IP limit and the email validator, all of which stay silent. The time-trap was the weakest signal in the stack, since any halfway-competent scraper already waits before submitting. Trading it for a user who does not vanish into a confirmation email that never arrives is a good trade.

Verifying the limit against the live endpoint

There is no test suite for this backend. php -l is all it gets before deploy, and both of the bugs above went past it and past code review into production. Nothing proves a rate limit works except exceeding it.

1. Send one well-formed POST and read the headers

   The Origin check runs before the throttle, so curl must send it or you get a 403 and never reach stage three.

   curl -sS -o /dev/null -D - -X POST \
     -H 'Origin: https://captainrandom.co.uk' \
     --data 'email=' \
     https://captainrandom.co.uk/api/subscribe/
   

   You should see X-RateLimit-Limit: 20, X-RateLimit-Remaining: 19, and an X-RateLimit-Reset epoch about 60 seconds ahead. With no rendered_at posted, the request dies at the time-trap in stage four, so no subscriber row is written and no mail is sent. A bot_detected event row is, for the reason the callout below spells out. The request still consumes throttle budget, because stage three ran first.

2. Fire 21 more requests and watch the window close

   for i in $(seq 1 21); do
     curl -sS -o /dev/null -w '%{http_code} ' -X POST \
       -H 'Origin: https://captainrandom.co.uk' \
       --data 'email=' \
       https://captainrandom.co.uk/api/subscribe/
   done
   

   Nineteen 302s, then two 429s. Do the arithmetic before you blame the limiter. Step one already spent request number 1 of the 20 the minute window allows, so this loop only has 19 left before the block. Throttle::check() appends to $state['min'] on every request it allows, whatever rejects that request further down the pipeline. Run the loop alone on a fresh window and you get 20 302s and one 429. The count you see depends on what you did in the previous 60 seconds.

3. Confirm it was logged, not just blocked

   SELECT kind, at, meta FROM events
    WHERE kind = 'rate_limited'
    ORDER BY at DESC LIMIT 5;
   

   You should see one row per blocked request, each with a salted ip_hash in meta and no raw address anywhere. A block that leaves no trace is a block you cannot investigate.

One thing this setup gets wrong, and you will too

The docblock at the top of Throttle.php says "The nightly cron purges files whose newest entry is >24h old." It does not. cron/purge.php contains no filesystem code at all, so the per-IP JSON files accumulate in ~/captainrandom-newsletter/throttle/ indefinitely. Each one is small, and each one is a salted hash of an IP address with a timestamp list attached.

The ICO's storage limitation guidance is that you must not keep personal data longer than you need it, and that a retention schedule should list every type of record you hold. My retention schedule covers subscribers and events. It does not cover the throttle directory, because when I wrote it I was thinking about the database, and the throttle files did not feel like a store. They are one. The comment claiming the cleanup exists is worse than no comment, because it stopped me looking.

Rate limiting creates state about people. Put it in the retention policy the day you write it.

The transferable rule

Every other control in this pipeline is defeated by reading the page source. Both the honeypot field name and the rendered_at value sit in the HTML, and the Origin header is one curl flag away. An attacker who spends five minutes on your form beats all three and never trips a single bot_detected row.

The per-IP window is different. It costs the attacker a resource they cannot fabricate from the browser, and it is the only control on the list that still works against somebody who has read your code. OWASP's guidance on emailed tokens arrives at the same place from the other direction. It tells you to implement protections against excessive automated submissions so an attacker cannot flood a victim with mail, and it names per-account rate limiting first among those protections, ahead of CAPTCHA and the other equivalent controls.

So when a tutorial hands you a working subscribe endpoint and a note telling you to add restrictions before production, treat that note as a specification. It is telling you the endpoint is unfinished. Twenty per minute and fifty per day, keyed on a salted hash, with the three X-RateLimit-* headers on every request that reaches the limiter and a Retry-After on the refusal, is about 110 lines of PHP and no new dependencies. There is no host so limited that you cannot afford it.

Chapter 7 picks up the other end of the lifecycle: the unsubscribe link, the soft delete, and the 30-day hard delete that turns Article 17 into a cron job.