> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/joey727/Phisherman/llms.txt
> Use this file to discover all available pages before exploring further.

# How it works

> A walkthrough of the Phisherman checker pipeline from HTTP request to verdict.

Phisherman processes every URL through a multi-stage pipeline: a cache lookup, parallel checker execution, score aggregation, and a cache write. This page walks through each stage in detail.

## Request lifecycle

```
HTTP request
     │
     ▼
 Rate limiter
     │
     ▼
 Redis cache check  ──── hit ────▶  return cached ScanResult
     │ miss
     ▼
 CheckerRegistry.runAll(url)
  ┌──┴─────────────────────────────────────┐
  │  heuristics  openphish  gsb  urlhaus   │  (parallel, 2500 ms timeout each)
  │  phishtank   phishstats               │
  └──┬─────────────────────────────────────┘
     │ []CheckResult
     ▼
 Aggregate score  →  Math.min(100, sum)
     │
     ▼
 Determine verdict  (safe / suspicious / phishing)
     │
     ▼
 Cache result in Redis  (if non-safe, or SCAN_CACHE_SAFE_RESULTS=true)
     │
     ▼
 Return ScanResult
```

## The CheckerRegistry

`CheckerRegistry` is a small registry class that holds a list of `Checker` objects and runs them all in parallel.

```typescript theme={null}
// src/CheckerRegistry.ts
class CheckerRegistry {
    private checkers: Checker[] = [];

    register(checker: Checker) {
        this.checkers.push(checker);
    }

    async runAll(url: string): Promise<{ checks: CheckResult[]; timing: Record<string, number> }> {
        const timing: Record<string, number> = {};
        const TIMEOUT_MS = 2500; // 2.5s maximum per checker

        const checks = await Promise.all(
            this.checkers.map(async (checker) => {
                const start = Date.now();
                try {
                    const checkPromise = checker.check(url);
                    const timeoutPromise = new Promise<CheckResult>((_, reject) =>
                        setTimeout(() => reject(new Error("Timeout")), TIMEOUT_MS)
                    );

                    const result = await Promise.race([checkPromise, timeoutPromise]);
                    timing[checker.name] = Date.now() - start;
                    return result;
                } catch (err: any) {
                    timing[checker.name] = Date.now() - start;
                    if (err.message === "Timeout") {
                        console.warn(`Checker ${checker.name} timed out for ${url}`);
                        return { score: 0, reason: `Checker ${checker.name} timed out` };
                    }
                    console.error(`Checker ${checker.name} failed:`, err);
                    return { score: 0, reason: `Checker ${checker.name} error` };
                }
            })
        );

        return { checks, timing };
    }
}
```

Key properties of this design:

* All checkers run concurrently via `Promise.all` — latency is bounded by the slowest checker, not the sum of all checkers.
* Each checker races against a **2500 ms timeout** via `Promise.race`. A timed-out checker contributes a score of `0` and does not fail the request.
* Per-checker execution time is recorded in the `timing` map and returned in the `ScanResult` as `executionTimeMs`.

### Checker registration

Checkers are registered in `Scanner.ts` at startup:

```typescript theme={null}
// src/Scanner.ts
registry.register(HeuristicsChecker);
registry.register(OpenPhishChecker);
registry.register(SafeBrowsingChecker);
registry.register(URLHausChecker);
registry.register(PhishTankChecker);
// registry.register(WebRiskChecker);  // disabled
registry.register(PhishStatsChecker);
```

Each checker implements the `Checker` interface:

```typescript theme={null}
// src/types.ts
export interface Checker {
    name: string;
    check: (url: string) => Promise<CheckResult>;
}
```

## Result caching

Scan results are cached in Redis to avoid re-running the full checker pipeline for recently seen URLs.

```typescript theme={null}
// src/Scanner.ts
const RESULT_CACHE_TTL_SECONDS = 300; // 5 minutes
const SCAN_CACHE_HASH = "scan_results";
const SCAN_CACHE_EXPIRY_ZSET = "scan_results_expiry";
```

**Cache key** — The URL is hashed with SHA-256:

```typescript theme={null}
function scanCacheId(url: string) {
  return crypto.createHash("sha256").update(url).digest("hex");
}
```

**Storage structure** — To avoid Redis key explosion, all scan results are stored as fields of a single hash (`scan_results`). A companion sorted set (`scan_results_expiry`) stores each field ID with its expiry timestamp as the score, enabling efficient batch cleanup.

**Cache read** — On a cache hit, Phisherman checks the `exp` field against `Date.now()`. Expired entries are deleted opportunistically before the fresh scan runs.

**Caching safe results** — By default, URLs that resolve to a `safe` verdict are not cached, because they are high-volume and low-value to retain. Set `SCAN_CACHE_SAFE_RESULTS=true` to cache them:

```typescript theme={null}
const CACHE_SAFE_RESULTS = (process.env.SCAN_CACHE_SAFE_RESULTS || "").toLowerCase() === "true";

// ...
if (CACHE_SAFE_RESULTS || result.verdict !== "safe") {
  // write to cache
}
```

## Background feed refresh

`CacheManager` runs a background loop that keeps all threat feed data current. It is started once at server startup:

```typescript theme={null}
// src/CacheManager.ts
async start(intervalMs: number = 3600000) { // Default 1 hour
    if (this.interval) return;
    await this.runAll();  // run immediately on startup
    this.interval = setInterval(() => this.runAll(), intervalMs);
}
```

On each cycle, `runAll()` invokes every registered `RefreshTask` in sequence, then runs three cleanup routines:

| Cleanup step           | What it removes                                              |
| ---------------------- | ------------------------------------------------------------ |
| `cleanupScanResults()` | Expired entries from `scan_results` hash + ZSET              |
| `cleanupWhois()`       | Expired WHOIS lookups from `whois_data` hash + ZSET          |
| `cleanupHashCaches()`  | Expired entries from GSB, GWR, and DNS `HashCache` instances |

Each feed source registers its own refresh task with a source-specific interval:

| Source     | Refresh interval |
| ---------- | ---------------- |
| URLHaus    | 5 minutes        |
| OpenPhish  | 15 minutes       |
| PhishTank  | 60 minutes       |
| PhishStats | 90 minutes       |

<Tip>
  Feed refresh is checked on every invocation — not on a separate per-source timer. The `CacheManager` loop fires every hour by default, but each source compares `Date.now()` against its own last-update timestamp and only refetches if its individual interval has elapsed.
</Tip>