Multimodal Phishing Triage: When URL Reputation Is Not Enough
A data-sovereign phishing triage pipeline that combines headless browser detonation, email header forensics, and local VLM analysis to distinguish credential harvesters from complex marketing mailers.
Multimodal Phishing Triage: When URL Reputation Is Not Enough
URL reputation alone fails against modern phishing. Attackers cloak payloads from automated scanners; legitimate marketing mailers trigger false positives through tracking redirects and look-alike branding.
I built a multimodal forensic triage pipeline that detonates URLs in a stealth headless browser, extracts email authentication context, and passes both visual and technical evidence to a locally hosted Llama 3.2-Vision model — keeping sensitive employee reports out of third-party APIs.
1. The Stealth Detonator (The Eyes)
Before we can analyze a site, we have to see it without being caught. Attackers often serve a benign page if they detect a headless browser. We use playwright-stealth to mask our automation footprint and simulate real human interaction.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
async def scan_url(url: str):
# Wrap the entire browser context in a stealth layer
async with Stealth().use_async(async_playwright()) as p:
browser = await p.chromium.launch(headless=True)
context = await browser.new_context(
viewport={'width': 1920, 'height': 1080},
user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)..."
)
page = await context.new_page()
# Navigate and capture the visual evidence
await page.goto(url, wait_until="networkidle")
# Human-like jitter to bypass basic anti-bot
await page.mouse.move(random.randint(100, 700), random.randint(100, 700))
screenshot_path = f"static/screenshots/scan_{uuid.uuid4().hex}.png"
await page.screenshot(path=screenshot_path)
return {"local_path": screenshot_path, "final_url": page.url}
2. The Forensic Ingestion (The Context)
Context is king. An email might look visually perfect, but a mismatch between the From address and the Return-Path is a massive red flag. Our parser extracts these technical “breadcrumbs” to feed the AI.
1
2
3
4
5
6
7
8
9
10
def extract_bundle(raw_eml: str):
msg = email.message_from_string(raw_eml, policy=policy.default)
headers = {
"from": msg.get("From"),
"return_path": msg.get("Return-Path"),
"spf": msg.get("Authentication-Results", "None")
}
# Extracting the body and truncating to stay within AI context limits
body = msg.get_body(preferencelist=('plain')).get_content()
return {"headers": headers, "body": body[:1500]}
3. The Multimodal Brain (The Verdict)
The magic happens when we hand the Screenshot + Headers + Body to Llama 3.2-Vision. Instead of just “reading” the text, the AI “looks” at the branding and compares it to the technical data.
Because local models can be “chatty,” we implemented a Heuristic Recovery layer to ensure we always get a structured verdict, even if the AI decides to write a three-paragraph review.
1
2
3
4
5
6
7
8
9
10
11
# The 'Heuristic Recovery' logic for stubborn local models
# Prioritizes JSON, falls back to Regex keyword extraction
verdict_pattern = r"(?:Verdict|Answer|Conclusion):\s*\*?\*?(CLEAN|SUSPICIOUS|MALICIOUS)\*?\*?"
match = re.search(verdict_pattern, raw_ai_output, re.IGNORECASE)
if match:
return {
"verdict": match.group(1).upper(),
"confidence": 85,
"status": "success"
}
Conclusion: Privacy-First Triage
Running this stack locally via Ollama and FastAPI produces a data-sovereign triage loop — the same architectural principle I later applied at enterprise scale: sensitive security telemetry stays inside the trust boundary, the model sees labeled evidence rather than raw authority, and verdicts are structured for analyst workflow integration.
In production phishing pipelines at Atlassian, this approach automated closure of ~80% of benign employee reports while escalating high-confidence malicious submissions to incident response — without shipping report content to external vector databases.
The hard problem is not building a classifier. It is designing the trust boundary so visual similarity, header mismatch, and redirect chains are weighed together without the model treating any single attacker-controlled field as instruction.
This work predates my Wrike role and reflects the same AI security principles I write about in AI Security — multimodal evidence, local inference, structured verdicts.