AI Pentester

Modern infrastructure changes continuously as teams deploy code and update cloud workloads multiple times a day.

Yet, traditional penetration testing remains fundamentally periodic and manual:

1. The Cadence Mismatch: Most organizations run penetration tests once a year or once a quarter. A static assessment is obsolete the moment a developer merges a configuration change or exposes a new test service the following Monday.
2. Low Signal of Checklist Scanners: Automated scanners compensate for speed by running brute-force signature sweeps. They produce massive PDF reports with thousands of uncontextualized alerts, low-severity noise, and false positives that overwhelm security teams.
3. Manual Detective Burnout: High-quality penetration testing is creative, iterative detective work. Most of the time, a human tester discovers a subdomain, notices an unusual HTTP header, maps the route, and chains subtle misconfigurations into a verified finding. However, repeating this reconnaissance loop across hundreds of domains is quite exhausting and leads to overlooked assets.
4. Operational Safety Hazards: Unconstrained automated scanners risk causing Denial of Service (DoS), touching out-of-scope third-party infrastructure, corrupting production databases, or triggering aggressive cloud provider rate limits.

We engineered AI Pentester to solve this problem from first principles: an autonomous continuous penetration testing platform that models the scientific method of a human security researcher.

Instead of running a fixed pipeline or giving an LLM unchecked terminal access, AI Pentester uses Mastra agent orchestration (similar to LangGraph) paired with a deterministic ScopeGuard, a tool registry, and an interface built on Svelte and PostgreSQL.

AI Pentester interface overview showing the attack surface graph, hierarchical ledger, and active campaign metrics.

---

High-Level System Architecture

AI Pentester separates probabilistic AI reasoning from deterministic safety enforcement and tool execution.

The system is organized into five core decoupled subsystems:

flowchart TD
    subgraph UI["Investigative Workbench"]
        GRAPH["Attack Surface Graph"]
        LEDGER["Investigation Ledger"]
        FINDINGS["Triage Console"]
        APPROVAL["Approval Modal"]
        REPORTS["Report Views"]
    end

    subgraph API["Control Plane"]
        ROUTER["API & SSE Stream"]
    end

    subgraph Core["Agentic Orchestration"]
        ORCH["Campaign Orchestrator"]
        COORD["Lead Coordinator"]
        SELECTOR["Action Selector"]
        SYNTH["Lead Synthesizer"]
    end

    subgraph Safety["Deterministic Governance"]
        GUARD["ScopeGuard Engine"]
        FINGERPRINT["Attempt Fingerprinter"]
        BUDGET["Budget Governor"]
    end

    subgraph Execution["Tool Execution"]
        REGISTRY["Tool Registry"]
        RUNNER["Scanner Runner"]
        TOOLS["Security Scanners"]
        NORM["Output Normalizer"]
    end

    subgraph Storage["Audit Persistence"]
        DB["PostgreSQL Ledger"]
    end

    UI <--> API
    API <--> ORCH
    ORCH --> COORD
    COORD <--> DB
    ORCH --> SELECTOR
    SELECTOR --> REGISTRY
    ORCH --> GUARD
    GUARD --> FINGERPRINT
    GUARD --> BUDGET
    GUARD -->|"Active Risk"| APPROVAL
    APPROVAL -->|"Approved"| ORCH
    ORCH --> RUNNER
    RUNNER --> TOOLS
    TOOLS --> NORM
    NORM --> SYNTH
    SYNTH --> DB
    DB --> ROUTER

Architectural Subsystems

1. Interface (Frontend): A console built with Svelte 5 runes, Tailwind, and Sigma.js canvas rendering. Operators can visualize expanding attack trees in real time, inspect hypotheses, approve or deny high-risk probes, and triage confirmed vulnerabilities.
2. Control Plane & SSE Event Stream: A lightweight TypeScript service built on Hono that provides REST endpoints for campaign lifecycle management and streams real-time state deltas over Server-Sent Events (SSE).
3. Mastra Agent Orchestrator: The brain of the detective loop. It tracks active hypotheses (Leads), queries the tool registry for applicable scanners based on target kind, and prompts the LLM to formulate an explicit security hypothesis and expected signal.
4. Safety Policies: An uncompromising, non-LLM policy layer. Every single probe proposed by the AI must pass strict target authorization, private/reserved IP blocking, port boundaries, duplicate fingerprint filtering, and resource budget checks before a tool is used.
5. Contract-Driven Tool Execution: A sandboxed runner that executes industry-standard security binaries (subfinder, dnsx, httpx, katana, nuclei), capturing raw stdout, stderr, and network artifacts, and normalizing them into typed observation records.
6. Relational Persistence Layer: PostgreSQL schema managed via Drizzle ORM that maintains an immutable audit trail of every asset, exposure, hypothesis, tool execution, payload evidence, and verified finding.

---

The Detective Metaphor: Modeling the Scientific Method

A major failure mode of early autonomous AI experiments is treating the agent as a conversational chatbot with a bash shell. Prompting an LLM with “Here is a domain, hack it and report what you find” results in hallucinations, runaway infinite loops, or illegal out-of-scope requests.

Instead, AI Pentester models a disciplined human investigator using a formal domain abstraction:

Detective       = Mastra Agent + Action Selector
Notebook        = PostgreSQL (Drizzle ORM)
Toolbox         = Contract-Driven Tool Registry (subfinder, httpx, nuclei, etc.)
Security Guard  = ScopeGuard Policy Engine
Case            = Campaign
Hypothesis      = Lead
Experimental Attempt = Attempt
Empirical Fact  = Observation
Forensic Proof  = Evidence
Confirmed Issue = Finding

stateDiagram-v2
    [*] --> Open
    Open --> Investigating: Claimed
    Investigating --> NeedsApproval: Active Risk
    NeedsApproval --> Investigating: Approved
    NeedsApproval --> Blocked: Rejected
    Investigating --> Supported: Signals Found
    Investigating --> Exhausted: No Signal
    Investigating --> Rejected: Disproven
    Investigating --> Blocked: Scope Violation
    Supported --> [*]
    Exhausted --> [*]
    Rejected --> [*]
    Blocked --> [*]

Avoiding Rigid Pipelines via Autonomous Backtracking

Most automated security workflows run straight pipelines:

subfinder -> dnsx -> httpx -> nuclei -> generate_report

If nuclei finds nothing on step 4, the entire scan stops and cannot adapt.

AI Pentester operates on an open-ended investigative graph. The agent explores subdomains, checks endpoints, and backtracks when branches run cold:

For example: When the agent reaches staging.example.com/admin (Lead 5), it tests for unauthenticated access. Finding a strict 401 response and no exploitable misconfigurations, it does not give up on the host. It marks Lead 5 as exhausted, cleanly backtracks to the open leads queue, selects Lead 6 (/api/openapi.json), crawls the endpoints with katana, validates an unauthenticated data export with nuclei, and logs a confirmed finding backed by raw HTTP proof.

---

Concurrency & Atomic Lead Claiming

In autonomous systems, race conditions can cause multiple concurrent worker routines to probe the same target simultaneously or re-execute expensive scanner runs.

Think of LeadCoordinator like a physical dispatch board in a detective room. Before any investigator leaves headquarters, they pull an active case from the rack and stamp it with their badge. If another detective checks the board a millisecond later, that case is gone, preventing two workers from repeating expensive runs.

To ensure strict determinism, LeadCoordinator implements an atomic database reservation pattern with stale-claim recovery:

export class LeadCoordinator {
  readonly #db: Database;
  readonly #staleClaimMs: number;
  readonly #now: () => Date;

  public constructor(db: Database, options: LeadCoordinatorOptions = {}) {
    this.#db = db;
    this.#staleClaimMs = options.staleClaimMs ?? 5 * 60_000;
    this.#now = options.now ?? (() => new Date());
  }

  public async claimNext(campaignId: string): Promise<LeadClaimResult> {
    // 1. Recover any claims abandoned by timed-out or crashed workers
    await this.#recoverStaleClaims(campaignId);

    // 2. Query candidates prioritized by value and breadth
    const candidates = await this.#db
      .select()
      .from(leads)
      .where(and(eq(leads.campaignId, campaignId), eq(leads.status, "open")))
      .orderBy(desc(leads.priority), asc(leads.depth));

    // 3. Optimistic atomic update
    for (const candidate of candidates) {
      const claimedAt = this.#now();
      const [claimed] = await this.#db
        .update(leads)
        .set({ status: "investigating", claimedAt, updatedAt: claimedAt })
        .where(and(eq(leads.id, candidate.id), eq(leads.status, "open")))
        .returning();

      if (claimed) {
        return { status: "claimed", lead: claimed };
      }
    }

    // 4. Return busy if other workers are active, or empty if campaign is complete
    const activeClaims = await this.#db
      .select()
      .from(leads)
      .where(and(eq(leads.campaignId, campaignId), eq(leads.status, "investigating")));

    return activeClaims.length > 0 ? { status: "busy" } : { status: "empty" };
  }
}

The sorting heuristic orderBy(desc(leads.priority), asc(leads.depth)) ensures the agent attacks high-value surfaces (e.g., paths containing admin, api, config, .env, or swagger) before trying wide sub-tree crawling.

---

Safety by Design

In autonomous security operations, the LLM is never trusted to police its own boundaries. System prompts instructing an LLM “Do not scan outside of example.com” are vulnerable to prompt injection, hallucination, and redirection traps.

Every proposed action must survive the ScopeGuard pre-flight inspection before any scanner binary is invoked:

flowchart TD
    ACT["Proposed Action"] --> SG["ScopeGuard Engine"]

    SG --> R1{"Tool Allowed?"}
    R1 -->|No| BLOCK["Block Action"]

    R1 -->|Yes| R2{"In Domain Scope?"}
    R2 -->|No| BLOCK

    R2 -->|Yes| R3{"Public IP?"}
    R3 -->|No| BLOCK

    R3 -->|Yes| R4{"Unique Hash?"}
    R4 -->|Duplicate| BLOCK

    R4 -->|Unique| R5{"Within Budget?"}
    R5 -->|Exceeded| BLOCK

    R5 -->|Allowed| R6{"Risk Tier?"}
    R6 -->|Passive| ALLOW["Execute Probe"]
    R6 -->|Active| GATE["Human Approval"]

    style SG fill:#1f1c19,stroke:#d97706,stroke-width:2px,color:#f4ede4
    style ALLOW fill:#064e3b,stroke:#10b981,stroke-width:2px,color:#d1fae5
    style GATE fill:#78350f,stroke:#f59e0b,stroke-width:2px,color:#fef3c7
    style BLOCK fill:#450a0a,stroke:#ef4444,stroke-width:1px,color:#fecaca

1. Private IP & SSRF Defense

If a discovered subdomain resolves to an internal address (such as cloud metadata 169.254.169.254, loopback 127.0.0.1, or RFC 1918 private subnets 10.0.0.0/8, 192.168.0.0/16), ScopeGuard immediately rejects the target unless explicitly specified in allowedDomains. This protects internal corporate networks from being inadvertently scanned through DNS rebinding or redirect chains.

2. Attempt Fingerprinting & Loop Prevention

LLM loops can occur during pentesting when an agent repeatedly tries the same tool with slight variations of prompt phrasing.

Before any scanner binary executes, the system computes a hash of the lead, tool, target, and sorted parameters. If the hash already exists in the system, the proposal is dropped immediately with duplicate_attempt, forcing the agent to formulate an alternative hypothesis instead of spinning in an LLM reasoning loop:

export function generateAttemptFingerprint(action: ProposedAction): string {
  const normalizedParams = Object.keys(action.parameters ?? {})
    .sort()
    .reduce<Record<string, unknown>>((acc, key) => {
      acc[key] = action.parameters?.[key];
      return acc;
    }, {});

  const raw = `${action.leadId}:${action.toolId}:${action.target}:${JSON.stringify(normalizedParams)}`;
  return createHash("sha256").update(raw).digest("hex");
}

If an attempt with this fingerprint has already run, the ScopeGuard drops the proposal with duplicate_attempt, forcing the agent to formulate an alternative hypothesis or abandon the branch.

3. Human-in-the-Loop Approval Gate

When an action is categorized as active risk (e.g. running vulnerability probes against an API endpoint with nuclei), the orchestrator pauses execution:

const approvalPolicy = context.scopePolicy.requireHumanApprovalRisk;
const needsApproval =
  approvalPolicy === "all" ||
  (approvalPolicy === "active" && tool.risk === "active");

if (needsApproval) {
  return {
    decision: "requires_approval",
    reason: `Action requires human approval: '${tool.name}' has '${tool.risk}' risk.`,
  };
}

The operator is presented with a dedicated approval modal in the interface, displaying:

• The exact target URI and resolved IP.
• The specific scanner tool ID and CLI parameters.
• A hypothesis explaining why it believes this check is warranted.
• The expected signal it hopes to verify.

Human-in-the-loop approval gate dialog showing target, tool parameters, hypothesis explanation.

Closing the modal makes no decision, so execution remains paused until the operator clicks Approve & run scan or Reject scan.

---

Contract-Driven Tool Registry & Output Normalization

Scanners in AI Pentester are treated as plugins implementing strict target contracts.

export type TargetKind =
  | "Domain"
  | "Hostname"
  | "IPAddress"
  | "HTTPService"
  | "WebEndpoint"
  | "NetworkService";

export interface ToolDefinition {
  readonly id: string;
  readonly name: string;
  readonly description: string;
  readonly risk: "passive" | "low" | "active";
  readonly accepts: readonly TargetKind[];
  readonly produces: readonly string[];
  readonly dockerImage: string;
  readonly binary?: string;
  readonly parametersSchema?: z.ZodTypeAny;
  validateParameters?(parameters?: ToolParameters): ToolParameters;
  buildArgs(target: string, parameters?: ToolParameters): readonly string[];
}

Dynamic Tool Filtering

Before the LLM is prompted to select an action, the registry filters the available tools based on the lead’s current target type:

const applicableTools = toolRegistry.toolsThatCanInvestigate(lead.targetType);

If the lead target is an HTTPService, the LLM only sees katana and nuclei. It is physically impossible for the agent to select subfinder (which requires a Domain) or dnsx (which requires a Hostname). This constraint eliminates tool hallucinations and parameter errors.

The ProjectDiscovery Tooling Suite

AI Pentester integrates best-in-class open-source security utilities:

Tool	Risk	Accepts	Produces	Responsibility
subfinder	Passive	Domain	Hostname	Fast, passive subdomain discovery using OSINT sources
dnsx	Low	Hostname	DNSRecord, IPAddress	Multi-purpose DNS resolver and validation probe
httpx	Low	Hostname, IPAddress	HTTPService, Fingerprint	Probes HTTP/HTTPS status, titles, TLS certs, technologies
katana	Active	HTTPService	WebEndpoint	Next-generation crawling and endpoint spidering
nuclei	Active	HTTPService, WebEndpoint	VulnerabilitySignal	Curated, template-based vulnerability assessment

Transforming Raw CLI Outputs into Structured Observations

Security tools have various outputs: some uses JSONL, others use ASCII tables or unstructured stdout. OutputNormalizer changes raw scanner logs into normalized records:

export function normalizeNucleiOutput(stdout: string): readonly NormalizedObservation[] {
  const observations: NormalizedObservation[] = [];
  const lines = stdout.split("\n").filter((l) => l.trim().length > 0);

  for (const line of lines) {
    try {
      const parsed = JSON.parse(line);
      observations.push({
        type: "vulnerability_signal",
        toolId: "nuclei",
        rawPayload: line,
        normalizedData: {
          templateId: parsed["template-id"],
          name: parsed.info?.name,
          severity: parsed.info?.severity,
          matchedAt: parsed["matched-at"],
          description: parsed.info?.description,
          extractedResults: parsed["extracted-results"],
          curlCommand: parsed["curl-command"],
        },
      });
    } catch {
      // Handle non-JSON lines gracefully
    }
  }

  return observations;
}

When a vulnerability signal is verified, the orchestrator links the observation to an immutable Evidence record storing the exact HTTP request/response transcripts, curl command, and reproduction steps.

---

Interactive Sigma.js Canvas Graph

Rather than rendering SVG trees that choke when attack surfaces scale to hundreds of nodes, AI Pentester uses Sigma.js v3 with Graphology to render a WebGL/Canvas attack graph:

Domain Node         -> #f4ede4 (Aged Bone)
Hostname Node       -> #60a5fa (Muted Blue)
HTTP Service Node   -> #34d399 (Emerald)
Web Endpoint Node   -> #a78bfa (Lavender)
Finding Node        -> #ef4444 (Crimson)

Active leads currently under investigation render with an animated pulsing radar ring, giving operators instant visual orientation of where the AI is probing.

Sigma.js attack surface graph visualization showing clusters of subdomains, endpoints, and highlighted vulnerability nodes.

The Hierarchical Investigation Ledger

To complement the graph, we have an Investigation Ledger:

Compact investigation ledger displaying hierarchical tree rows, status badges, and expandable probe outcomes

Severity-First Findings Triage

When vulnerabilities are discovered, analysts do not want to dig through raw log dumps. The Findings View aggregates confirmed vulnerabilities into a structured triage table sorted by severity (Critical > High > Medium > Low > Info).

Selecting any finding opens an inline inspection drawer containing:

1. Target & Affected Asset: Full URL.
2. Reproduction Proof: Exact curl command with headers and payload.
3. Raw Network Evidence: Collapsible HTTP request and response transcripts.
4. AI-Enhanced Remediation: Generated natural-language remediation guidance explaining root cause and recommended code patches.

Severity-first findings triage console showing vulnerability details, reproduction steps, and AI-assisted remediation.

---

Dual-Audience Automated Reporting

One of the most tedious aspects of traditional penetration testing is manual report generation. Security engineers spend hours compiling screenshots and formatting Word documents.

AI Pentester automates this via dual-mode report generation:

flowchart TD
    STATE["Campaign Audit State"] --> REP["Report Generator Engine"]
    REP --> TECH["Technical Dossier (Markdown)"]
    REP --> EXEC["Executive Summary (Agent Synthesis)"]

1. The Technical Report

A Markdown document mainly for developers and security engineers. It lists every discovered hostname, open port, and web endpoint, followed by categorized vulnerabilities containing exact proof-of-concept curl commands and verified HTTP request/response payloads.

2. The CISO Executive Summary

Generated using Mastra’s LLM agent, the executive summary translates technical findings into executive business risk:

• Synthesizes risk posture across the organization.
• Quantifies findings by severity and asset criticality.
• Outlines high-priority remediation actions in plain language.
• Highlights systemic misconfigurations (e.g., recurring staging environment exposures or missing authentication headers).

Technical and Executive report views showing markdown reading presentation and export options.

---

Engineering Takeaways & Lessons Learned

1. Deterministic Guardrails Beat System Prompts

Attempting to control an autonomous AI agent through natural language instructions alone (e.g. skills, prompts) is inherently fragile. Moving boundary verification, target scope checking, private network filtering, and loop detection into a deterministic, non-LLM ScopeGuard was the single most important architectural decision in the project. The LLM is used for creative hypothesis generation, while code enforces safety rules.

2. Relational Graphs Outperform Raw Chat Histories

Long-running penetration testing campaigns easily generate tens of thousands of tokens of tool outputs. Putting this entire history into an LLM context window causes output degradation and inflates API costs. By persisting observations, hypotheses, and evidence in a normalized relational database (PostgreSQL), the agent is provided only with concise, targeted context slices for the specific lead under investigation.

3. Human-in-the-Loop Creates Real-World Trust

Fully autonomous tools that launch exploit payloads without human verification are rarely permitted in production corporate environments. By building an approval gate for active scans with target details, tool parameters, and hypothesis reasoning, AI Pentester allows for speed and compliance.