availability

Package: Invicti AppSec Core (on-demand)

Configure target crawler behavior

Control how the crawler navigates your target so scans stay relevant and stable. This document explains the Crawling Options on the target's Scan configuration tab, including scope controls, request constraints, and custom request metadata.

To open the page: select Inventory > Targets from the left-side menu, click the target name, then select Scan configuration.

Why this matters

Default crawling behavior isn't always safe or efficient for every target. If scope is too broad, scans waste time on irrelevant paths. If scope is too narrow, you miss reachable attack surface. Configuring crawler behavior helps you reduce noise, protect fragile routes, and improve scan accuracy.

Set crawl identity and path sensitivity

Use these controls to define how the crawler presents itself and interprets URLs:

• User agent - choose the request identity that best fits your traffic handling and allowlist rules.
• Case sensitive paths - set Yes when /Admin and /admin must be treated as different paths; set No when your routing is case-insensitive.

Restrict crawler scope to what you actually want tested

Use these controls to keep scan scope intentional:

• Limit crawling to address and sub-directories only - set Yes to stop crawler expansion outside the target's base path.
• Excluded paths - add path patterns to skip areas you don't want scanned.
• Custom URL rewrite rules - add regex-based rewrite rules when route normalization is required before crawling.
• Excluded HTTP methods - omit methods that should never be used during scan traffic for this target.

Validate exclusion and rewrite patterns

Overly broad patterns can hide vulnerable endpoints from the scan. Test pattern changes carefully before relying on them in production.

Prevent disruptive crawler behavior during authenticated sessions

Use these toggles to avoid known session and form side effects:

• Restrict testing login forms - set Yes if automated login form testing can trigger account locks or monitoring alerts.
• Restrict navigation in new tabs - set Yes when your app invalidates session state after new-tab navigation.
• Block requests to advertising services - set Yes to reduce third-party traffic noise and avoid scanning unrelated ad domains.

Add request metadata required by your application

Use optional key-value rows to include request-level metadata during crawling:

• Custom headers - add headers (for example, tenant IDs or feature flags) when routes require them.
• Custom cookies - add cookie values when route access depends on specific session or context cookies.

Save changes with Save target configuration.

Troubleshooting

The crawler skips pages I expected to be scanned

Review Excluded paths, URL rewrite regex rules, and the Limit crawling to address and sub-directories only toggle. Any of these can remove valid paths from crawl scope.

Authenticated routes fail during scanning

Check whether required headers or cookies are missing in Custom headers or Custom cookies. Also confirm Restrict navigation in new tabs matches your application's session behavior.

---

Need help?

Invicti Support team is ready to provide you with technical help. Go to Help Center