availability

Package: Invicti AppSec Core (on-demand)

Configure target crawler behavior

Control how the crawler navigates your target so scans stay relevant and stable. This document explains the Crawling options section on the target's Scan configuration tab, including scope controls, request constraints, and custom request metadata.

Open the page before you make these changes:

1. From the left-side menu, select Inventory > Targets.
2. In the targets list, select the target you want to update.
3. On the target details page, select Scan configuration.
4. Scroll down to locate the Crawling options section.

Why this matters

Default crawling behavior isn't always safe or efficient for every target. If scope is too broad, scans waste time on irrelevant paths. If scope is too narrow, you miss reachable attack surface. Configuring crawler behavior helps you reduce noise, protect fragile routes, and improve scan accuracy.

Set crawl identity and path sensitivity

Use these controls to define how the crawler presents itself and interprets URLs:

User agent

Each HTTP request sent by the crawler contains a "User Agent" header identifying the browser and operating system. Web servers may present different content based on this header.

Available options include:

• Google Chrome - Chrome browser identifiers
• Internet Explorer 11 - Internet Explorer browser identifiers
• Firefox - Firefox browser identifiers
• Opera - Opera browser identifiers
• Safari, iPhone with iOS 6 - Safari on iPhone identifiers
• Webkit on Android 4.0.3 - Android mobile device identifiers
• Custom - Enter your own custom user agent string in the field that appears

Use the default (Google Chrome) unless your target requires a specific browser or device identity for proper content delivery.

Case sensitive paths

By default, Invicti AppSec automatically detects whether your target's web server treats URLs as case-sensitive. Use the toggle to override this behavior:

• Set to No (default) to let the platform automatically detect case sensitivity.
• Set to Yes when you know the target is case-sensitive and want to force case-sensitive crawling (for example, when /Admin and /admin are different paths).

Override auto-detection only when you know the target's behavior differs from what the platform detects.

Restrict crawler scope to what you actually want tested

Use these controls to keep scan scope intentional:

Limit crawling to address and sub-directories only

Control whether the crawler stays within your target's base path or expands to the entire domain:

• Set Yes (default for new targets) to keep crawling within the path specified in your target URL
• Set No to allow the crawler to expand beyond the target URL path to the entire domain

Path behavior examples:

• Target URL http://www.example.com/ with Yes - crawls the entire domain
• Target URL http://www.example.com/app/ with Yes - crawls only /app/ and subdirectories
• Target URL http://www.example.com/app (no trailing slash) with Yes - crawls only / (treats /app as a file, not a directory)

Excluded paths

Specify URL patterns to exclude from crawling and scanning. This is useful when your application is large or when scanning certain paths triggers unwanted actions (like data submission).

Excluded paths use regular expressions with PCRE syntax. Enter patterns in the Excluded paths field. Multiple patterns can be added for each target. Once a path is excluded, all subdirectories below it are also excluded.

Excluded paths formatting:

• Start with a forward slash: /path
• Format example: /dir1/dir2(/.*)?$ excludes /dir1/dir2 and all content beneath it
• After you configure a path as excluded, the entire directory tree below it is also excluded because the crawler can't know what exists in unvisited directories

Common exclusion patterns:

Description	Regular expression	Matches (excludes)	Doesn't match
Wildcard	/dir.*/otherdir	/dir/otherdir, /dir1/otherdir	/dir, /dir/dir1
Single character wildcard	/dir.?/otherdir	/dir/otherdir, /dir1/otherdir	/dir, /dir/dir123
Digit wildcard	/dir[\d]+/otherdir	/dir1/otherdir, /dir99/otherdir	/dir/otherdir, /dir1a/otherdir
Exclude nested paths	(/.+){2,}	/dir/dir1, /dir/dir1/subdir	/dir, /file.html
Exclude specific directory	/dir(/.*)?$	/dir, /dir/file.html	/dir1, /dirname

tip

Test your regular expression patterns in a tool like Regex101 before applying them.

Custom URL rewrite rules

Add regular expression-based rewrite rules when your target requires route normalization before crawling. This is rarely needed - use only when standard crawling doesn't reach all routes.

Example rewrite patterns:

• /categories/(\d+)/products(\d+) - Rewrites URL patterns with numeric IDs
• /api/v\d+/ - Normalizes versioned API paths

Excluded HTTP methods

Choose which HTTP methods to ignore during scanning. More than one method can be excluded.

Available methods to exclude:

• GET
• POST
• PUT
• DELETE
• PATCH
• HEAD
• OPTIONS
• CONNECT
• TRACE

Exclude methods that should never be used during scan traffic for this target (for example, exclude DELETE if your target doesn't allow deletes during scans).

Validate exclusion and rewrite patterns

Overly broad patterns can hide vulnerable endpoints from the scan. Test pattern changes carefully before relying on them in production.

Prevent disruptive crawler behavior during authenticated sessions

Use these toggles to avoid known session and form side effects:

Restrict testing login forms

Set Yes to prevent the crawler from testing login forms. This is useful when:

• Automated login form submissions trigger account lockouts or security monitoring alerts
• Your application has rate-limiting on failed login attempts
• You want to avoid triggering multi-factor authentication flows during scanning

Restrict navigation in new tabs

Set Yes when your application invalidates session state or authentication after a new browser tab is opened. Some applications limit authenticated sessions to a single tab for security or technical reasons. When enabled, the scanner restricts crawling to a single tab to preserve session continuity.

Block requests to advertising services

Set Yes (enabled by default) to prevent the crawler from loading external advertising networks and third-party services. This reduces:

• Scan time by skipping unnecessary external requests
• Traffic noise from unrelated domains
• The risk of scanning external ad platforms by mistake

Add request metadata required by your application

Custom headers

Custom headers are additional HTTP headers included in every request the scanner sends to your target during a scan. Because they are set at the target level, they apply to all scans on that target.

Common use cases include:

• Authentication - pass tokens or session identifiers the scanner needs to access protected pages (for example, Authorization: Bearer eyJhbGci...)
• Bypassing WAF or CDN blocks - mark scanner traffic as authorized to prevent it from being blocked (for example, X-Scanner-Allowed: true)
• Environment routing - direct requests to a specific environment or tenant (for example, X-Tenant-ID: staging)
• Required application headers - meet any header requirements your application enforces (for example, X-API-Version: 2)

In the Custom headers section, enter key and value pairs for each header the scanner needs to send.

Custom cookies

Custom cookies are sent with each request the scanner makes to your target, allowing the scanner to crawl and test your application correctly when cookie-based authentication or session management is required.

In the Custom cookies section, enter cookie name and value pairs for each cookie the scanner needs to send.

Save changes with Save target configuration.

Troubleshooting

The crawler skips pages that should be scanned

Review Excluded paths, URL rewrite rules, and the Limit crawling to address and sub-directories only toggle. Any of these can remove valid paths from crawl scope.

Authenticated routes fail during scanning

Check whether required headers or cookies are missing in Custom headers or Custom cookies. Also confirm Restrict navigation in new tabs matches your application's session behavior.

Excluded paths pattern accidentally blocks legitimate routes

Regular expression patterns can be broad and hide valid paths from the scan. Test your regex pattern carefully using a tool like Regex101 with real examples from your target's URL structure before applying it. Start with a narrow pattern and expand only if needed. If routes are still missing after excluding a pattern, review the pattern logic to ensure it only matches the paths you intend to exclude.

---

Need help?

Invicti Support team is ready to provide you with technical help. Go to Help Center