Before starting, you can fine-tune how the Site Crawler behaves. Here’s what each setting does:
Crawl Type
-
Crawl and Capture: Automatically finds and captures each page.
-
Crawl Only: Finds pages but does not capture them—useful for reviewing results first.
Layer Depth
-
Controls how far the crawler goes into the website from the starting page.
-
Example: A 1-layer crawl will capture just the starting page and its links. A 2-3 layer crawl will dig deeper into subpages.
-
Max depth is 5 layers.
URL Limit
-
Set how many pages you want to crawl.
-
Useful for smaller sites or if you want to limit the crawl to a specific size.
- Max limit is 5,000 URLs
Cookies
-
You may need cookies to prevent pop-ups or mimic a sign-in.
-
In Capture Mode, cookies are automatically saved.
-
In the Portal, you can paste in cookies manually (if needed).
- View our help article here for instructions on copying cookies
Folder
-
Choose a destination folder for your captured files.
-
Helps keep your projects organized.