How To Start A Crawl

Option 1: Page Vault Browser

1. Open the Page Vault Browser.
2. Navigate to the website you want to crawl.
3. Close any pop-ups and sign in if needed. Your session and cookies will carry over to the crawl.
4. In the sidebar, click Crawl website.
5. In the Crawl panel that opens, choose your crawl type:
   1. Crawl only — Generates a list of URLs found on the domain without saving any captures.
   2. Crawl & capture — Finds every URL and captures each one as a PDF (default).
6. Set Layers to crawl. Defaults to All. Reduce this to limit how deep the crawler follows links from the starting page.
7. Set the URL maximum. Defaults to 2,500. Lower or increase this (up to 20,000) if you want to cap the size of the job.
8. Choose the folder where captures should be saved (required).
9. (Optional) Add a Case matter ID if your organization uses CMIDs.
10. (Optional) Check Advanced crawl options to narrow the scope further:
    1. Required URL text — Only include URLs that contain this string (for example, /collections/womens-all).
    2. Excluded URL text — Skip any URLs that contain these strings. Enter one per line.
11. Click Crawl site to start the job.
12. You can close the Page Vault Browser — your crawl will continue in the background.
13. View the results in the Site Crawler section of your Portal when the job completes.

Option 2: Portal

1. Log in to your Page Vault Portal

2. Click Site Crawler at the top.

3. Select New Crawl.

4. Enter the website URL.

5. Add cookies manually (only if needed).

6. Choose your crawl type:

   • Crawl and Capture – finds and saves each page.

   • Crawl Only – finds pages, but doesn’t save them yet.

7. Set how deep to crawl (layers) and how many pages to include.

8. Choose a folder to save your captures (optional).

9. Click Start Crawl.