Site Crawler FAQs
Common Questions About Using the Site Crawler
What types of websites work best with the Site Crawler?
The Site Crawler is best for small to medium-sized websites that have clear navigation and consistent structure.
Can I use the Site Crawler on social media?
No. Crawls are not supported on social media platforms.
What types of URLs will be captured?
The crawler will collect URLs included in a site’s page. External links, videos, and email domain links will not be included. Any duplicate links will automatically be removed.
What if I have two captures that are identical, but the URLs are different?
Occasionally, a website will have two URLs that point to the same page, resulting in two identical captures. In its current state, our crawler removes all duplicate URLs, but we cannot detect if two different URLs point to an identical page. If you are seeing the same URLs being captured multiple times, please reach out to support@page-vault.com
Where do my captures go?
You can choose a folder when setting up your crawl. All captured pages will be saved there for easy access and organization.
How long does a crawl take?
Crawl times vary depending on the number of pages and the complexity of the website. Smaller crawls may finish in minutes, while larger ones may take longer.
Do I need to stay logged in during the crawl?
No. Once a crawl begins, it runs in the background. You can safely close Capture Mode or log out of the portal.
Is the Site Crawler available to all users?
No. The Site Crawler is a premium feature. Contact support@page-vault.com to enable it on your account or learn more about pricing.