How To Crawl Your Site Like a Search Engine

But what is a site crawler, and why does it matter? As you probably already know, search engines crawl the web and index the results. If you can test your site’s “crawlability,” and see any potential pitfalls that might block or inhibit search engine bots, you can work to fix them and avoid loss of time that can risk in poor search results and delays.

It can take up to six months for changes made to your website to start showing significant search gains. Imagine if you were unaware that an important part of your site was not being crawled for six months? Here is where a site crawler can save you.

And, up to a certain number of URLs, the Screaming Frog tool is free.

Now it’s time to run a crawl of your site. For this part of the tutorial, just make sure that you have the primary domain of your website handy because we will use it to initiate our crawl.

1. Open SEO Spider on your computer
2. Fill in your URL
3. Click Start

Check out these other resources from the support center and blog:

• How to set your homepage meta title and description
• SEO for beginners
• Adding nofollow links to WordPress