Why You Need to Regularly Crawl Your Website's XML Sitemap
877-655-8227

Blog

Why You Need to Regularly Crawl Your Website's XML Sitemap

Analyst, Search & Social Strategy
August 31, 2017

Spiders, crawlers, frogs, oh my! When it comes to your site’s health, the results can sometimes spook you.

You may know that you should regularly crawl a specific site domain to check for any unwanted anomalies like duplication and 300 or 400-level status codes—but are you paying attention to your website’s XML sitemap?

The XML sitemap should be combed through periodically to ensure search engines are receiving all of the correct signals. If not, you could be unknowingly hurting your rankings and negatively impacting user experience through missed response error codes or inaccurate noindex tags.

With the help of tools like Screaming Frog and Deep Crawl, it’s time to make these crawls a regular habit. That way, you won’t be frightened by what’s hiding in your sitemap.

 

What exactly is an XML Sitemap?

An XML sitemap provides search engines with a list of all pages on your site, even those that might not be found through their traditional crawling means.

The sitemap allows crawlers to understand the content better, which leads to a more comprehensive crawl. From there, thorough sitemap crawls make it easy to quickly maneuver through all pages seen by search engines to determine if there are any outstanding issues.

Still aren’t convinced a sitemap crawl is worth your time? These 3 reasons should change your mind.

 

1) It’s easy to clean up pesky response codes

A sitemap crawl helps you to find and organize each one of your pages’ response codes. While there are a plethora of different HTTP response codes, there are a few common ones you should steer clear of.

Insider tip: Search engines are more likely to trust sitemaps that are free of error codes. Through an XML sitemap crawl, you might find pages that are returning codes you don’t want—and weren’t even aware of.

From there, your account analyst can modify the pages’ codes you need to change.

 

 

2) Canonicals are king

Sitemap crawls also give you the ability to see if and where specific URLs are canonicaling to. A canonical URL tag tells search engines that multiple pages should be seen as one, without actually redirecting the user to the new page.

 

 

 

While they are helpful for both web developers and search engines, you can run into problems if they are canonicaling to an incorrect or completely different URL.

If this is the case, search engines won’t know which URL is preferred, and they will end up ignoring them all. Sitemap crawls can pinpoint which URLs have canonicals and where they are pointing to without requiring someone to spot check each page one by one.

 

3) Noindex, no problem...?

Noindex tags let search engines know that you do not want specific pages to be indexed in the SERP. You want to use the noindex tag sparingly. They could apply to employee-only pages or thank you pages after a customer purchases a product or service.

Unfortunately, there are times when you’re completely unaware that a page is being noindexed. If an important page has a noindex tag on it, it won’t get traffic—and that’s a big deal.

Wondering how to avoid this problem? The handy dandy sitemap crawl comes to the rescue.

While you’re busy poking around in your newly completed crawl, you can easily filter and scan all pages appearing to not be indexed to make sure all of them belong there, and take note of those that need to be modified. Problem solved.

 

--

There you have it! Three (out of many!) reasons why you should be crawling your website’s sitemap. Any questions? Tweet us @Perfect_Search.  

Analyst, Search & Social Strategy
Courtney Culligan is recent DePaul University graduate, but she still loves cheering on her home state’s baseball team. (The Minnesota twins, obviously.) Her dream birthday present might include a ton of Sour Skittles, cheese curds, and a trip to Greece.
"Perfect Search was our first investment in search engine optimization...we found that their results-driven approach and testing mentality would yield the largest ROI."
Michelle Houser

Tweets

Perfect Search Media
6h

RT : Amazon is working on smart glasses to house Alexa AI, says FT

Perfect Search Media
10h

71% of customers feel disengaged. Make sure your business is engaging--and retaining!--your clients. Read this.

Perfect Search Media
13h

Do you mix up Panda & Penguin? Get all the algorithm updates straight with help from this guide.

Perfect Search Media
1d

Unchecked, self-reported user data + an algorithm = significant issues for . Thoughts?

Perfect Search Media
1d

RT : The Anatomy of a $97 Million Page: A CRO Case Study by Jasper Kuria

Perfect Search Media
2d

Don't forget about your old . It's time to brush it off and breathe some new life into it. Here's how.

Perfect Search Media
2d

Celebrating with the best 🍔 from .

Perfect Search Media
2d

The battle of the ridesharing apps continues. Google might invest in .

Perfect Search Media
5d

Congrats to Eric for constructing the most functional paper airplane and Justin for crafting the ~flyest~ paper air…

Perfect Search Media
5d

Is your business prepping for Q4? You might need some help allocating your budget. Read this.

Perfect Search Media
5d

Pumped for the new wireless charging? Forgetting your charger might be a thing of the past.

Perfect Search Media
6d

We had our inaugural Perfect Search paper airplane contest today. Tune in tomorrow to find out who won...

Perfect Search Media
6d

RT : The Beginner’s Guide to Structured Data for SEO: How to Implement [Part 2] By…

Perfect Search Media
6d

On the hunt for a new job? Make sure your online presence is in tip-top shape. Check out our post for some advice!

Perfect Search Media
1w

Links are a hugely important ranking signal for . Strong backlink profile = strong showing in the SERP.

Perfect Search Media
1w

Missed yesterday's event? Get the scoop on the iPhone X, face ID & more with this rundown from .

Perfect Search Media
1w

New season, newfound obsession with Pumpkin Spice Lattes, new PSM team spotlight. Get to know Courtney!

Perfect Search Media
1w

RT : Wireless charging will be available on the iPhone 8

Perfect Search Media
1w

Always wanted a .pizza domain? Go for it! Just make sure you read this article first.

Perfect Search Media
1w

We had a blast volunteering at with !

Perfect Search Media
1w

Fascinating article on how Google plays a role in addiction treatment--and why that can be flawed.

Perfect Search Media
1w

RT : E-Commerce Benchmark KPI Study 2017: 15 Essential Takeaways By:

Perfect Search Media
1w

Old and old customers are marketing gold. Find out how you can make the most of them here.

Perfect Search Media
1w

Bigger isn't always better. confirms that large sites don't get higher SERP rankings.

Perfect Search Media
1w

to Wisconsin sunsets, summer, team float trips, and Quinn's super reflective hat.

Perfect Search Media
1w

RT : How to Diagnose Pages that Rank in One Geography But Not Another - Whiteboard Friday By…

Perfect Search Media
1w

Did you know has its own platform? Read this article to find out why they might have a leg up.

Perfect Search Media
2w

Yes, another article about how might have inaccurate measurements. (But the U.S. Census Bureau might too.)

Perfect Search Media
2w

RT : 20 of Google’s limits you may not know exist by

Perfect Search Media
2w

Our CEO was featured in . Check it out & learn how Perfect Search was founded!

Find us on Facebook

RT : Amazon is working on smart glasses to house Alexa AI, says FT