Why You Need to Regularly Crawl Your Website's XML Sitemap

Why You Need to Regularly Crawl Your Website's XML Sitemap

Analyst, Search & Social Strategy
August 31, 2017

Spiders, crawlers, frogs, oh my! When it comes to your site’s health, the results can sometimes spook you.

You may know that you should regularly crawl a specific site domain to check for any unwanted anomalies like duplication and 300 or 400-level status codes—but are you paying attention to your website’s XML sitemap?

The XML sitemap should be combed through periodically to ensure search engines are receiving all of the correct signals. If not, you could be unknowingly hurting your rankings and negatively impacting user experience through missed response error codes or inaccurate noindex tags.

With the help of tools like Screaming Frog and Deep Crawl, it’s time to make these crawls a regular habit. That way, you won’t be frightened by what’s hiding in your sitemap.

 

What exactly is an XML Sitemap?

An XML sitemap provides search engines with a list of all pages on your site, even those that might not be found through their traditional crawling means.

The sitemap allows crawlers to understand the content better, which leads to a more comprehensive crawl. From there, thorough sitemap crawls make it easy to quickly maneuver through all pages seen by search engines to determine if there are any outstanding issues.

Still aren’t convinced a sitemap crawl is worth your time? These 3 reasons should change your mind.

 

1) It’s easy to clean up pesky response codes

A sitemap crawl helps you to find and organize each one of your pages’ response codes. While there are a plethora of different HTTP response codes, there are a few common ones you should steer clear of.

Insider tip: Search engines are more likely to trust sitemaps that are free of error codes. Through an XML sitemap crawl, you might find pages that are returning codes you don’t want—and weren’t even aware of.

From there, your account analyst can modify the pages’ codes you need to change.

 

 

2) Canonicals are king

Sitemap crawls also give you the ability to see if and where specific URLs are canonicaling to. A canonical URL tag tells search engines that multiple pages should be seen as one, without actually redirecting the user to the new page.

 

 

 

While they are helpful for both web developers and search engines, you can run into problems if they are canonicaling to an incorrect or completely different URL.

If this is the case, search engines won’t know which URL is preferred, and they will end up ignoring them all. Sitemap crawls can pinpoint which URLs have canonicals and where they are pointing to without requiring someone to spot check each page one by one.

 

3) Noindex, no problem...?

Noindex tags let search engines know that you do not want specific pages to be indexed in the SERP. You want to use the noindex tag sparingly. They could apply to employee-only pages or thank you pages after a customer purchases a product or service.

Unfortunately, there are times when you’re completely unaware that a page is being noindexed. If an important page has a noindex tag on it, it won’t get traffic—and that’s a big deal.

Wondering how to avoid this problem? The handy dandy sitemap crawl comes to the rescue.

While you’re busy poking around in your newly completed crawl, you can easily filter and scan all pages appearing to not be indexed to make sure all of them belong there, and take note of those that need to be modified. Problem solved.

 

--

There you have it! Three (out of many!) reasons why you should be crawling your website’s sitemap. Any questions? Tweet us @Perfect_Search.  

Analyst, Search & Social Strategy
Courtney Culligan is recent DePaul University graduate, but she still loves cheering on her home state’s baseball team. (The Minnesota twins, obviously.) Her dream birthday present might include a ton of Sour Skittles, cheese curds, and a trip to Greece.

What our clients say

"Perfect Search Media made it very easy for us to fully customize our website with high res images and the latest widgets…"
Scott Baker
Precious Metals Buyer

Tweets

Perfect Search Media
2h

Being closed on means missing out on prime shopping hours. Yet some states mandate closings. Is that…

Perfect Search Media
5h

Craving real-world experience in digital marketing? We've got it. Apply to be a winter intern in digital marketing,…

Perfect Search Media
7h

Pinterest is a leading social network platform for e-commerce sites. Learn how to add widgets to your e-…

Perfect Search Media
10h

Amazon Echo vs. Google Home. Which sale should you choose? weighs in.

Perfect Search Media
1d

Just in time for , lets users shop at Costco without a membership. Bulk bags of stuffin…

Perfect Search Media
1d

We're thankful for Ben Temple, a new Business Development Associate at Perfect Search. Get to know him (and who his…

Perfect Search Media
1d

Bring more people to your brick-and-mortar locations by optimizing your online presence. Learn four things you can…

Perfect Search Media
1d

Have you caught up on how AI is changing ? Get familiar with evolving ad creation, rotation, and bidding th…

Perfect Search Media
1d

Looking for a side hustle? These flexible work ideas will add to your holiday budget.

Perfect Search Media
2d

Getting ready for ? You might notice ads that use some of these social psychology principles.

Perfect Search Media
2d

RT : Facebook briefly hid its 'delete' option on posts, and users panicked:

Perfect Search Media
5d

It's time to think local. Follow these 4 local practices and your business will be in good shape.

Perfect Search Media
5d

Our Director of SEO & Content Justin spoke to students at on careers in this week. 📚💻🧠

Perfect Search Media
6d

82% of users say they'd leave a site that isn't secure. Great post by on why it matters to get a secure site--…

Perfect Search Media
6d

Everyone's trusty way to get from point A to point B just got a makeover. Are you a fan?

Perfect Search Media
1w

We <3 our interns. Get to know our Fall 2017 group in the latest round of internviews. (You know, interviews with i…

Perfect Search Media
1w

Happy birthday to Kayla, our Manager of Copywriting & Content Strategy! Thanks to Spinning J for the beaut pies and…

Perfect Search Media
1w
Perfect Search Media
1w

According to , only 7% of people trust Google's featured snippets. Are you in the 7%? If not, why not?

Perfect Search Media
1w

RT : It’s the most wonderful time of the year! 🎅 🎄 How are you optimizing your ad campaigns this holiday season?

Perfect Search Media
1w

Wishing a sweet birthday to Junior Analyst ! We’re big fans of you—and of .

Perfect Search Media
1w

We just wrote about how to write for the skimmer! Steal techniques from journalists. Check…

Perfect Search Media
1w

Thank you!

Perfect Search Media
1w

What techniques do you think are most effective for linkbuilding after content creation?

Perfect Search Media
1w

An important clarification. It is likely the later.

Perfect Search Media
1w

Yes. We like to be able to share them with clients as well.

Perfect Search Media
1w

Exactly. It's important to cater to several different styles of learning/information gathering to reach the most people.

Perfect Search Media
1w

Whoops I meant to say question seven!

Perfect Search Media
1w

Q6: Some ROI metrics we look out for are increases in domain authority, traffic, and social media following

Perfect Search Media
1w

A6: We've found success with infographics. They're a unique way of sharing information and showing off our design s…

Find us on Facebook