XML Sitemap - Get Your Content Found

/in , /by

How Do I Make Sure My Website is Being Found?

This question has a multi-faceted answer. There truly is a laundry list of things you need to do to ensure your website and all its pages are being found by search engines and eventually web searchers. This article will focus on XML sitemaps and how we can manipulate them to submit only the most important pages we want search engines to find. But before we dig in, you might be asking…

What is an XML Sitemap?

“An XML sitemap is a file designed for search engine consumption. Search engine spiders, aka crawlers, search bots, web robots, etc. crawl a website and all of its available links to ensure all your important web pages are found and represented.” In other words, it’s a file containing all the pages of your website, listed in such a way that each one is easily found by any crawler that comes to your website at any given time.

Yoast SEO XML Sitemap Example

Most website XML sitemaps are accessible by adding sitemap.xml or sitemap_index.xml to the end of your root domain, ex: www.MyWebsite.com/sitemap.xml. If you can’t find yours, check with your webmaster to be sure your website has one and that you are submitting it. When reviewing this file you will likely find a fairly ordered list of your website page URLs. As you take a closer look at the list, a good question to ask yourself is, “Do all these pages really need to be submitted and indexed?”

Why should I adjust my XML Sitemap?

Some would argue that submitting each and every page of your website is the best way to ensure ranking status, more pages equals more chances of being found and seen right? Not necessarily. What we really want to do to drown out the competition, is evaluate the quality of the page or pages we are submitting versus the quantity. One really well-written page of content that targets an entire concept by answering questions, providing resources, how-to’s, relevant imagery and quality content will be ‘rewarded’ with higher ranking status than, let’s say… 5 pages of mediocre content even if targeting the same concept, these 5 pages will inevitably compete against each other in search.

Google wants readers to focus on what they call “beneficial purpose” of content and consider  a “crawl budget”. Though it may seem Google has an infinite amount of resources, they don’t and they admit to it. Selecting and indexing only your most popular and content rich pages, we can better ensure Google finds the pages we want them to find versus the pages we don’t want them to find.

“So many sites create content solely for Google, without the user in mind. They just want their content with ‘keyword keyword keyword’ to rank well, and hope the person converts when they get there, whether by clicking an ad or affiliate link or perhaps going deeper into the site. But Google wants their raters to think about whether a piece of content has a beneficial purpose or not, and this is something that any site owner, content creator or SEO should think about when writing new content or auditing current content on a site.”

– SearchEngineLand – Google updates its search quality rating guidelines

How do I determine what I should and shouldn’t index?

There are two main places to start. The first is from a content/SEO point of view, while the second is from a technical view. A full review of all of your website’s URLs produced by a crawl tool like ScreamingFrog is a great start to evaluate what pages are being found by the search engines and what pages you actually want them to see.

  • CONTENT EVALUATION

    Depending on the main purpose of your website, there are some pages that you may not want to submit in your XML Sitemap as they are merely a part of the site for esthetic reasons (think image galleries) or for value added reasons (think guest comments or print and favorites pages.) There is no real SEO value to a gallery of images page unless you’ve spent time captioning them and making them and the other facets of the page SEO friendly, including alt tags, etc. However, this typically isn’t the case and those pages can stop the search bot from getting to a page that really does matter. Same goes for a print page for your blog article that is auto-generated; this is for user ease and there is no SEO value, so don’t submit it!

  • TECHNICAL EVALUATION

    In terms of the technical evaluation of pages submitted there are typically many pages that are auto-generated as part of website setups. For example, as part of a WordPress set up, the “Posts” section, more commonly known as the Blog, creates /tag/ and /author/ pages as you add these small features to your build out. /tag/ pages typically have no SEO value and are a waste to submit for index, especially if you have many of them throughout your blog. Adjusting these to a “noindex” setting is typically always advised, especially if your blog is not the main focus of your website. However, if you are primarily a blogger, these pages most certainly are important to you. You want to build them out, make them SEO rich, as well as ensure they are indexed.

How do I update my XML sitemap?

Whether you still have to create XML sitemaps manually or you use a plugin tool like Yoast that automatically creates your XML, it can be rather easy to omit the pages you don’t want indexed from your sitemap.

In Yoast, it is simply the click of a button under each of the various taxonomies and page types in the settings section.As for a manual build, simply do not include the URLs you don’t want in the file you are creating.  Be sure to add a ‘noindex’ tag to the pages  you are excluding to ensure they are not found. Just because you didn’t include the page in the XML doesn’t mean a crawl bot can’t find them!

Yoast SEO XML Sitemap Settings
  • What is a no-index tag?

    A noindex tag is a meta tag placed in the code of a page that tells search engines to move on and NOT crawl this particular page when it finds it on it’s crawling mission.

    These tags are extremely powerful and can take down your website with the click of a mouse so PLEASE be careful if you are executing these types of changes without webmaster supervision!

    It is really easy to determine if you have a noindex tag on a page:

    1. Right-click the webpage you want to check for a noindex tag
    2. Select “Inspect Source/Code”
    3. A new window will open with tons of scary lines of code. Simply hit Ctrl+F and type ‘noindex’
      1. If you get 0/0 the page has no tag and search engines can and will likely find it!
      2. If you get a highlighted ‘noindex’, this page WILL NOT be crawled and indexed by search engines, making it all the more difficult for web  searchers to find it

    Search engines are smart. If you submit a sitemap with a page in it marked as ‘noindex’ they will call you on it and report the error to Google Search Console, the one source for monitoring the index status of your website. So be sure you have your website set up and verified with GSC. This will greatly assist you as you work through this exercise with your website and allow you to monitor your website daily.

Submit Your Sitemap for Indexing

Once you’ve made all the adjustments you wish to your XML sitemap, (and setup and verified your GSC account) it is time to submit that bad boy for indexing! Navigate to the Sitemaps section of GSC and hit ‘Submit’! Your sitemap will likely go into ‘Pending’ status but in a short time (sometimes hours, sometimes days) you’ll start to see what Google, and therefore the world, is seeing of your website.

Remember too that Google isn’t the only player out there. Verifying your website and submitting your sitemap to Bing Webmaster Tools is also advised. It’s easy enough, and you should take every advantage to reach your target audience where they browse or shop.

Have a million other things going on, let Blizzard help! Contact our team today to see how we can help ensure your website is being found on the web.