Pages not being crawled? Here’s the fix

It feels like you’ve tried every possible step, but traffic to your site still isn’t increasing. If you are trying to increase traffic from search engines, indexing is absolutely essential. 

Website indexation, better known as indexing, refers to the process of search engines crawling a web page to figure out what it’s about. If the search engine can understand what the web page is about and it meets quality standards the content is added to the search engine’s index. 

Why search engine indexing is so important

While crawling a web page, the search engine is taking note of keywords, meta data and other factors that indicate what a web page is about and what it should rank for. This information is indexed and then factors into what shows up on a search engine results page.

Your site needs to be indexed by search engines to be displayed in relevant search results. But that’s not all! For the biggest SEO benefit you want your site to be continuously recrawled. It’s better for the search rankings of existing pages and it helps ensure new pages get indexed.

Now that we’ve laid out why indexing is crucial, let’s concentrate on how to repair errors that prevent indexing and improve the chances that a page will appear in the search results.

How to fix crawl problems on web pages

A page status of “crawled – presently not indexed” indicates that Google has seen the page but has chosen not to include it in its search index. When this happens the URL will be hidden from search engine results. The kicker is Google doesn’t explain why a page received this status. It’s given to pages that don’t meet Google’s quality standards, and there are a few common reasons why a page doesn’t meet the minimum requirements.

Weak internal link structure that needs to be enhanced 

If your website has a weak internal link structure or the page in question does not have any internal links, Google may feel that it’s an orphaned page and indexing is unnecessary. Orphaned pages do not have any other pages that link to them.

You can use the evisio platform to figure out if this is part of the problem. You can also manually check the web page in question to see if there are any links. Do not overlook the importance of internal linking on your site! If there are no links add one to a relevant page about a related topic. This should fix an orphan page problem while improving the internal linking structure. 

Thin content or low-quality content that needs to be improved

When a web page has a low word count Google may consider it “thin content” that doesn’t include enough information. The page will get crawled, but it will not appear in search results. The same thing can happen if Google deems the content to be low-quality.

In most cases, the issue of thin content can be addressed by adding quality content to the page in question and cleaning up the existing content that’s on the page. You should make the page as long as necessary to cover the issue entirely without making it any longer than necessary.

Mismatched search query for the web page

A web page may be “crawled – currently not indexed” due to a mismatch between the search query you’re targeting and the page content. This indicates a disparity between your content and what Google thinks is appropriate for the topic. 

To ensure your content matches user expectations, research what appears on the first page of Google for the search term you’re targeting. Next, rewrite the existing content or add new content that’s in line with what’s being ranked for the search term. 

Structured data incompatibility

Structured data assists search engines in comprehending the content of a page. You can use structured data in a standardized format to classify the web page and provide Google with more information about the page. This is particularly significant for e-commerce websites that sell tangible things.

You’ll need to be able to edit your HTML to add or alter structured  data if it doesn’t exist or isn’t accurate. You can input structured data snippets to specify the type of content, a title for the content, a description and other details. The more details you add, the better. 

Key takeaways

Any website, regardless of its size or purpose, can end up receiving a “crawled – presently not indexed” status. It’s important to check all web pages to ensure they are being indexed properly, and if not, action needs to be taken to correct whatever issue is holding back the indexation. Typically, it’s an issue that involves links or weak content that can be easily corrected. 

But first you have to know the problem exists. Catching crawl errors is what evisio was designed to do. Instead of using manual labor, you can let evisio do the monitoring for you so you have more time to improve your content and boost search rankings.

If you’re looking for SEO project management software to better manage your workflow, clients, and business – evisio.co is your solution. Try evisio.co for free here!