This article covers the initial steps for understanding how to get your website indexed by Google. I’ll cover a quick overview of what Google looks for when indexing websites on its search results pages and a step-by-step tutorial for getting your site listed on Google search.
This article is intended for those with little understanding of Search Engine Optimization (SEO) but can serve as a checklist of sorts for the experienced as well. If you need help, get in touch with me to discuss an advanced analysis of your search rankings.
If you just redesigned your site and are having SEO issues go to our SEO troubleshooting guide related to organic traffic drops after a new site launch,
How Google Indexes Websites
Search engines provide a result for a search query. Google wants to provide the best, most accurate results as high as possible on the search results page. To do this, Google crawls millions of web pages, catalogs information, and ranks the best results on the search engine results page in order of relevance to the user.
There are dozens of factors that play into how well your website ranks for a given search query. My goal isn’t to explain all the tactics to rank on Google, but to give you a better understanding of basic best practices for making your website show up on Google search.
How to Use Google Webmaster Tools to Get Your Website Indexes
Webmaster Tools is critical for the health of your website on Google’s search properties. It’s free and takes very little time to set up. You’ll get everything from alerts about malware and hacks to the extremely useful ability to force Google to crawl your website on command.
The very first thing you need to do is register for a Google Webmaster Tools account. Google Webmaster Tools is a powerful search engine optimization tool that helps link your website to Google’s search engine. It also provides data from scrapes that Google’s search bots perform on your website. This gives you an understanding of how healthy your website looks to Google’s bots and gives you suggestions on things to improve.
To add a site, you’ll click “Add Site” and enter your domain. Please note, www.yoursite.com is different from http://www.yoursite.com. Follow the on-screen prompts to sync and verify your Google account with Google Webmaster Tools. Verifying your website will require an active Google Analytics account or server access. Accessing your server is covered in the next step.
<H2>How to Prevent Common Web Crawling Issues</H2>
To get the most out of this guide you’ll need to have server access so you can edit files on the back-end of your website. Speak with your website administrator or hosting company. If you use a self-hosting company like Siteground, you’ll have access to a C-Panel for editing this sort of information. I prefer using a client like Filezilla for editing server files.
FTP Quick Overview
FTP access to your website is important for transferring files to and from your server. This lets you check and edit your Robots.txt file to ensure it’s not set up to block search bots from crawling your website.
Using an FTP client is like navigating an operating system’s file explorer. There are folders and files. However, be careful when editing files on your server as you can do a lot of damage to a website through the server. Make sure to copy all files before editing so you have a quick backup available to replace any unintentional changes. If you’re unsure how to edit a web server through the FTP then get help from your website administrator.
Make sure your Robots.txt file isn’t blocking Google search bots.
Search engine robots need to be able to crawl your website easily. The Robots.txt file is a powerful file that can block your entire website from search bots with just a simple line of text called “disallow: /”.
You can see in the above screenshot that my Robots.txt file is blocking WordPress core files (/wp-admin/ and /wp-includes/ directories). This isn’t recommended anymore, so don’t take this advice. But you can at least see what a Robots.txt file looks like.
The Robots.txt file may seem like nothing more than a notepad file, but it’s incredibly powerful. This file lives on your server in the main directory of your website and controls search engine crawling behavior on your site. With a simple line of text, you can block your entire website from being indexed on Google.
If you’re not seeing your business on Google’s search engine and you just performed a website launch or re-design, this file could be blocking Google’s search engine crawl bots. When website developers build websites they often place a line of code here to block search engine crawl bot access. This is to prevent any visitors from stumbling onto a partially functional website.
Write relevant search-engine optimized meta titles, meta descriptions and headings.
The back end HTML code of a website holds some serious search engine goodies; a search robot is just code, after all, so you want to make sure your HTML elements are doing you favors.
Your website and web pages have a framework built using a code called HTML. This code has several tags that highlight information as being important. Google’s search engine scans these tags and logs this information.
Two very important meta tags are your meta title and meta description tags. These two tags control what shows up in Google search engine results pages and help identify your page’s general topic. They also act as “advertisements” of sorts meaning the effectiveness of the copy will increase the likelihood of someone clicking your webpage.
Your heading tags also send signals to Google in terms of the content of your page. You want to make sure your page has one H1 tag for your on-page title and it doesn’t hurt to have an H2 subheader including important keywords either.
Include natural keywords in your webpage URL.
Your webpage URL is a critical part of how Google’s search engine ranks your website. When you build a webpage, make sure you have a natural sounding URL that is descriptive. It should loosely summarize your page content. Here are good and bad examples:
Create a user-friendly and Google-robot-friendly page hierarchy.
This website navigation shows that the site has services with a drop-down menu highlighting the most important ones. This is a strong indicator to Google that these pages are important and should carry more weight than less important pages on the site.
How your pages are organized on your website matters; Google’s search engine bots need to easily crawl your website repeatedly and relay information that makes sense. That means you have to categorize your website and create paths of information.
For example, place multiple services under a “Services” tab on your navigational menu. If you have several business locations, place them all under a “Locations” tab on your menu. Good housekeeping goes a long way to ensure visitors and Google’s search engine bots find what they’re looking for.
Make logical internal links between pages on your website and links to off-site pages.
Just like a person visiting your site, Google’s search engine bot needs to click a link to visit a page. Make sure your webpages link to relevant pages. Every webpage you want to rank should have a link going to it. The next step helps with this process, but try to have at least one link in your main page copy. And don’t go overboard with linking – too many links can be a spam signal and hurt your search results page rankings.
Include an HTML sitemap in the footer of your website.
A good way to ensure all your pages have at least one link going to them is to use an HTML Sitemap. An HTML sitemap is a webpage that provides a link to every page and post on your website. This not only helps people find things they need; it helps Google’s search engine bots easily reach every page on your site.
A great use for an HTML sitemap is embedding it on your 404 error page. That way, when a visitor reaches a dead end on your site, they’re given a full map of places to go. You’ll also want to have a generic privacy page and terms of service pages as those have been shown to be trust signals.
Submit an XML sitemap to Google.
An XML sitemap is similar to an HTML sitemap but it’s submitted directly to Google through Google Webmaster Tools. It’s very important to do this as it’s a way of giving Google’s search engine bots a map of your website. It may seem repetitive, but there’s no harm in making sure search engines have more than enough information to find all your web pages.
Have proper schema.org markup and validate with Google’s data highlighter.
Google uses a web standard set by Schema.org to help show relevant content to people in different ways. An example of this is the rich snippet of a news article that shows on Google News.
If you’re a local business, I highly recommend using the data highlighter tool in Webmaster Tools to highlight your business’s contact page information. Combined with Google Local, this can be a great way of getting your website visited by people searching for related terms near your business location.
How to Get Your Website Indexed by Google With One Click
In Google Webmaster Tools, there’s a feature called Google Fetch that lets you manually tell Google’s search engine bot to crawl your website or a specific page. This is a great way to get content ranked quickly on Google.
Select your site, click “Crawl” on the left side and select “Fetch As Google.” Enter the page URL you’d like to crawl (excluding the domain information that’s listed) and hit “Fetch.” I make it a habit to manually fetch all new pages I create on my websites.
Whenever I edit a URL, place a redirect, or do any kind of significant change to the page structure, I re-fetch the page through Google Fetch. It can take a few minutes or days to see the changes in Google.
To see if your page has been crawled by Google, type in “site:yourwebsite.com” in Google’s search bar. Exclude the quotations and use your domain without the www. or http://. This shows all pages crawled by Google with your domain and is a great way to identify missing pages from Google’s search index. If you have comments, questions or need help at all feel free to reach out.