Search Engine Basics: Indexing, Crawling & Relevancy

If you have three minutes to spare and want to understand how search engines work, check out this “How Search Works” video from Google. Even though it is older, it is still relevant and explains the concept simply:

The internet as we know it, at its most simple, is just a collection of files and pages that are connected to one another by links. Much like how your own home computer has a file address for every single document on your hard drive, each site on the internet also has a file address for every single component that makes up a web page. These addresses on the internet are called links, or hyperlinks, and let both human users and search engines browse and explore the internet to discover new pages, images and files.

Prior to the existence of search engines, one would have to know the exact location in order to retrieve a page or a file! In order to solve for that issue, search engines were created. Therefore, when you do a search through an engine like Google, despite the fact that we think that we are searching the internet, we are actually only searching through Google’s vast index of sites and files.

Search engines were created with three main functions:

• to “crawl” all of the links on the web,
• to index all of these links into their database,
• and then to rank these backlinks.

Search engines rely on automated software, often referred to as spiders or bots, to crawl and seek out content on the internet. When arriving on a new site, all search engine bots look for a file called the robots.txt file. Think of this as your website’s map that it offers to a search engine bot. It tells the bot what is important on your site, where it shouldn’t go and what it should ignore.

This file is important to a website because a search engine like Google will allow each website a limited amount of “crawl budget.” So, if you have a website with thousands of files and pages, but only a few of them are important to a search engine, a robot.txt file can point the bot in the right direction and help maximize crawl budget. We’ll talk more about your robot.txt file and crawl budget in another article, but for now, it’s great to know what it is and what it does!

Once a search engine bot has crawled a site, it stores and compiles that information to help build the larger database of the search engine. This is called Indexing, and once a site, page or file is indexed, it becomes available for a user to query. Only when this process has occurred can your potential client search for services that you offer and see your website in the search engine results.

During the crawling phase, a bot is also determining what your site is about, as well as looking at various signals to understand how trustworthy and authoritative your site is. This process is important because part of the job of the search engine is to determine which pages it has within its index are the most relevant to the user’s query. After all, if you are searching for “best ice cream cones near me,” you don’t want to get served up a website for an ice cream shop 100 miles away. So, the moment that a user types their query into the search engine, it is performing a variety of functions to determine which results it can offer that would be the most relevant.

There are a number of things that search engines are looking at when trying to understand a page or site, and to determine its relevance, trustworthiness and authority. The first question that a search engine is trying to answer is “what is this site about?” Some of the most important things that a search engine is looking at to answer this question are:

1. The title tag of the web page. The title tag within a webpage is one of the biggest key indicators of what content will follow on the page.
2. Analysis of the page content. In years past this used to be very tied to the frequency and density of specific keyword phrases. Today, a search engine is looking at the context of the words and phrases used on a page.
3. Anchor text used in links to the page. Whenever you click on linked text, the words that are clicked on are called anchor text. Search engines and humans use anchor text as a kind of label for what to expect from the linked page.
4. The subject matter of linking page. If another site is linking to yours, it is helpful for it to be relevant subject matter to help search engines see that your content is relevant and useful in your niche.
5. The topic of the site where the page lives. If your site is about your creative business, but your page is about losing weight, this topic mismatch creates confusion and signals a lack of overall relevance.
6. How users respond to the results from the search engine. Search engines look at user behavior. So, if they deliver a page for a specific query and users hit the back button immediately, it gives the search engine negative signals about the relevancy of that content.
7. Importance and popularity. Search Engines also look at the importance of a site (think the New York Times) and the popularity of its content in determining whether or not a page is a relevant match for a user’s query.

Indexing Your Site

The first step to making sure that your business’ website can be found by a search engine is to make sure that it has been indexed. There is a very simple way to do this. First let’s go over to Google.com and type in your site’s address in this format: site:www.yoursite.com. Make sure that you replace your website in that format!

Here is an example using phlearn.com:

Both Google and Bing will automatically crawl your site and look for updates regularly. You can also use Google’s Search Console and Bing’s Webmaster Tools to submit your site and pages for indexing. By submitting your site and pages, you are asking the search engine to crawl your content in the hopes that it will be indexed faster.

Using Google Search Console

If you have your Search Console set up, go to https://www.google.com/webmasters/. If you haven’t set up your Google Search Console, there will be a few extra steps.

To add your website to Google Search Console, click on “Add property” on the top left:

You will then be prompted to add in the URL address of your website. (Remember, if your site loads with an https or a www, you must include this in the URL! Load your site in another browser just to make certain you are using the correct address.)

From here, Google will walk you through a process to verify ownership of your site. There are several methods to do this, so use whichever one you find the easiest.

Once you have verified your site, you are ready to submit your site or future pages to the index! On the left hand side of your Search Console interface, you will see URL Inspection. Click on this and you will be able to perform a search for any page on your domain to see if it is indexed by Google.

If Google has indexed your site or page, you will get a response that the URL is on Google with a message like this one:

However, if your URL has not been indexed, you will be informed with a message like this one and given an option to submit it:

Using Bing Webmaster Tools

The process for Bing is similar, and you can check out their instructions here.

If you don’t already have a Bing Webmaster Tools account, go ahead and create one:

Next you’ll need to add your website and will be taken to a form to fill out a little bit of information about your site:

Once you have added your site, you will need to verify it:

After you verify your website, you can submit a sitemap to Bing Webmaster Tools, which will be the quickest way for the search engine to find your content.

Later, you can add individual URLs.

Once your site is indexed it can be considered for ranking. This is where the true work of SEO marketing begins!

Ranking Your Site

Ultimately, it is simply not enough for your website to be indexed. In order to receive traffic, as a business owner you must prove to a search engine like Google that your website has the most relevant, helpful and important information for a user looking for services that you offer. Working to make sure that your site sends the right signals to that search engine is where the heavy lifting of an SEO marketing strategy happens.

At its most simple, the rank is the order in which websites appear in the results page after a user completes a search. When a search engine receives a query, its ultimate goal is to return to the user the most relevant and helpful results possible. According to Google, there are over 200 unique signals that are looked at when attempting to help a user find the most relevant result or answer to their question.

Additionally, Google employs over 10,000 search raters around the world who evaluate search results and then rate the quality of the pages that appear at the top of the organic results. The data on the quality of these results is then used to help inform the algorithms that Google uses to automate the ranking of pages. If you would like a peek into what search raters, and ultimately Google, are looking for when assessing website quality, they have published a nearly-200 page handbook here.

However, instead of making you scour a dense handbook or research the 200 signals that Google uses to rank, we are going to talk about the things that are in your control and that can make the most impact in your SEO marketing strategy.

Content

More than anything, Google is looking for comprehensive and relevant information to answer a user’s query. So what does that ideal content look like? Think about your potential client and their questions. Now imagine answering those questions in long-form content including images, graphics and video. In this answer you provide anything and everything that the client could want to know.

Studies have shown that long-form content performs better than shorter content, but don’t write a long article simply for the sake of cramming a lot of words in. Google is looking for relevance and quality, and if it sees that users are clicking away without spending time digesting your content, that sends a bad signal. Find ways to make the content on your site both comprehensive and relevant to your potential client’s needs.

Inbound Links

Links originating from other websites and pointing to yours are generally positive signals to a search engine. The more relevant and popular another website is, the better it is to have a link pointing to your website from it. In the past, marketers would try to gain as many links as possible to their website, whether it was from a spammy site or a quality one. Over time, this practice became hurtful to marketers, as algorithmic penalties were issued, and once-ranking websites began to drop out of the organic search results.

Creating relevant and useful content for your audience is the first step in creating a healthy inbound link strategy. Once you have several great pieces of content on your site, the next step is to find other relevant sites and have them share your work. Many photographers do this by creating fun and unique photo sessions and getting news outlets and blogs to feature the work.

While the promotional factor of having a news or blog outlet feature your work is great, the long-term benefits are the signals that the search engine sees from that inbound link. Your content is now seen as more relevant than it was before, and you will begin generating more traffic for queries that are relevant to your content.

If you’re having trouble thinking of great original content, then use others who are authorities in your areas of expertise as inspiration. Don’t steal! But using the Skyscraper link building technique, find other relevant content and then find ways to improve upon it with more detail or a unique spin.

Mobile First

Face it, we all live on our phones these days and it’s a mobile first world. Back in 2016, Google made a change to a mobile first index, where it began to primarily crawl the mobile version of a website rather than a desktop version. So making sure that your website is mobile-friendly is extremely important! Designing your site and content with a mobile device in mind is even better.

If a user can’t easily navigate your website on a mobile device, it’s very likely that they will give up and leave. The same thing can be said if your site loads slowly on a mobile device. When a user abandons your site, this sends negative signals to search engines and can hurt your ability to rank. According to Google, even back in 2010, “two seconds is the threshold for ecommerce website acceptability.” While you may not be running an ecommerce website, this goes to show just how important page load was even eight years ago. In fact, according to Google, over half of website visits are abandoned if a website takes more than three seconds to load!

If you are curious to know how fast your site loads, you can check your page speed using Google’s PageSpeed Insights tool. If you find that your page loads slowly, look for ways to compress images without losing quality. That can have a great effect on speeding up your page’s loading time. Online image compression tools like TinyJPG or TinyPNG are great. Other auditing tools such as Pingdom can help you find other ways to speed up your site.

HTTPS

There are also several technical factors that can help with rankings. One of the most important is that Google has begun to favor websites that use HTTPS encryption. If you type your website address in the URL bar and see that the address begins with HTTPS, then you’re already set up for this! However, if your site is not using HTTPS, contact your website provider and ask what steps need to be taken to make the switch. Google is already marking sites that are not utilizing HTTPS as Not Secure:

Why SEO Is Important

Now that we have discussed how a search engine’s “mind” works as it moves through the internet trying to make sense of all of the content available, let’s talk about how and why this is important to you and your business. Many business owners become frustrated with SEO, feeling like the marketing techniques in Search Engine Optimization are trying to “game” the system.

True, there are techniques that aim to do just that, and the internet used to be full of practitioners using those tactics to reap giant rewards. As we have discussed in an earlier article about SEO, these techniques now come with high penalties and are worth avoiding. Instead let’s think about search engines as giant machines that we are trying to teach about our website, our business and our offerings, all through a language we call SEO.

It begins with building a website that search engines can understand on a basic level. But a search engine, on its own, is not built to understand the quality of the content on a site without other signals to help it understand the relevance and importance of that content. “Build it and they will come” simply does not work for your business on today’s internet. A search engine is looking to understand how people interact with your content: how they talk about it and how they share it.

For someone just getting started, this might seem like a bit of a Catch 22: first you have to create the content and then you have to prove to the search engine that the content is the best that there is. But you can only do this by showing the search engine that others value your content. This is why SEO is a long game investment of time and effort. But the payoff for your business is consistent traffic month over month, which ideally converts into leads for your business.

Ultimately, SEO is important to your business because your prospective clients are important to your business. At its core, search engines are trying to serve the user. This is the biggest clue for us in our marketing: ask what would serve your user or your prospective client. SEO is no longer about keywords. It is about helping a user along on their journey, and hopefully letting them know that you have the answers and services that they need in that journey!

Your Takeaway

Understanding how a search engine operates and how its bots move through and “see” the internet is important because then you know how to teach the search engine about your site. Additionally, it gives you important context to understand how to approach your business’ SEO strategy. When you know how and why a search engine favors certain pages and delivers them as answers to a user’s query, you can reverse engineer this to help create similarly relevant and helpful pages on your own website. But, ultimately, your marketing strategy should have a user-first mentality, and give your prospective client what they most need when they are searching for it!