Connect with us


Google releases new “How Search Works” episode about crawling

Google releases new “How Search Works” episode about crawling


Google has released the latest installment of its educational video series “How Search Works,” explaining how its search engine discovers and accesses web pages through crawling.

Google Engineer Detailed Crawl Process

In a seven-minute episode hosted by Google analyst Gary Illyes, the company details the technical aspects of how Googlebot (the software Google uses to crawl the web) works.

Illyes outlines the steps Googlebot takes to find new and updated content from the internet's trillions of web pages and make it searchable on Google.

Ilies explains:

“Most new URLs that Google discovers come from other known pages that Google has previously crawled.

Consider a news site with various category pages that link to individual news articles.

Google can find most published articles by visiting the category page from time to time and extracting the URL that leads to the article. ”

How Googlebot crawls the web

Googlebot starts by discovering new URLs by following links from known web pages. This process is called URL discovery.

Avoid site overload by crawling each site at its own customized speed based on server response time and content quality.

Googlebot uses current versions of the Chrome browser to render pages, run JavaScript, and correctly display dynamic content loaded by scripts. It also only crawls public pages, not pages after login.

Related: Google answers questions about crawl budgets

Improved discovery and crawlability

Illyes emphasized the usefulness of sitemaps (XML files that list a site's URLs) to help Google discover and crawl new content.

He advised developers to let their content management systems automatically generate sitemaps.

Optimizing technical SEO factors such as your site's architecture, speed, and crawl directives will also improve its crawlability.

Here are some additional tactics to make your site more crawlable.

Avoid exhausting your crawl budget – Websites that update frequently can exceed Googlebot's crawl budget and prevent new content from being discovered. Careful configuration of your CMS and the use of rel= “next” / rel= “prev” tags can help. Implement good internal linking – Linking to new content from category pages and hub pages helps Googlebot discover new URLs. Effective internal link structure improves crawlability. Ensure pages load quickly – Sites that are slow to respond to Googlebot's fetches may have their crawl speeds throttled. Optimizing your page performance will make it crawl faster. Eliminating Soft 404 Errors – Fixing soft 404 errors caused by CMS misconfiguration will ensure that URLs are directed to valid pages and increase crawl success rates. Consider adjusting robots.txt – A strict robots.txt setting can block useful pages. An SEO audit may uncover restrictions that can be safely removed.Latest educational video series

The latest video comes after Google last week launched an educational “How Search Works” series that sheds light on the search and indexing process.

Our newly released episode on Crawls provides insight into one of the most basic operations of search engines.

In the coming months, Google will produce additional episodes exploring topics such as indexing, quality assessment, and search refinement.

The series is available on Google Search Central's YouTube channel.

FAQ What is Google's crawl process?

Google's crawl process, outlined in a recent episode of our “How Search Works” series, includes the following key steps:

Googlebot discovers new URLs by following links from known pages it has previously crawled. It strategically crawls your site at a customized speed to avoid overloading your servers, taking into account response time and content quality. The crawler also uses the latest version of Chrome to render pages, correctly display content loaded by JavaScript, and access only publicly available pages. Optimizing technical SEO elements and using a sitemap will help Google crawl your new content. How can a marketer ensure that his company's content is effectively discovered and crawled by his Googlebot?

Marketers can employ the following strategies to increase the discoverability and crawlability of their content by Googlebot.

Implement automatic sitemap generation within your content management system. Focus on optimizing technical SEO factors such as site architecture and loading speed, and use crawl directives appropriately. By configuring your CMS efficiently and using pagination tags, you ensure that frequent content updates don't deplete your crawl budget. Create an effective internal link structure to help discover new URLs. Check and optimize your website's robots.txt file to make sure it's not overly restrictive for Googlebot.




The mention sources can contact us to remove/changing this article

What Are The Main Benefits Of Comparing Car Insurance Quotes Online

LOS ANGELES, CA / ACCESSWIRE / June 24, 2020, / Compare-autoinsurance.Org has launched a new blog post that presents the main benefits of comparing multiple car insurance quotes. For more info and free online quotes, please visit https://compare-autoinsurance.Org/the-advantages-of-comparing-prices-with-car-insurance-quotes-online/ The modern society has numerous technological advantages. One important advantage is the speed at which information is sent and received. With the help of the internet, the shopping habits of many persons have drastically changed. The car insurance industry hasn't remained untouched by these changes. On the internet, drivers can compare insurance prices and find out which sellers have the best offers. View photos The advantages of comparing online car insurance quotes are the following: Online quotes can be obtained from anywhere and at any time. Unlike physical insurance agencies, websites don't have a specific schedule and they are available at any time. Drivers that have busy working schedules, can compare quotes from anywhere and at any time, even at midnight. Multiple choices. Almost all insurance providers, no matter if they are well-known brands or just local insurers, have an online presence. Online quotes will allow policyholders the chance to discover multiple insurance companies and check their prices. Drivers are no longer required to get quotes from just a few known insurance companies. Also, local and regional insurers can provide lower insurance rates for the same services. Accurate insurance estimates. Online quotes can only be accurate if the customers provide accurate and real info about their car models and driving history. Lying about past driving incidents can make the price estimates to be lower, but when dealing with an insurance company lying to them is useless. Usually, insurance companies will do research about a potential customer before granting him coverage. Online quotes can be sorted easily. Although drivers are recommended to not choose a policy just based on its price, drivers can easily sort quotes by insurance price. Using brokerage websites will allow drivers to get quotes from multiple insurers, thus making the comparison faster and easier. For additional info, money-saving tips, and free car insurance quotes, visit https://compare-autoinsurance.Org/ Compare-autoinsurance.Org is an online provider of life, home, health, and auto insurance quotes. This website is unique because it does not simply stick to one kind of insurance provider, but brings the clients the best deals from many different online insurance carriers. In this way, clients have access to offers from multiple carriers all in one place: this website. On this site, customers have access to quotes for insurance plans from various agencies, such as local or nationwide agencies, brand names insurance companies, etc. "Online quotes can easily help drivers obtain better car insurance deals. All they have to do is to complete an online form with accurate and real info, then compare prices", said Russell Rabichev, Marketing Director of Internet Marketing Company. CONTACT: Company Name: Internet Marketing CompanyPerson for contact Name: Gurgu CPhone Number: (818) 359-3898Email: [email protected]: https://compare-autoinsurance.Org/ SOURCE: Compare-autoinsurance.Org View source version on accesswire.Com:https://www.Accesswire.Com/595055/What-Are-The-Main-Benefits-Of-Comparing-Car-Insurance-Quotes-Online View photos


to request, modification Contact us at Here or [email protected]