Google Launches Effort to Make Robots Exclusion Protocol an Internet Standard, Open Sources Robots.txt Parser

Website owners have been excluding web crawlers using the Robots Exclusion Protocol (REP) on robots.txt files for 25 years. More than 500 million websites use robots.txt files to talk to bots, according to Google’s data. Up until now, there has never been an official Internet standard, no documented specification for (more…)

Continuer la lecture Google Launches Effort to Make Robots Exclusion Protocol an Internet Standard, Open Sources Robots.txt Parser

Google Adds New Desktop/Mobile Selector to the Rich Results Testing Tool

Google’s rich results testing tool has been updated to include a selector for desktop or mobile so website owners can check their results on both platforms. This tool shows if pages are properly structured for display in Google’s carousels, images, breadcrumbs, events, books, and other types of rich snippets. A (more…)

Continuer la lecture Google Adds New Desktop/Mobile Selector to the Rich Results Testing Tool

Google’s Mobile-First Indexing is Now Default for New Domains

Google is enabling mobile-first indexing (crawling sites with a mobile user-agent) by default for new domains as of July 1, 2019. These are sites Google defines as “previously unknown to Google Search.” Mobile searches are the primary way that users engage with the search engine for the past few years, (more…)

Continuer la lecture Google’s Mobile-First Indexing is Now Default for New Domains