Google’s Mobile-Friendly Test Doesn’t Follow Robots.txt Rules

  • August 15, 2019
  • SEO

credit: Matt southern

Google’s John Mueller recently advised the company’s mobile-friendly test doesn’t follow the rules written in robots.txt files.

This topic came up in a Google Webmaster Central hangout where a question was asked about the mobile-friendly test and its relation to crawling and indexing.

Here is the question that was submitted:

“If I can see my text in Google’s mobile-friendly test, can I ensure that Google doesn’t have a problem with reading and indexing my content and my pages?”

In response, Mueller said it’s correct that if the mobile-friendly test can render a page then that’s a sign it can be crawled and indexed just fine.

The one thing to watch out for is robots.txt files, which the mobile-friendly test doesn’t respond to. That means it won’t tell you the server response for a page, or whether any JavaScript files are being blocked.

So, in addition to the mobile-friendly test, site owners should be using the inspect URL tool in Search Console to make sure everything is working as intended.

Hear Mueller’s full response in the video below, starting at the 28:36 mark:

“For the most part – yes, that’s correct. There can be subtle issues with JavaScript but, in general, the mobile-friendly test is something that uses the new chromium setup that we use for Googlebot. So if [the web page] works in a mobile-friendly test it should work for indexing as well.

One of the things to watch out for here is the mobile-friendly test – as far as I know – doesn’t follow the robots.txt rules.

So that’s something where you might want to use the Inspect URL tool in Search Console to make sure that, actually, your JavaScript files, and your server responses, and all of that are working well from that point of view.”

If you need any help with your SEO or Internet marketing, please contact, 833 377 9337 x 701 or

Jim is a Senior Digital Marketing Strategist Spyder Digital and has over 19 years of experience in the field. His insight and ability to drive new business for his clients from the Internet is unparalleled.


About SPYDER, web marketing agency

We are a digital marketing company with a focus on helping our customers achieve great results from various Internet marketing services across several key areas. 833 377 9337

Request a free quote

We offer professional web marketing services including content strategies,  SEO,  PPC, local marketing, Email campaigns, Social Media Strategies and digital consulting. Let's talk.


More from our blog

See all posts