fbpx

Bridging Gap

Bridging Gap

Integrated Marketing Communication Agency.

We craft beautifully useful marketing and digital products that grow businesses.

T (917) 720 3126
Email: gaurav.sodhi@bridginggap.in

Bridging gap (B.Gap Pvt. Ltd.)
244 Fifth Avenue, Manhattan New York, NY, US 10001

Get in touch: +91-983-383-0474
  • MY CART
    No products in cart.
  • About us
  • Voice Your Business
    • India
    • USA
  • Services
    • Web & Mobile Development
    • SEO Services
    • Graphic Design
    • Marketing
      • Experiential Marketing (Events)
      • Email Marketing
      • Social Media Marketing
      • Hotel Marketing
    • Social Media
    • Brand Building
  • Portfolio
    • Strategic Creations
  • Beyond the Bridge
  • Contact us
Enquiry
0
Friday, 21 January 2022 / Published in News, Uncategorized, Web Design

Does Google Have A Problem With Big Robots.txt Files? – Search Engine Journal

Are large robots.txt files a problem for Google? Here’s what the company says about maintaining a limit on the file size.
Google addresses the subject of robots.txt files and whether it’s a good SEO practice to keep them within a reasonable size.
This topic is discussed by Google’s Search Advocate John Mueller during the Google Search Central SEO office-hours hangout recorded on January 14.
David Zieger, an SEO manager for a large news publisher in Germany, joins the livestream with concerns about a “huge” and “complex” robots.txt file.
How huge are we talking here?
Zieger says there’s over 1,500 lines with a “multitude” of disallows that keeps growing over the years.
The disallows prevent Google from indexing HTML fragments and URLs where AJAX calls are used.
Zieger says it’s not possible to set a noindex, which is another way to keep the fragments and URLs out of Google’s index, so he’s resorted to filling the site’s robots.txt with disallows.
Are there any negative SEO effects that can result from a huge robots.txt file?
Here’s what Mueller says.
A large robots.txt file will not directly cause any negative impact to a site’s SEO.
However, a large file is harder to maintain, which may lead to accidental issues down the road.
Mueller explains:
“No direct negative SEO issues with that, but it makes it a lot harder to maintain. And it makes it a lot easier to accidentally push something that does cause issues.
So just because it’s a large file doesn’t mean it’s a problem, but it makes it easier for you to create problems.”
Zieger follows up by asking if there are any issues with not including a sitemap in the robots.txt file.
Mueller says that’s not a problem:
“No. Those different ways of submitting a sitemap are all equivalent for us.”
Zieger then launches into a several more follow-up questions that we’ll take a look at in the next section.
Related: Google SEO 101: Blocking Special Files in Robots.txt
Zieger asks Mueller what would be the SEO impact of radically shortening the robots.txt file. Such as removing all the disallows, for example.
The following questions are asked:
He sums up his questions by stating most of what’s disallowed in his robots.txt file are header and footer elements that aren’t interesting for the user.
Mueller says it’s difficult to know exactly what would happen if those fragments were suddenly allowed to be indexed.
A trial and error approach might be the best way of figuring this out, Mueller explains:
“It’s hard to say what you mean with regards to those fragments
My thought there would be to try to figure out how those fragment URLs are used. And if you’re unsure, maybe take one of these fragment URLs and allow its crawling, and look at the content of that fragment URL, and then check to see what happens in search.
Does it affect anything with regards to the indexed content on your site?
Is some of that content findable within your site suddenly?
Is that a problem or not?
And try to work based on that, because it’s very easy to block things by robots.txt, which actually are not used for indexing, and then you spend a lot of time maintaining this big robots.txt file, but it actually doesn’t change that much for your website.”
Related: Best Practices for Setting Up Meta Robots Tags & Robots.txt
Zieger has one last follow-up regarding robots.txt files, asking if there are any specific guidelines to follow when building one.
Mueller says there’s no specific format to follow:
“No, it’s essentially up to you. Like some sites have big files, some sites have small files, they should all just work.
We have an open source code of the robots.txt parser that we use. So what you can also do is get your developers to run that parser for you, or kind of set it up so that you can test it, and then check the URLs on your website with that parser to see which URLs would actually get blocked and what that would change. And that way you can test things before you make them live.”
The robots.txt parser Mueller refers to can be found on Github.
Hear the full discussion in the video below:

Featured Image: Screenshot from YouTube.com/GoogleSearchCentral, January 2022.
Get our daily newsletter from SEJ’s Founder Loren Baker about the latest news in the industry!
Matt Southern has been the lead news writer at Search Engine Journal since 2013. With a degree in communications, Matt … [Read full bio]
Subscribe to our daily newsletter to get the latest industry news.
Subscribe to our daily newsletter to get the latest industry news.

source

  • Tweet

What you can read next

5 Critical Priorities for the US Health Care System – Harvard Business Review
SRK, Munawar, Shami – 2021 showed the fragility of Indian Muslims’ celebrity status – ThePrint
How To Get Travel Insurance For USA – Forbes Advisor UK – Forbes

Recent Posts

  • SEO service in Bandra

    Beyond Keywords: How Search Intent is Shaping SEO Strategies in 2025

    In the dynamic realm of digital marketing, unde...
  • Best Hotel Marketing Agency

    OTA vs Direct bookings- How Hotels can achieve Maximum Revenue ?

    Best Hotel Marketing Agency...
  • Google Vs SEO

    Google Ads vs. SEO – Which Is Better? Get Expert Strategy from Bridging Gap, Mumbai

    In the fast-paced world of digital marketing, b...
  • best digital marketing agency in Delhi

    Branding Beyond the Logo: The Emotional Triggers That Make Customers Buy

    Introduction to Branding Branding is much more ...
  • Bridging Gap: 40% Revenue Increase for a Resort Through Smart OTA Strategies

    The hospitality industry is fiercely competitiv...

Archives

  • February 2025
  • January 2025
  • December 2024
  • May 2024
  • April 2022
  • March 2022
  • February 2022
  • January 2022
  • December 2021
  • June 2017

Categories

  • Branding
  • Marketing
  • News
  • SEO
  • Social Media
  • Uncategorized
  • Web Design

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org
Company
  • About us
  • Voice Your Business
  • Services
  • Portfolio
  • Beyond the Bridge
  • Contact us
Social
  • Instagram
  • Facebook
  • Twitter
Support
  • FAQ
  • Terms
  • Privacy

Bridging Gap

Call USA :+1-347-587-8585

Call IND: +91-983-383-0474

info@bridginggap.in

© 2025 All rights Reserved @Bridging Gap.

TOP