Most robots.txt problems come from a short list of mistakes: a leftover Disallow: / that blocks the whole site, using Disallow to try to remove pages from Google, unsupported noindex or crawl-delay lines, blocked CSS and JavaScript, a file in the wrong place, and server errors on /robots.txt. Each issue below has its cause and… Continue reading Robots.txt Disallow: 12 Common Issues and How to Fix Them
Category: Technical SEO
Technical SEO: crawling, indexing, robots.txt, site structure and the fixes that let search engines read your site.