How to Block Bad Bots in Magento or Any Website Using .htaccess

Spread the love

Bots are a normal part of the internet. Some bots are useful, such as Googlebot, Bingbot, Google AdsBot, and Google Inspection Tool. These bots help search engines crawl your website, index your products, and support SEO and advertising.

However, many unwanted bots can hurt website performance. Scraper bots, SEO crawling bots, AI training bots, and automated tools can create thousands of unnecessary requests. On Magento websites, this can be even more dangerous because layered navigation, category filters, search pages, and product pages can become very heavy.

At EcomPlugins, we often see Magento stores slow down because of aggressive bot traffic. A simple and effective first step is to block unwanted bots from the .htaccess file.

Why Block Bad Bots?

Bad bots can cause several issues:

High server CPU and memory usage
Slow Magento category and search pages
Heavy layered navigation URLs
Increased hosting cost
Unnecessary database load
Poor customer experience
Fake crawling of filtered URLs
Possible scraping of product data and pricing

For Magento, URLs with many filters can be very expensive because they may trigger Elasticsearch, OpenSearch, MySQL queries, layered navigation logic, and full-page cache misses.

Safe Bot Blocking Strategy

The best approach is:

Allow trusted search and ads bots first.
Block unwanted scraper, AI, and SEO bots.
Optionally block heavy query URLs only for bad bots.

This avoids blocking important crawlers like Googlebot while still protecting the website from unnecessary traffic.

Example .htaccess Code to Block Bots

Add the following code near the top of your website .htaccess file.

<IfModule mod_rewrite.c>
    RewriteEngine On

    #################################################################
    # 1) Allow major search / ads bots first
    #################################################################
    RewriteCond %{HTTP_USER_AGENT} (Googlebot|Googlebot-Image|AdsBot-Google|Google-InspectionTool|bingbot|adidxbot) [NC]
    RewriteRule .* - [L]

    #################################################################
    # 2) Block unwanted / heavy / AI / scraper bots
    #################################################################
    RewriteCond %{HTTP_USER_AGENT} (AhrefsBot|SemrushBot|MJ12bot|DotBot|BLEXBot|MegaIndex|SeznamBot|Baiduspider|PetalBot|YandexBot|Bytespider|ClaudeBot|GPTBot|ChatGPT-User|CCBot|Amazonbot|Diffbot|DataForSeoBot|SeekportBot|Sogou|SerpstatBot|ZoominfoBot) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (python-requests|curl|wget|Go-http-client|libwww-perl|scrapy|httpclient|java/) [NC]
    RewriteRule .* - [F,L]

    #################################################################
    # 3) Optional: block some bad query-heavy bots only on layered URLs
    #################################################################
    RewriteCond %{REQUEST_URI} ^/(shop-all-carpet-tile\.html|tile\.html|engineered-hardwood\.html)$ [NC]
    RewriteCond %{QUERY_STRING} .{150,} [OR]
    RewriteCond %{QUERY_STRING} (carpet_color=|size_range=|tufted_weight_range=|species=|thickness=|wear_layer=|manufacturer=|pattern=) [NC]
    RewriteCond %{HTTP_USER_AGENT} (Baiduspider|SemrushBot|AhrefsBot|MJ12bot|DotBot|PetalBot|YandexBot) [NC]
    RewriteRule .* - [F,L]
</IfModule>

Explanation of the Rules

1. Allow Important Bots First

RewriteCond %{HTTP_USER_AGENT} (Googlebot|Googlebot-Image|AdsBot-Google|Google-InspectionTool|bingbot|adidxbot) [NC]
RewriteRule .* - [L]

This allows trusted search and advertising bots before applying any blocking rules.

This is important because you do not want to accidentally block Googlebot, Google AdsBot, or Bingbot.

2. Block Heavy and Unwanted Bots

RewriteCond %{HTTP_USER_AGENT} (AhrefsBot|SemrushBot|MJ12bot|DotBot|BLEXBot|MegaIndex|SeznamBot|Baiduspider|PetalBot|YandexBot|Bytespider|ClaudeBot|GPTBot|ChatGPT-User|CCBot|Amazonbot|Diffbot|DataForSeoBot|SeekportBot|Sogou|SerpstatBot|ZoominfoBot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (python-requests|curl|wget|Go-http-client|libwww-perl|scrapy|httpclient|java/) [NC]
RewriteRule .* - [F,L]

This blocks many common SEO crawlers, AI bots, scraper bots, and automated tools.

The [F,L] flag returns a forbidden response and stops further rewrite processing.

3. Block Query-Heavy Layered Navigation URLs

RewriteCond %{REQUEST_URI} ^/(shop-all-carpet-tile\.html|tile\.html|engineered-hardwood\.html)$ [NC]
RewriteCond %{QUERY_STRING} .{150,} [OR]
RewriteCond %{QUERY_STRING} (carpet_color=|size_range=|tufted_weight_range=|species=|thickness=|wear_layer=|manufacturer=|pattern=) [NC]
RewriteCond %{HTTP_USER_AGENT} (Baiduspider|SemrushBot|AhrefsBot|MJ12bot|DotBot|PetalBot|YandexBot) [NC]
RewriteRule .* - [F,L]

This rule is useful for Magento category pages with layered navigation filters.

For example, bots may crawl URLs like:

/category.html?color=red,blue&size=large&manufacturer=abc

These URLs can create unnecessary load on Magento, especially when filters are combined in many different ways.

Important Notes Before Using This Code

Before applying bot blocking rules, always test carefully.

Recommended steps:

Backup your .htaccess file.
Add the rules on staging first.
Test important pages.
Test Google Search Console URL Inspection.
Check server logs.
Monitor 403 responses.
Adjust bot names based on real traffic.

For Magento Websites

Magento websites are more sensitive to bot traffic because many pages require heavy backend processing.

Common high-load areas include:

Category pages
Search result pages
Layered navigation filter URLs
Product listing pages
Sitemap URLs
Checkout and customer URLs
REST API endpoints

Bot blocking can help improve Magento performance, but it should be combined with proper caching, server tuning, CDN configuration, and SEO-friendly URL management.

Better Protection with Cloudflare or Server Firewall

.htaccess is useful, but it is not always the best long-term solution for high-traffic websites.

For better protection, you can also use:

Cloudflare WAF rules
Cloudflare Bot Fight Mode
Rate limiting
Nginx rules
Apache server-level rules
Fail2ban
Magento full-page cache
Varnish cache
Robots.txt crawl-delay rules

Need Help Blocking Bad Bots in Magento?

At EcomPlugins, we help Magento and Adobe Commerce store owners improve website performance, reduce bad bot traffic, optimize servers, and fix crawling issues.

If your Magento website is slow because of bots, layered navigation URLs, or heavy server load, our team can help.

Final Thoughts

Blocking unwanted bots is a simple but powerful way to protect your Magento or eCommerce website.

The goal is not to block every bot. The goal is to allow useful bots and stop harmful or unnecessary crawlers that waste server resources.

A carefully written .htaccess rule can reduce server load, improve website speed, and protect your Magento store from aggressive crawling.