Spread the love

Search engine bots like Googlebot and Bingbot are essential for SEO, but not every bot visiting your website is beneficial. Many websites experience heavy traffic from SEO crawlers, AI bots, scrapers, and automated tools that consume server resources without generating revenue.

If you’re running a Magento, Adobe Commerce, WordPress, Shopify, or any other website on Nginx, blocking unwanted bots can significantly reduce CPU usage, improve page load times, and lower hosting costs.

At EcomPlugins, we’ve helped numerous eCommerce businesses optimize their servers by identifying and blocking aggressive bot traffic while ensuring legitimate search engines continue to crawl the site.

Why Block Bad Bots?

Unwanted bots can cause serious performance issues, including:

High CPU and memory usage
Increased PHP-FPM processes
Slow Magento category and product pages
Excessive Elasticsearch/OpenSearch queries
Increased MySQL load
Higher bandwidth consumption
Product data scraping
Crawling of millions of unnecessary filtered URLs

For Magento stores, layered navigation pages can generate thousands of unique URL combinations, making them an attractive target for crawler bots.

Which Bots Should Be Allowed?

Never block major search engines that help your SEO.

Recommended bots to allow:

Googlebot
Googlebot-Image
Google Inspection Tool
Google AdsBot
Bingbot
Microsoft adidxbot

These bots index your website and help customers discover your products.

Which Bots Should Be Blocked?

Many SEO tools and AI crawlers generate heavy traffic without providing direct value.

Common examples include:

AhrefsBot
SemrushBot
MJ12bot
DotBot
BLEXBot
MegaIndex
YandexBot
Baiduspider
PetalBot
Bytespider
GPTBot
ClaudeBot
ChatGPT-User
CCBot
Amazonbot
Diffbot
DataForSeoBot
SerpstatBot
ZoominfoBot

You may also want to block command-line clients and scraping tools such as:

curl
wget
python-requests
scrapy
Go-http-client
libwww-perl

Method 1: Simple Nginx Configuration

Inside your server {} block:

set $block_bad_bot 0;

# Allow search engine bots
if ($http_user_agent ~* "(Googlebot|Googlebot-Image|AdsBot-Google|Google-InspectionTool|bingbot|adidxbot)") {
    set $block_bad_bot 0;
}

# Block unwanted bots
if ($http_user_agent ~* "(AhrefsBot|SemrushBot|MJ12bot|DotBot|BLEXBot|MegaIndex|SeznamBot|Baiduspider|PetalBot|YandexBot|Bytespider|ClaudeBot|GPTBot|ChatGPT-User|CCBot|Amazonbot|Diffbot|DataForSeoBot|SeekportBot|Sogou|SerpstatBot|ZoominfoBot)") {
    set $block_bad_bot 1;
}

# Block scraping tools
if ($http_user_agent ~* "(python-requests|curl|wget|Go-http-client|libwww-perl|scrapy|httpclient|java/)") {
    set $block_bad_bot 1;
}

if ($block_bad_bot) {
    return 403;
}

This immediately blocks requests from unwanted bots while allowing trusted search engines.

Method 2: Recommended Approach Using map

For production websites, using map is cleaner, faster, and more efficient than multiple if statements.

Place the following inside the http block:

map $http_user_agent $bad_bot {

    default 0;

    ~*(AhrefsBot|SemrushBot|MJ12bot|DotBot|BLEXBot|MegaIndex|SeznamBot|Baiduspider|PetalBot|YandexBot|Bytespider|ClaudeBot|GPTBot|ChatGPT-User|CCBot|Amazonbot|Diffbot|DataForSeoBot|SeekportBot|Sogou|SerpstatBot|ZoominfoBot) 1;

    ~*(python-requests|curl|wget|Go-http-client|libwww-perl|scrapy|httpclient|java/) 1;
}

Then inside your server block:

if ($bad_bot) {
    return 403;
}

This approach is easier to maintain and performs better on busy servers.

Protect Heavy Magento Layered Navigation Pages

Magento category pages with layered navigation can create thousands of URL combinations.

Example:

/shop-all-carpet-tile.html?manufacturer=ABC&color=Blue&size=24x24&pattern=Loop

These pages often trigger:

Elasticsearch/OpenSearch
Layered navigation processing
Product collection filtering
Cache generation
Database queries

Instead of allowing bots to crawl every filter combination, block unwanted bots from these URLs.

Example:

location ~* ^/(shop-all-carpet-tile\.html|tile\.html|engineered-hardwood\.html)$ {

    if ($http_user_agent ~* "(AhrefsBot|SemrushBot|MJ12bot|DotBot|YandexBot|PetalBot|Baiduspider)") {
        return 403;
    }

}

This protects your most expensive pages without affecting real customers.

Test Your Configuration

Before restarting Nginx:

sudo nginx -t

If everything is valid:

sudo systemctl reload nginx

Never restart Nginx without testing the configuration.

Verify the Rules

You can test using curl:

curl -A "AhrefsBot" https://example.com

Expected response:

403 Forbidden

Now test Googlebot:

curl -A "Googlebot" https://example.com

The page should load normally.

Additional Best Practices

Blocking bots is only one part of website optimization.

For Magento stores, we also recommend:

Enable Full Page Cache
Use Varnish Cache
Configure Cloudflare WAF
Enable Rate Limiting
Optimize robots.txt
Configure canonical URLs
Prevent infinite crawl spaces
Tune Elasticsearch/OpenSearch
Monitor server logs regularly

Together, these improvements can significantly reduce server load while improving user experience.

Common Mistakes

Avoid these common issues:

Blocking Googlebot or Bingbot accidentally
Returning 500 errors instead of 403
Using too many nested if statements
Forgetting to test configuration with nginx -t
Blocking legitimate API clients unintentionally
Ignoring server logs after deploying rules

Conclusion

Every website receives bot traffic, but not all bots deserve access to your server resources.

A properly configured Nginx server can block unwanted SEO crawlers, AI bots, and scraping tools before they reach your application. This reduces CPU usage, improves website speed, and keeps your Magento store responsive for real customers.

If you’re experiencing high server load, slow Magento performance, or excessive bot traffic, implementing these Nginx rules is an effective first step toward better performance and security.

Need Help Optimizing Your Magento Store?

At EcomPlugins, we specialize in Magento and Adobe Commerce performance optimization, server tuning, shipping solutions, and custom development.

Whether you’re dealing with bad bots, slow category pages, Elasticsearch issues, or high CPU usage, our team can help.

Contact us today:
https://www.ecomplugins.com/index.php?route=information/contact