Site icon EcomPlugins Blog

How to Block Bad Bots in Nginx to Protect Magento and High-Traffic Websites

Vipin Garg

Search engine bots like Googlebot and Bingbot are essential for SEO, but not every bot visiting your website is beneficial. Many websites experience heavy traffic from SEO crawlers, AI bots, scrapers, and automated tools that consume server resources without generating revenue.

If you’re running a Magento, Adobe Commerce, WordPress, Shopify, or any other website on Nginx, blocking unwanted bots can significantly reduce CPU usage, improve page load times, and lower hosting costs.

At EcomPlugins, we’ve helped numerous eCommerce businesses optimize their servers by identifying and blocking aggressive bot traffic while ensuring legitimate search engines continue to crawl the site.

Why Block Bad Bots?

Unwanted bots can cause serious performance issues, including:

For Magento stores, layered navigation pages can generate thousands of unique URL combinations, making them an attractive target for crawler bots.

Which Bots Should Be Allowed?

Never block major search engines that help your SEO.

Recommended bots to allow:

These bots index your website and help customers discover your products.

Which Bots Should Be Blocked?

Many SEO tools and AI crawlers generate heavy traffic without providing direct value.

Common examples include:

You may also want to block command-line clients and scraping tools such as:

Method 1: Simple Nginx Configuration

Inside your server {} block:

set $block_bad_bot 0;

# Allow search engine bots
if ($http_user_agent ~* "(Googlebot|Googlebot-Image|AdsBot-Google|Google-InspectionTool|bingbot|adidxbot)") {
    set $block_bad_bot 0;
}

# Block unwanted bots
if ($http_user_agent ~* "(AhrefsBot|SemrushBot|MJ12bot|DotBot|BLEXBot|MegaIndex|SeznamBot|Baiduspider|PetalBot|YandexBot|Bytespider|ClaudeBot|GPTBot|ChatGPT-User|CCBot|Amazonbot|Diffbot|DataForSeoBot|SeekportBot|Sogou|SerpstatBot|ZoominfoBot)") {
    set $block_bad_bot 1;
}

# Block scraping tools
if ($http_user_agent ~* "(python-requests|curl|wget|Go-http-client|libwww-perl|scrapy|httpclient|java/)") {
    set $block_bad_bot 1;
}

if ($block_bad_bot) {
    return 403;
}

This immediately blocks requests from unwanted bots while allowing trusted search engines.

Method 2: Recommended Approach Using map

For production websites, using map is cleaner, faster, and more efficient than multiple if statements.

Place the following inside the http block:

map $http_user_agent $bad_bot {

    default 0;

    ~*(AhrefsBot|SemrushBot|MJ12bot|DotBot|BLEXBot|MegaIndex|SeznamBot|Baiduspider|PetalBot|YandexBot|Bytespider|ClaudeBot|GPTBot|ChatGPT-User|CCBot|Amazonbot|Diffbot|DataForSeoBot|SeekportBot|Sogou|SerpstatBot|ZoominfoBot) 1;

    ~*(python-requests|curl|wget|Go-http-client|libwww-perl|scrapy|httpclient|java/) 1;
}

Then inside your server block:

if ($bad_bot) {
    return 403;
}

This approach is easier to maintain and performs better on busy servers.

Protect Heavy Magento Layered Navigation Pages

Magento category pages with layered navigation can create thousands of URL combinations.

Example:

/shop-all-carpet-tile.html?manufacturer=ABC&color=Blue&size=24x24&pattern=Loop

These pages often trigger:

Instead of allowing bots to crawl every filter combination, block unwanted bots from these URLs.

Example:

location ~* ^/(shop-all-carpet-tile\.html|tile\.html|engineered-hardwood\.html)$ {

    if ($http_user_agent ~* "(AhrefsBot|SemrushBot|MJ12bot|DotBot|YandexBot|PetalBot|Baiduspider)") {
        return 403;
    }

}

This protects your most expensive pages without affecting real customers.

Test Your Configuration

Before restarting Nginx:

sudo nginx -t

If everything is valid:

sudo systemctl reload nginx

Never restart Nginx without testing the configuration.

Verify the Rules

You can test using curl:

curl -A "AhrefsBot" https://example.com

Expected response:

403 Forbidden

Now test Googlebot:

curl -A "Googlebot" https://example.com

The page should load normally.

Additional Best Practices

Blocking bots is only one part of website optimization.

For Magento stores, we also recommend:

Together, these improvements can significantly reduce server load while improving user experience.

Common Mistakes

Avoid these common issues:

Conclusion

Every website receives bot traffic, but not all bots deserve access to your server resources.

A properly configured Nginx server can block unwanted SEO crawlers, AI bots, and scraping tools before they reach your application. This reduces CPU usage, improves website speed, and keeps your Magento store responsive for real customers.

If you’re experiencing high server load, slow Magento performance, or excessive bot traffic, implementing these Nginx rules is an effective first step toward better performance and security.

Need Help Optimizing Your Magento Store?

At EcomPlugins, we specialize in Magento and Adobe Commerce performance optimization, server tuning, shipping solutions, and custom development.

Whether you’re dealing with bad bots, slow category pages, Elasticsearch issues, or high CPU usage, our team can help.

Contact us today:
https://www.ecomplugins.com/index.php?route=information/contact

Exit mobile version