When Bot Traffic Becomes an Infrastructure Problem

Bot traffic used to be background noise for us.

It accounted for perhaps 10% to 15% of our requests. It was something we monitored, occasionally adjusted a firewall rule around, and otherwise treated as a normal part of operating a public e-commerce platform.

Then it crossed 50%.

At times, it climbed closer to 80%.

That completely changes the infrastructure problem.

When automated traffic becomes the majority of your workload, you are no longer sizing infrastructure primarily for customers. You are sizing it for crawlers, scrapers, shopping integrations, AI agents, vulnerability scanners, SEO tools, advertising platforms, unknown automation, and whatever distributed botnet decided to visit that afternoon.

Recent industry data suggests this is no longer unusual. The 2026 Thales Bad Bot Report found that automated activity represented approximately 53% of observed internet traffic during 2025, while bad bots alone accounted for roughly 40%.

The difficult part is not recognizing that bots exist.

The difficult part is answering five much harder questions:

1. Which bots provide business value?
2. Which bots are simply consuming capacity?
3. Where should those decisions be enforced?
4. How much false-positive risk can the business tolerate?
5. At what point does serving automated traffic cost more than the traffic is worth?

For a modern e-commerce architecture, those questions cannot be answered with a single “block bots” switch.

Our frontend runs on Next.js, while Magento 2 provides the backend commerce APIs. Traffic passes through Cloudflare, the Next.js frontend runs on AWS Fargate, Magento runs on EC2, and both ultimately place pressure on RDS and several other supporting services.

That architecture creates an additional complication: Cloudflare does not automatically understand the business relationship between our Next.js servers and Magento.

From the edge, a large volume of repeated requests from a relatively small set of server-side clients can look exactly like automated traffic—because technically, it is automated traffic.

The request may be legitimate server-to-server communication that powers the storefront, but it can still resemble a bot.

That is where bot management stops being a security checkbox and becomes an architectural discipline.

Bots Do Not Create One Problem

#

It is tempting to think of bot traffic as a bandwidth problem.

In practice, bandwidth is often the least interesting part of it.

A bot request can pass through several layers of infrastructure:

1
2
3
4
5
6
7
8
9

Cloudflare
    ↓
AWS Application Load Balancer
    ↓
Next.js on Fargate
    ↓
Magento GraphQL on EC2
    ↓
Redis / OpenSearch / RDS / external services

A single request might trigger:

• A new Fargate task because the frontend crossed a scaling threshold.
• Server-side rendering in Next.js.
• One or more Magento GraphQL requests.
• PHP execution on an EC2 instance.
• Redis session or cache lookups.
• OpenSearch queries.
• Multiple RDS queries.
• Logging and tracing across several services.
• Calls to search, recommendation, inventory, tax, shipping, or advertising systems.

The bot does not pay for any of that.

We do.

This is why request counts alone are not enough. Ten thousand requests to cached static assets are not equivalent to ten thousand uncached product-detail requests that invoke GraphQL, PHP, OpenSearch, and RDS.

The meaningful metric is not:

1

Requests per second

It is closer to:

1

Infrastructure cost per request class

That means I need to understand not only who is requesting a page, but what that request causes the system to do.

Autoscaling Protects Availability, Not the Budget

#

Autoscaling is an essential part of our architecture.

It allows the frontend and backend to absorb legitimate changes in demand. It helps us survive promotions, seasonal events, marketing campaigns, unexpected customer interest, and ordinary variance in traffic.

But autoscaling has a dangerous characteristic:

It will scale for bad demand just as willingly as it scales for good demand.

Fargate does not know whether CPU utilization came from a customer trying to purchase a suspension kit or a scraper requesting thousands of product combinations.

An EC2 Auto Scaling Group does not know whether Magento is processing customer GraphQL requests or a poorly behaved crawler repeatedly requesting expensive category trees.

RDS does not know whether a query eventually contributed to revenue.

From the infrastructure’s perspective, work is work.

This makes autoscaling both the safety net and the mechanism through which bot traffic becomes expensive.

A sufficiently aggressive crawler can produce a cycle like this:

1. Automated traffic spikes.
2. Application latency increases.
3. CPU, request count, or response-time alarms cross a threshold.
4. Fargate or EC2 adds capacity.
5. RDS connections and query volume increase.
6. The crawler consumes the additional capacity.
7. Scaling continues until the traffic declines or a configured maximum is reached.

The platform remains online, which is good.

The cloud bill increases, which is not.

Even worse, reactive scaling can lag behind a bot spike. That leaves a period in which legitimate customers compete with automation while new capacity starts.

Autoscaling therefore needs to be treated as the final availability control, not the primary bot-management strategy.

The First Mistake: Treating Every Bot the Same

#

Some automation is directly tied to revenue.

Examples include:

• Googlebot crawling pages for organic search.
• Google product and advertising crawlers validating landing pages.
• Meta systems checking products or ad destinations.
• Amazon marketplace or integration traffic.
• Bing and other search engines.
• Uptime and availability monitors.
• Approved accessibility or compliance tools.
• Internal services and trusted integration partners.
• Customer-authorized AI agents or shopping assistants.

Other automation may provide indirect value:

• SEO analysis tools.
• Price-comparison services.
• Affiliate systems.
• Brand-monitoring platforms.
• Emerging AI discovery engines.

Then there is traffic that may provide little or no value:

• Unidentified product scrapers.
• Aggressive price-monitoring crawlers.
• Credential-stuffing tools.
• Vulnerability scanners.
• Inventory hoarders.
• Card-testing bots.
• Search and filter enumeration.
• Automated requests that ignore robots.txt.
• AI training crawlers with no meaningful referral value.
• Distributed low-rate scrapers designed to avoid volumetric detection.

Cloudflare maintains a verified-bot classification for known, transparent services such as search crawlers and monitoring systems. Its Bot Management product also assigns requests a bot score from 1 to 99, with lower scores indicating a greater likelihood of automated traffic.

Those signals are useful, but they are not a complete business policy.

A bot can be technically verified and still be too expensive.

An unknown bot can occasionally be valuable.

A request can also be correctly identified as automated but still belong to our own application.

The goal is not to divide all requests into “bot” and “human.”

I find it more useful to classify them into four groups:

This distinction is the foundation of everything else.

Our Architectural Complication: Next.js Is an Automated Client

#

Our storefront is not a browser directly communicating with Magento for every operation.

Next.js performs server-side work.

Depending on the route and rendering strategy, the frontend may request product, category, pricing, inventory, navigation, or customer-related data from Magento.

From Cloudflare’s perspective, these requests can exhibit several characteristics commonly associated with bots:

• High request volume.
• Repeated request patterns.
• A limited range of source addresses.
• Similar or identical headers.
• No browser interaction.
• No JavaScript challenge support.
• Machine-consistent timing.
• Repeated GraphQL operations.
• A user agent associated with a server-side HTTP client.

Therefore, a rule such as “challenge all low bot scores on api.example.com” can become catastrophic.

A browser may be able to complete a managed challenge.

A Next.js server cannot interact with a challenge page like a human.

The result can be intermittent API failures, failed page rendering, broken product pages, or an apparent Magento outage caused by the security layer in front of it.

The important architectural lesson is:

Trusted server-to-server traffic must be identified cryptographically or through a similarly strong identity mechanism. It should not be trusted merely because of its IP address or user-agent string.

Split the Problem Between www and api

#

Trying to use one bot policy across both the public storefront and the backend API is a mistake.

The two hostnames serve different purposes and have different legitimate traffic patterns.

The www Hostname

#

The public storefront handles:

• Human browsers.
• Search crawlers.
• Advertising crawlers.
• AI retrieval agents.
• SEO tools.
• Scrapers.
• Headless browsers.
• Cacheable HTML and static resources.
• Dynamic account and checkout flows.

At www, browser-oriented controls are appropriate.

We can use:

• Managed challenges.
• JavaScript detections.
• Browser integrity signals.
• Cookie presence.
• Request behavior.
• Bot scores.
• Verified-bot classifications.
• Path-specific rate limits.
• Cache policy.
• Robots directives.

The api Hostname

#

The API handles a very different mix:

• Next.js server-side requests.
• Browser-originated GraphQL traffic, where applicable.
• Mobile applications.
• Internal services.
• Approved integrations.
• Third-party systems.
• Unauthenticated public API requests.
• Scrapers calling Magento directly.

At api, identity and request shape are more valuable than browser behavior.

We should use controls such as:

• Service authentication.
• Mutual TLS.
• Signed requests.
• JWT validation.
• API schema validation.
• Operation allowlists.
• Persisted GraphQL operations.
• Per-client quotas.
• Per-operation rate limits.
• Request-size limits.
• GraphQL depth and complexity limits.

Cloudflare explicitly supports API protections including schema validation, per-endpoint rate limiting, JWT validation, sequence analysis, and mutual TLS.

The key is not merely having two sets of rules.

It is establishing two different trust models.

Give Next.js a Real Service Identity

#

The cleanest solution to the Next.js classification problem is to make Next.js identifiable as a trusted service.

There are several ways to do this.

Option 1: Cloudflare Access Service Tokens

#

Cloudflare Access service tokens are designed for automated systems that need to authenticate to protected applications. Requests include a client ID and client secret in dedicated headers, and Cloudflare validates them before granting access.

A simplified Next.js server-side request could look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

interface MagentoRequestOptions {
  query: string;
  variables?: Record<string, unknown>;
}

export async function requestMagento<T>({
  query,
  variables = {},
}: MagentoRequestOptions): Promise<T> {
  const endpoint = process.env.MAGENTO_GRAPHQL_URL;
  const clientId = process.env.CF_ACCESS_CLIENT_ID;
  const clientSecret = process.env.CF_ACCESS_CLIENT_SECRET;

  if (!endpoint || !clientId || !clientSecret) {
    throw new Error("Magento service authentication is not configured");
  }

  const response = await fetch(endpoint, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",

      // Cloudflare Access service identity.
      "CF-Access-Client-Id": clientId,
      "CF-Access-Client-Secret": clientSecret,

      // Useful for application logging, but not sufficient as authentication.
      "X-Calling-Service": "nextjs-storefront",
    },
    body: JSON.stringify({
      query,
      variables,
    }),
    cache: "no-store",
  });

  if (!response.ok) {
    throw new Error(
      `Magento request failed with status ${response.status}`
    );
  }

  return (await response.json()) as T;
}

The secret values should come from a managed secret store such as AWS Secrets Manager or Systems Manager Parameter Store, not from source control or a baked container image.

Cloudflare can then apply a narrowly scoped skip rule for requests that have successfully passed the service-authentication policy.

Cloudflare’s WAF supports a skip action that can exempt trusted traffic from selected features, including rate-limiting rules, managed rules, and bot protections.

The word selected matters.

I would not create a rule that bypasses every security control for every authenticated request.

Instead, I would skip only controls that are incompatible with trusted server traffic, while retaining protections that still provide value.

For example:

1
2
3
4
5
6
7

Authenticated Next.js service request
    ├── Skip browser challenge
    ├── Skip generic bot mitigation
    ├── Apply a dedicated Next.js rate limit
    ├── Continue schema validation
    ├── Continue request-size enforcement
    └── Continue logging and anomaly detection

Trusted does not mean unlimited.

Option 2: Mutual TLS

#

Mutual TLS gives the calling service a client certificate that the receiving edge validates.

Cloudflare API Shield supports mTLS for API protection, and Cloudflare also supports Authenticated Origin Pulls to ensure that origin requests came through Cloudflare. These solve related but different trust problems. API mTLS authenticates clients calling the API, while Authenticated Origin Pulls authenticate Cloudflare to the origin.

For high-value server-to-server paths, mTLS has several advantages:

• The identity is cryptographic.
• It is not based on a spoofable user agent.
• It is not dependent on stable source IP addresses.
• Certificates can be rotated.
• Different services can receive separate identities.
• Policies can distinguish Next.js from other integrations.

The operational cost is certificate issuance, distribution, storage, renewal, and revocation.

That overhead can be worthwhile when the API is important enough.

Option 3: Signed Requests

#

Another model is to have Next.js sign requests using a secret or asymmetric key.

A basic HMAC design might include:

1
2
3
4

X-Service-Id: nextjs-storefront
X-Request-Timestamp: 1785451200
X-Request-Nonce: 834db8c0-...
X-Request-Signature: sha256=...

The signature should cover at least:

1
2
3
4
5
6

HTTP method
hostname
path
timestamp
nonce
body hash

Cloudflare Workers or the application can validate the signature before allowing the request.

The system also needs:

• A short timestamp tolerance.
• Nonce tracking to prevent replay.
• Key rotation.
• Constant-time signature comparison.
• Separate credentials per service.
• Logging that never exposes secrets.

This is more custom work than using Access or mTLS, but it provides complete control.

What I Would Not Use as the Primary Identity

#

I would not rely exclusively on:

1

X-Internal-Request: true

Any external client can send that header unless Cloudflare strips incoming copies and injects a trusted replacement.

I would also avoid treating a fixed user-agent string as authentication:

1

User-Agent: RoughCountry-NextJS

That is useful for logs and observability, but it is trivial to copy.

Source-IP allowlisting can be useful as one signal, particularly when requests leave through controlled NAT gateways with static addresses. However, it becomes operationally fragile and should not be the only identity control.

The strongest design combines multiple attributes:

1
2
3
4
5

Valid service credential
AND expected hostname
AND approved operation
AND reasonable rate
AND valid request structure

Do Not Give Trusted Services an Infinite Budget

#

Once Next.js is authenticated, the easiest mistake is allowing it to bypass all rate limits.

That trades one failure mode for another.

A frontend bug, cache miss storm, accidental recursion, malformed deployment, or compromised credential could cause the trusted service itself to overload Magento.

I want a dedicated rate-limit budget for Next.js based on its expected behavior.

That rate limit should be much higher than the public API limit, but it should still exist.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

Public unauthenticated API client:
    60 requests per minute per session or client identifier

Authenticated partner:
    Contract-specific quota

Next.js storefront:
    High service-wide threshold
    + per-operation ceilings
    + concurrency protection

Administrative service:
    Very low request volume
    + strict operation allowlist

Cloudflare rate-limiting rules can match specific expressions and apply thresholds to selected request groups. Bot scores can also be used as part of rate-limiting criteria.

The rate limit should protect the actual constrained resource.

If Magento can safely process 500 instances of one cheap operation per second but only 40 concurrent requests for a complex category operation, one global requests-per-second threshold is too blunt.

GraphQL Requires Operation-Level Protection

#

GraphQL makes bot management more complicated because many workloads use the same path:

1

POST /graphql

A path-based rate limit sees every request as identical.

The database does not.

These two GraphQL operations can have radically different costs:

1
2
3
4
5
6

query StoreConfig {
  storeConfig {
    store_name
    base_currency_code
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

query LargeCategoryPage {
  categoryList(filters: { ids: { eq: "42" } }) {
    products(pageSize: 200) {
      items {
        sku
        name
        description {
          html
        }
        media_gallery {
          url
        }
        ... dozensOfAdditionalFields
      }
      aggregations {
        ... everyAvailableFilter
      }
    }
  }
}

They both count as one HTTP request.

They do not represent one unit of work.

For GraphQL, I would consider:

• Named operations.
• Persisted operations.
• An operation allowlist for storefront traffic.
• Maximum query depth.
• Maximum query complexity.
• Maximum aliases.
• Maximum batch size.
• Variable and page-size limits.
• Per-operation rate limits.
• Separate limits for anonymous and authenticated customers.
• Rejection of anonymous introspection in production.
• Response caching where business rules allow it.

A request should ideally carry an operation identifier that Cloudflare can inspect:

1

X-GraphQL-Operation: ProductDetail

Again, the header should not be blindly trusted from the public internet.

A Cloudflare Worker, API gateway, GraphQL proxy, or application middleware can parse the request, validate the operation, and attach normalized metadata for downstream policy.

A persisted-operation request is even easier to govern:

1
2
3
4
5
6

{
  "operationId": "product-detail-v3",
  "variables": {
    "urlKey": "example-product"
  }
}

Now the edge can make a policy decision based on a stable identifier rather than parsing an arbitrary GraphQL document.

That allows rules such as:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

product-detail-v3:
    high permitted volume
    cacheable under defined conditions

category-page-v5:
    moderate permitted volume
    strict page-size limit

customer-orders-v2:
    authentication required
    low per-customer rate limit

unknown operation:
    reject

This turns GraphQL from an open-ended execution endpoint into a controlled application interface.

Rate Limit by Behavior, Not Just IP Address

#

IP-based rate limiting is still useful, but modern bot traffic is often distributed.

A scraper can rotate through residential proxies, cloud providers, IPv6 addresses, mobile networks, or compromised devices.

If each address sends only a small number of requests, no individual IP crosses the threshold.

The combined workload can still overwhelm the platform.

Better policies can incorporate:

• Bot score.
• Verified-bot status.
• Session or cookie identifier.
• API credential.
• JWT subject or account.
• Autonomous system number.
• Country or region.
• User agent.
• TLS and client fingerprint.
• Request path.
• GraphQL operation.
• Query complexity.
• Cache status.
• Request sequence.
• Error rate.
• Ratio of page views to asset requests.
• Navigation speed.
• Product enumeration behavior.

A normal shopper may request:

1
2
3
4
5
6
7

Homepage
Category page
Product page
Images and JavaScript
Add to cart
Cart
Checkout

A scraper might request:

1
2
3
4
5
6

Product 1
Product 2
Product 3
Product 4
Product 5
...

with no assets, no cart activity, no meaningful pauses, and sequential pagination.

Each request may look valid in isolation.

The sequence reveals the intent.

Cloudflare’s API Shield supports sequence analytics and sequence mitigation for authenticated API clients, allowing policies to reason about the order of endpoint calls rather than treating each request independently.

For our use case, I would also look for behaviors such as:

• Iterating every category page.
• Requesting maximum page sizes repeatedly.
• Walking product IDs sequentially.
• Requesting every vehicle fitment combination.
• Repeating expensive searches with minor variations.
• Fetching product data without associated static assets.
• Ignoring cache validators.
• Generating a high proportion of cache misses.
• Requesting out-of-stock or obsolete products at scale.
• Repeating malformed GraphQL operations.
• Producing large numbers of 404s or validation errors.

These are stronger signals than “this IP made 100 requests.”

Make Bots Pay the Cheapest Possible Cost

#

Blocking is not the only way to reduce bot cost.

Another strategy is to make automated requests consume less infrastructure.

The ideal request path for a public crawler is:

1
2
3
4
5

Bot
    ↓
Cloudflare cache
    ↓
Response

The expensive path is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

Bot
    ↓
Cloudflare
    ↓
ALB
    ↓
Fargate
    ↓
Magento EC2
    ↓
RDS / Redis / OpenSearch
    ↓
Response

Every request that terminates at the edge is a request that does not trigger application compute or database work.

Cache Public Product and Category Responses

#

Public, anonymous product data is an obvious candidate for carefully designed caching.

The cache key must account for business dimensions such as:

• Country.
• Currency.
• Store view.
• Customer group, where applicable.
• Vehicle selection.
• Inventory region.
• Relevant cookies.
• Query variables.
• Authorization state.

Incorrect caching can be worse than no caching, especially when pricing or availability differs between customers or regions.

But the existence of complexity is not an argument for caching nothing.

It is an argument for defining explicit cache-safe operations.

Use Stale Responses During Spikes

#

Where appropriate, stale-while-revalidate or stale-if-error behavior can protect the origin during sudden demand.

A product description that is several minutes old may be perfectly acceptable.

A cart total or inventory commitment is not.

The distinction must be made by operation.

Precompute Expensive Public Data

#

If crawlers repeatedly request the same derived information, it may be cheaper to publish a static or precomputed representation.

Examples include:

• Product feeds.
• Sitemaps.
• Category indexes.
• Structured-data endpoints.
• Public inventory summaries.
• Merchant feeds.
• Dedicated crawler-facing exports.

This is especially important for business-critical partners.

If Google, Meta, Amazon, affiliates, or advertising systems need product information, forcing them to crawl the customer storefront may be the least efficient integration model.

A dedicated feed gives us:

• A predictable format.
• A predictable refresh schedule.
• Lower infrastructure cost.
• Better validation.
• Easier monitoring.
• Less dependence on rendering.
• Fewer opportunities for bot misclassification.

Degrade Before Blocking

#

For ambiguous traffic, a degraded response can be more appropriate than a hard block.

Possible approaches include:

• Smaller page sizes.
• Fewer product attributes.
• No personalization.
• No expensive recommendations.
• Cached content only.
• Lower request priority.
• Reduced search functionality.
• A 429 Too Many Requests response with a retry interval.
• A dedicated machine-readable feed.

The goal is to preserve useful discovery while preventing unknown automation from receiving the most expensive version of every page.

robots.txt Helps, but It Is Not Enforcement

#

A robots.txt file tells compliant crawlers which URLs they may access and is commonly used to reduce unnecessary crawling. Google explicitly states that it uses robots.txt primarily to manage crawler access and avoid overloading sites.

For example:

1
2
3
4
5
6
7

User-agent: *
Disallow: /checkout/
Disallow: /customer/
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?pageSize=200
Disallow: /*?vehicle=

The exact rules would need careful testing because overly broad parameter rules can hide legitimate pages from search engines.

I would also use separate directives for crawlers where their identity and behavior are known.

However, robots.txt is a request, not an access-control system.

Good crawlers generally follow it.

Bad crawlers generally do whatever they want.

It should be used to reduce unnecessary legitimate crawling, not as the primary defense against abusive traffic.

Verified Does Not Mean Unrestricted

#

Search and advertising crawlers may be critical to the business.

Blocking them can result in:

• Pages disappearing from search.
• Product feeds failing validation.
• Advertising destinations being rejected.
• Products becoming ineligible.
• Merchant listings being suspended or disabled.
• SEO visibility declining.
• New pages taking longer to appear.
• Stale pricing or availability being shown elsewhere.

Google publishes lists of its common crawlers and notes that its automatic crawlers obey robots.txt.

Cloudflare also allows policies to distinguish verified bots through fields such as cf.client.bot or its newer verified-bot classifications.

I would generally allow verified business-critical bots, but I would still monitor:

• Request volume.
• Paths requested.
• Cache-hit ratio.
• Origin requests.
• Response status.
• RDS impact.
• Crawl timing.
• Duplicate fetching.
• Query-string explosion.

If a legitimate crawler is creating excessive load, the first response should usually be optimization rather than blocking:

1. Improve caching.
2. Remove crawler traps.
3. Correct canonical URLs.
4. Restrict useless parameters.
5. Improve sitemaps.
6. Provide feeds.
7. Return correct cache headers.
8. Remove infinite navigational spaces.
9. Contact the platform through its supported channels.
10. Apply a narrowly scoped limit only when necessary.

Bot Traps Hidden in E-Commerce Sites

#

E-commerce applications can accidentally create nearly infinite crawl spaces.

Common examples include:

• Layered navigation.
• Sort parameters.
• Page-size parameters.
• Pagination.
• Vehicle selectors.
• Faceted search.
• Internal search pages.
• Session identifiers.
• Tracking parameters.
• Currency selectors.
• Store selectors.
• Product comparison pages.
• Wish-list URLs.
• Duplicate category paths.
• Filter combinations.
• Calendar-like date navigation.

Consider a category with these dimensions:

1
2
3
4
5
6
7

10 brands
8 price ranges
12 vehicle years
20 makes
100 models
5 sort orders
4 page sizes

The theoretical URL space becomes enormous before pagination is even considered.

A crawler does not need malicious intent to overload the platform. It only needs to discover enough combinations.

This is why bot management is partly an SEO and application-design problem.

Useful controls include:

• Canonical URLs.
• Parameter normalization.
• Redirecting equivalent URLs.
• Preventing indexing of low-value combinations.
• Excluding internal search pages.
• Limiting filter combinations.
• Avoiding session identifiers in URLs.
• Returning consistent 404 and 410 responses.
• Ensuring expired pages do not remain soft 404s.
• Generating focused sitemaps.
• Removing links to nonsensical combinations.
• Capping page size at the application layer.

The cheapest request is often the one the crawler never discovers.

The Cost Model: When Is Blocking Cheaper?

#

The question I ultimately care about is not:

Are bots increasing our AWS bill?

They clearly are.

The better question is:

Does the expected value of allowing this traffic exceed the full cost of serving and governing it?

I would model the cost of a traffic class as:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

Total Bot Cost
=
Edge Cost
+ Load Balancer Cost
+ Frontend Compute Cost
+ Backend Compute Cost
+ Database Cost
+ Cache/Search Cost
+ Logging and Observability Cost
+ Third-Party API Cost
+ Scaling Headroom Cost
+ Engineering Labor
+ Incident Risk
+ Opportunity Cost

The opportunity cost matters because bot traffic can reduce customer performance even when the website remains technically available.

The value side might be:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

Expected Bot Value
=
Attributed Revenue
+ Organic Search Value
+ Advertising Eligibility
+ Marketplace Eligibility
+ Referral Value
+ Monitoring Value
+ Integration Value
+ Strategic Discovery Value

Then:

1
2
3
4
5

Net Bot Value
=
Expected Bot Value
- Total Bot Cost
- Risk-Adjusted False-Positive Cost

The false-positive term is critical.

Blocking an unidentified scraper may have almost no business downside.

Blocking a product-validation crawler might disable a major advertising channel.

The cost of a mistake can be much larger than the infrastructure savings.

A More Practical Decision Table

#

The goal is not to produce a single universal threshold.

It is to create a threshold for each traffic class.

Measuring Cost per Traffic Class

#

This model only works if traffic can be connected to infrastructure consumption.

I would build observability around dimensions such as:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

hostname
path
GraphQL operation
bot classification
bot score range
verified-bot category
cache status
response status
calling service
country
ASN
user agent family
rate-limit action
WAF action

Then correlate those dimensions with:

• Fargate task count.
• Fargate CPU and memory.
• EC2 instance count.
• PHP-FPM utilization.
• Magento response time.
• RDS CPU.
• RDS connections.
• Database load by query.
• Read and write IOPS.
• Redis operations.
• OpenSearch CPU and query latency.
• ALB processed bytes and request count.
• Cloudflare cache-hit ratio.
• Origin request count.
• Error rate.
• Customer conversion rate.
• Advertising and organic referral traffic.

I would want a dashboard that answers:

1

Which automated clients caused the most origin requests?

1

Which automated clients caused the most database time?

1

Which GraphQL operations are most expensive?

1

What percentage of verified-bot traffic is served from cache?

1

How many Fargate task-hours were associated with low-value automation?

1

Did bot traffic increase customer latency or abandonment?

Without this correlation, bot management becomes rule-writing based on intuition.

A Useful Approximation for Marginal Bot Cost

#

Perfect cost attribution may not be possible.

A practical approximation is still better than none.

For a given time window:

1
2
3
4
5

Marginal Infrastructure Cost
=
Observed Cost
-
Estimated Human-Traffic Baseline Cost

Then estimate:

1
2
3
4
5
6

Cost per 1,000 Origin Bot Requests
=
Marginal Infrastructure Cost
/
Origin Bot Requests
× 1,000

This will not be exact because human and bot workloads share resources.

A more accurate model can use regression or controlled experiments:

• Compare similar periods with different bot ratios.
• Challenge or limit one traffic class for a short period.
• Measure the change in origin requests.
• Measure the change in Fargate, EC2, and RDS utilization.
• Control for human sessions, promotions, and revenue.
• Calculate the avoided cost.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

Before rule:
    4,000 origin requests/second
    180 Fargate tasks
    14 Magento EC2 instances
    70% RDS CPU

After constraining one scraper class:
    2,900 origin requests/second
    125 Fargate tasks
    10 Magento EC2 instances
    48% RDS CPU

That provides a much stronger basis for policy than the raw number of blocked requests.

Cost-Based Adaptive Mitigation

#

Static thresholds are simple, but traffic and infrastructure conditions change.

A more advanced model would adjust mitigation based on platform pressure.

Under normal conditions:

1
2

Unknown automation:
    Allow at a constrained rate

When RDS load or origin latency rises:

1
2
3
4

Unknown automation:
    Serve cached content only
    Reduce quotas
    Challenge more aggressively

During severe pressure:

1
2
3
4

Unknown automation:
    Block expensive operations
    Preserve verified and authenticated traffic
    Reserve capacity for customer checkout

This is similar to load shedding.

Not all traffic deserves equal priority when the system is constrained.

A simple priority model might be:

During a resource emergency, protections should become stricter from the bottom upward.

I would not allow anonymous product scraping to compete equally with checkout traffic for the same database capacity.

Preserve Capacity for Revenue-Producing Paths

#

Infrastructure limits should reflect business priority.

For example, Magento might have separate protections for:

1
2
3
4
5
6
7

/checkout
/customer
/cart
/graphql product operations
/graphql category operations
/graphql search operations
/integration endpoints

At the application layer, we can also use:

• Separate worker pools.
• Separate concurrency limits.
• Query timeouts.
• Database resource governance.
• Circuit breakers.
• Bulkheads.
• Queues for noninteractive work.
• Read replicas for eligible read traffic.
• Cached fallback data.
• Per-operation database timeouts.

A bot should not be able to exhaust all PHP-FPM workers with expensive catalog requests while customers are trying to place orders.

Likewise, one GraphQL operation should not be able to consume the entire RDS connection pool.

Blocking Is Not Free

#

Blocking traffic creates its own costs:

• Cloudflare configuration and licensing.
• Engineering time.
• Rule testing.
• Incident investigation.
• False-positive analysis.
• SEO monitoring.
• Advertising-feed monitoring.
• Customer-service escalations.
• Integration support.
• Credential and certificate rotation.
• Dashboard development.
• Ongoing policy maintenance.

Bot behavior also changes.

A rule that works today may be ineffective next month.

Static IP blocks decay quickly. User agents change. Distributed bots rotate infrastructure. Headless browsers gain more realistic behavior. New AI agents appear. Legitimate platforms add crawlers that are not yet understood by internal teams.

Therefore, the business case for bot protection should include human labor.

A $2,000 monthly infrastructure saving is not necessarily valuable if it requires $5,000 worth of monthly engineering attention and creates continual revenue risk.

On the other hand, spending engineering time to build durable service identity, operation controls, and cost attribution may continue paying off for years.

The goal should be to invest in controls that reduce recurring decision-making.

The Layered Architecture I Would Build

#

I would approach the problem in layers.

Layer 1: Classify

#

At Cloudflare:

• Separate verified bots.
• Record bot scores.
• Identify known partners.
• Identify trusted internal services.
• Group unknown automation.
• Track known abusive patterns.

Do not block broadly during the first phase.

Log and measure.

Layer 2: Authenticate Internal Traffic

#

For Next.js-to-Magento requests:

• Use Cloudflare Access service tokens, mTLS, or signed requests.
• Give each service a distinct identity.
• Rotate credentials.
• Store credentials in AWS Secrets Manager.
• Strip spoofable identity headers from public requests.
• Apply dedicated quotas.
• Retain schema and request validation.

Layer 3: Separate www and api Policies

#

For www:

• Use browser-oriented bot controls.
• Challenge suspicious interactive traffic.
• Protect login, cart, and checkout.
• Optimize cache behavior.
• Control crawl spaces.
• Allow business-critical crawlers.

For api:

• Require identity where possible.
• Validate schemas and tokens.
• Enforce GraphQL operation controls.
• Rate limit by operation and client.
• Reject oversized or excessively complex requests.
• Block direct anonymous access to operations intended only for Next.js.

Layer 4: Reduce Origin Work

#

• Cache safe public operations.
• Use persisted GraphQL operations.
• Normalize cache keys.
• Precompute feeds.
• Serve stale content where safe.
• Cap pagination.
• Remove unnecessary fields.
• Avoid invoking Magento for data Next.js can cache.

Layer 5: Apply Graduated Mitigation

#

Use a progression such as:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

Log
    ↓
Lower quota
    ↓
Serve cached/degraded response
    ↓
Managed challenge
    ↓
Temporary block
    ↓
Permanent policy

Do not jump immediately from unrestricted access to permanent blocking unless the traffic is clearly hostile.

Layer 6: Protect Critical Capacity

#

• Reserve checkout capacity.
• Cap expensive catalog operations.
• Apply database timeouts.
• Separate worker pools where practical.
• Use circuit breakers.
• Trigger stricter bot controls when origin health degrades.

Layer 7: Measure Business Outcomes

#

For every major policy change, monitor:

• Organic sessions.
• Indexed pages.
• Merchant Center or advertising errors.
• Product approvals.
• Marketplace health.
• Referral traffic.
• Conversion.
• Origin requests.
• Infrastructure cost.
• RDS utilization.
• Customer latency.
• False positives.

A bot rule is not successful merely because it blocked millions of requests.

It is successful when it reduces cost or risk without damaging the business.

Example Cloudflare Policy Structure

#

The exact expressions depend on the Cloudflare plan and enabled products, but the conceptual ordering should look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Rule 1: Known malicious indicators
Action: Block

Rule 2: Valid Next.js service identity on api.example.com
Action:
    Skip browser-oriented bot challenge
    Continue API validation
    Continue dedicated service rate limits

Rule 3: Business-critical verified bots on approved www paths
Action:
    Allow or skip selected bot controls
    Continue monitoring and crawler-specific quotas

Rule 4: Direct public access to internal-only GraphQL operations
Action: Block

Rule 5: Suspicious bot score on login, cart, or checkout
Action: Managed Challenge or Block

Rule 6: Unknown automation requesting expensive catalog operations
Action: Rate Limit

Rule 7: Unknown automation with high 404 or validation-error rate
Action: Temporary Block

Rule 8: General public traffic
Action: Standard WAF and rate-limiting policy

Cloudflare recommends considering built-in bot controls before layering custom rules, and its documentation provides examples that exclude verified bots while challenging low-scoring traffic.

I would manage these rules as code wherever possible.

A Terraform-managed policy provides:

• Change review.
• Version history.
• Repeatability.
• Staging deployment.
• Rollback.
• Documentation next to the rule.
• Reduced dashboard drift.

Every rule should include a reason.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

description = <<-EOT
Rate-limit anonymous high-cost catalog GraphQL operations.

Purpose:
- Protect Magento PHP-FPM and RDS from distributed scraping.
- Excludes authenticated Next.js service traffic.
- Excludes approved verified commerce crawlers.
- Initially deployed in log mode on 2026-08-01.

Rollback:
- Disable this rule if product-page error rate or advertising
  validation failures increase.
EOT

A firewall rule without business context eventually becomes a mystery that no one is comfortable removing.

Safe Deployment Process

#

Bot rules should be deployed like application changes.

1. Start in Observation Mode

#

Record what the rule would match.

Do not immediately block.

Review:

• Top user agents.
• Source networks.
• Paths.
• Countries.
• Bot classifications.
• Referrers.
• Response codes.
• Known integrations.
• Search and advertising crawlers.

2. Test Against Business-Critical Flows

#

At minimum:

• Homepage.
• Category page.
• Product page.
• Search.
• Vehicle selection.
• Login.
• Cart.
• Checkout.
• Google and advertising validation.
• Next.js server rendering.
• Internal API calls.
• Mobile applications.
• Third-party integrations.

3. Use a Narrow Initial Action

#

Begin with:

• A high threshold.
• A single expensive operation.
• A short mitigation period.
• A small traffic segment.
• A managed challenge instead of a block, where appropriate.

4. Monitor Revenue and Infrastructure Together

#

Do not look only at firewall events.

Compare:

• Infrastructure reduction.
• Conversion.
• Error rate.
• Organic traffic.
• Advertising status.
• Product eligibility.
• Page latency.
• Customer-support reports.

5. Maintain a Kill Switch

#

Every significant mitigation should be quickly reversible.

A security control that cannot be disabled during an incident becomes another source of operational risk.

Gotchas and Edge Cases

#

Cloudflare May Correctly Call Next.js a Bot

#

From a behavioral perspective, it is automated traffic.

The mistake is not Cloudflare detecting automation.

The mistake is relying on behavioral detection to determine whether a first-party service should be trusted.

Give the service an identity.

Verified Crawlers Can Still Be Expensive

#

Verification establishes identity, not economic value or acceptable volume.

Monitor and optimize even trusted crawlers.

A Cache Hit Is Not Always Cheap Enough

#

Cloudflare may serve the response without reaching AWS, but very high edge traffic can still affect plan usage, logging volume, analytics, or contracted limits.

Cost analysis should include the edge.

Rate Limiting by IP Can Punish Shared Networks

#

Corporate offices, schools, mobile carriers, and carrier-grade NAT can place many legitimate customers behind one public address.

IP should be one signal, not always the entire key.

Challenges Do Not Work for Every Legitimate Client

#

Server-side services, native applications, accessibility tools, advertising crawlers, and API integrations may be unable to complete browser challenges.

Use authentication or explicit policies for non-browser clients.

Allowlisting a Header Creates a Bypass

#

A custom header is not proof of identity unless the edge removes client-supplied versions and inserts or validates the trusted value itself.

GraphQL Batching Can Multiply Work

#

One HTTP request can contain multiple operations.

A rate limit based only on HTTP requests can dramatically underestimate application work.

Limit batch size or disable batching on expensive public paths.

Blocking Search Parameters Can Damage Discovery

#

Some filtered pages may have legitimate SEO value.

SEO, merchandising, advertising, and engineering need to agree on which URL spaces should remain crawlable.

Aggressive Logging Can Become Its Own Cost

#

Logging every blocked request with full request bodies can create substantial ingestion and retention costs.

Log enough to investigate and measure, but sample repetitive events and avoid recording sensitive content.

A Bot Spike Can Look Like Customer Growth

#

If dashboards do not separate human and automated traffic, teams may scale infrastructure or celebrate traffic increases that have no connection to revenue.

Traffic quality must be part of capacity planning.

The Bigger Capacity-Planning Lesson

#

Capacity planning used to be built primarily around customers:

• Normal traffic.
• Seasonal growth.
• Promotions.
• Failure headroom.
• Regional growth.
• Product launches.

That is no longer enough.

We now need separate demand models for:

1
2
3
4
5

Human demand
Trusted automated demand
Unknown automated demand
Hostile automated demand
Internal service demand

Human demand may be relatively predictable.

Automated demand can be abrupt, distributed, repetitive, and detached from business activity.

That makes a single requests-per-second forecast increasingly meaningless.

I would rather plan around:

• Expected human peak.
• Required business-critical crawler budget.
• Internal-service budget.
• Absorbable unknown-bot budget.
• Emergency hostile-traffic ceiling.
• Reserved capacity for checkout and order placement.

Autoscaling still matters.

But it should scale within a system that has already decided which work deserves to reach the origin.

Final Thoughts

#

The bot problem is not going away.

Search engines, advertising systems, marketplaces, monitoring tools, scrapers, and AI agents will continue generating a growing share of internet traffic.

The answer cannot be to block everything automated.

For an e-commerce business, that could be more expensive than the AWS bill it saves.

The answer also cannot be to let every crawler trigger Fargate tasks, EC2 instances, PHP workers, OpenSearch queries, and RDS load without restriction.

That is effectively allowing anonymous third parties to control the size of our infrastructure.

The architecture I want is more deliberate:

• Humans receive a fast and reliable storefront.
• Next.js has an authenticated identity when calling Magento.
• Business-critical crawlers receive controlled access.
• Trusted integrations have explicit quotas.
• Public GraphQL operations are bounded.
• Expensive requests are cached or precomputed.
• Unknown automation receives a limited budget.
• Abusive traffic is stopped before it reaches AWS.
• Checkout capacity is protected from catalog scraping.
• Every decision is tied to cost and business value.

The real question is not whether a request came from a bot.

The real question is:

Is this request valuable enough to justify the infrastructure, operational risk, and human effort required to serve it?

Once bot traffic becomes the majority of the workload, that question belongs in architecture reviews, cost planning, security policy, SEO strategy, and incident response.

It is no longer just a firewall problem.