# Handling bot traffic on government websites

 

 

Date: Thursday, March 19, 2026

 By  [Daniel Mundra](/team/daniel-mundra)

By Daniel Mundra, Associate Director of Drupal

Bot traffic has increased on the internet. These bots are both “good” bots (bots we want accessing our websites, like search index crawlers), as well as “bad” bots that don’t identify themselves and keep hitting our sites with multiple requests. These bots, whether unintentionally or intentionally, consume a lot of our resources and can cause the website to perform poorly for all users and increase infrastructure costs. We have seen these requests increase since the middle of 2024 and throughout 2025. Over the course of two years, our team has tackled a surge in bot requests and implemented multiple mitigation strategies to keep government websites up and performing well.

This is Part 1 of a two-part post. In this post, we will analyze bot traffic on a large government website, how we identify bots, and how we are currently restricting them. In our follow-up post, we will explore our work with stabilizing performance with a Drupal website hosted in an Amazon Web Services (AWS) cloud environment.

## Identifying suspicious bot traffic

Over the course of two years, we found that bot requests mostly concentrated on hitting all possible combinations of query and URL patterns that were mostly valid. Because these requests would come in at the same time and the combinations suggested that they were not human users, those requests were not always cached. That led to requiring more processing from the website, thus causing spikes in resource consumption. While requesting all combinations is not an issue unto itself; for example, part of search indexing is figuring out all parts of your website.

The issue was that the bots would create a large spike of these requests at the same time in a distributed manner. The “good” bots would identify themselves with a user agent when doing such a crawl and also usually used one IP address.

The bad bots would not identify themselves or use user agents with old versions of browsers and operating systems and also distributed the IPs (sometimes ranging in the hundreds of thousands of IPs). We also saw both a consistent surge of requests from specific regions and requesting the same page/search string over and over and over again. We believe that a lot of requests come from data centers hosted in specific countries that are likely to collect data for large language model training or related data harvesting (including China, Singapore, Brazil, Hong Kong, Indonesia, and Vietnam).

Here are some statistics of the bot requests we saw:

- We saw specific pages that provided search results with filter options receive 65% to 70% traffic just from Brazil for a period of a few days until we rate limited and blocked those requests.
- We saw one page get 252,000 requests in one hour during a particular incident.
- At the end of the year, we saw a daily average of around 26.6 million allowed queries/day from China over a 40 day period until we blocked it.

## Identifying bot traffic

As we have seen, most good bots will identify themselves with a user agent string, but for all other requests, we have to make assumptions based on the request itself, the available logs, and the tools at our disposal. Earlier in the year, one approach was using the IP and available lists of bad IPs. We collaborated with security team members who could look up those lists and analyze IPs to glean any useful information. We created a list of IPs we blocked and also used AWS available tools that would block bad IPs.

After we found that IP blocking was not enough, we switched to analyzing regions and requested URL frequency. Finally, towards the end of the year, we came across fingerprinting that provided another avenue to identify and block bots that used similar fingerprints.

All these approaches are a whack-a-mole problem, even if we blocked one IP or one region, requests would flood from another IP or region. Identifying bot requests is the ultimate challenge. It is the ongoing issue we have to be vigilant about and keep fine-tuning our approach.

## Restricting requests with rate limiting and blocking

Eventually, we needed to find ways to restrict requests. Throughout the year, we debated which restrictions were reasonable and would not affect regular users. Restrictions usually ranged from outright blocking requests to versions of [rate limiting](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/WAF-one-click-rate-limiting.html).

Rate limiting is when we limit the number of requests per time period based on a criterion (e.g., 500 requests in 5 minutes per IP). A few times in the last year, the websites faced bot attacks (large volumes of requests). In those cases, we settled on blocking the requests, especially blocking specific regions. We implemented this blocking using a web application firewall (WAF), specifically the AWS-managed one connected to CloudFront.

We found that when requests from China, Brazil, and Singapore would often flood the website, we had to block those regions. We focused our blocking on URLs that were at higher risk due to more cache misses, more query parameters, performance issues, or similar criteria. On those URLs, we also had aggressive rate limiting that used the IP address as the distinguishing factor.

We found that the requests came from so many IPs (millions) that rate limiting would not slow them down. The blocking only worked because we had good logs at all levels (CloudFront, WAF, and Drupal logs in CloudWatch) for our application, and a great team that analyzed them to identify where the requests were coming from and what they were requesting. Those logs would tell us the IP, the region, the pages requested, and information such as the referrer and user agent (though these didn’t give us much insight, since the referrer was usually blank and the user agent is also spoofable).

We recommend having good, easy-to-collect, and accessible logs. The downside of blocking is that it prevents legitimate users in those blocked regions from accessing the website. Towards the end of the year, we started looking at other ways to rate-limit and block requests, rather than the whack-a-mole approach of blocking specific regions.

## Restricting requests with challenges, CAPTCHAs, and fingerprints

After successfully stopping bot attacks with our regional blocking approach above, we found that bots from other regions were still hitting the websites. These bots would hit many URLs, including 404 responding ones. Even though these increased requests would not take down the website, we were being vigilant and wanted to prevent them from getting to that point. We also saw that increased website requests, even cached ones, would still increase the cost to serve them.

Earlier in the year, we implemented what is called a [challenge](https://docs.aws.amazon.com/waf/latest/developerguide/waf-captcha-and-challenge.html). A challenge is an action that runs silently on the client to verify that the user is legitimate by asking them to solve a problem and generate a token (see [Protect against bots with AWS WAF Challenge and CAPTCHA actions | Networking & Content Delivery](https://aws.amazon.com/blogs/networking-and-content-delivery/protect-against-bots-with-aws-waf-challenge-and-captcha-actions/)).

After implementing that challenge, we were able to block many bots that didn’t support JavaScript (a technology that all modern browsers support) or bots that were not able to solve the challenge, especially for the URLs we were particularly concerned about.

The solution: we just needed to add a little JavaScript code to our website, and AWS WAF handled the rest. We were worried this change would affect users, but it hasn’t had any negative impact, so the next step was to expand it to more URLs on the website, including ones that receive more traffic or are hot spots.

The challenge functionality also includes the option to use CAPTCHA, which presents a puzzle for the user to solve to prove they are human. This one is still being debated, as it can be inaccessible and jarring for users. Many websites have implemented this, and it might be an eventuality, but we hope to exhaust other options first.

The challenge functionality was implemented manually using AWS WAF, and we also enabled AWS WAF Bot Control to help with bot mitigation.

The AWS WAF Bot Control tool is a black box. We spent last year using basic features to more easily detect bots and experimented with machine learning features. The downside of these manual and automated approaches is figuring out the impact on regular users and also “good” bots.

We also want to support non-malicious scraping of our site, especially by [archive.org](http://archive.org), but some of the rules above will block those requests as well. So a lot of it requires fine-tuning and analysis.

At the end of 2025, we worked with security professionals to evaluate JA4 “fingerprinting” as an option for restricting requests. The fingerprint is generated by analyzing packets captured from the client’s network requests. A fingerprint is created by abstracting data from those packets (e.g., TCP protocol, TLS version, number of cipher suites, number of extensions, among others). These fingerprints can then identify subsequent packets from the same source. (For more information on different types of fingerprinting techniques, check out this [excellent blog post by John Althouse](https://blog.foxio.io/ja4%2B-network-fingerprinting)). [AWS WAF provides us with fingerprints](https://docs.aws.amazon.com/waf/latest/APIReference/API_JA4Fingerprint.html) for all requests, including those from bots and regular users.

We noticed a consistent spike in requests from many IPs that share the same fingerprint. The request would have the same daily pattern. That detail was enough to make a case for blocking requests with that fingerprint.

We found other fingerprints as well, but also found that while the fingerprint is unique, it is also shared by everyone who uses the same browser, operating system, and extensions. So we again went with blocking the fingerprint, limited by region.

Doing that handled additional bot requests and helped reduce costs from the increase in traffic. Using a fingerprint is a promising approach, such as rate-limiting it instead of IP. We can also break down a [fingerprint](https://blog.foxio.io/ja4%2B-network-fingerprinting) and use the data to determine whether a request requires further scrutiny, as it might be a bot. Currently, blocking by fingerprint is reactive versus proactive, and we hope to explore how to dynamically block abusive fingerprints in 2026.

## What’s next?

Above, we shared how we have mitigated the increased bot requests on government websites. We have to remain vigilant and continue improving our approaches, as bots have changed how they send requests to websites. We are looking forward to exploring:

- Evolve our rate-limiting strategies with minimal impact on users and continue to explore options such as CAPTCHAs, fingerprints, and more.
- Continue to improve our identification of bot requests.

In Part 2, we will cover our approaches to stabilize website performance as we handle these increases in requests.

Thank you to Marissa Fox, Sharjeel Aziz, Tamara Maniscalco, Bob Schmitt, Eric Napier, and Scott McGee for your support, thoughts, and feedback.