By Daniel Mundra, Associate Director of Drupal
In Part 1, we covered the increase in bot traffic and our approaches to identify the bots and ways to mitigate their requests. In this post, we will cover our approaches to maintain a government website that is serving users and also handling spikes in requests. Here are some strategies, what worked, pros and cons, future risks, and what we want to look at next. We continue to focus on our work with a Drupal website hosted in an Amazon Web Services (AWS) cloud environment.
Baseline
Before getting started with the strategies, we recommend knowing how your website is currently performing and the resources it uses under normal traffic. Our approach is to use a dashboard that shows all our website resources, how much they are used over time, and when traffic typically increases and decreases.
Our government website was not new; it was migrated from older technology to Drupal. As we rolled out the new website in phases and transitioned more requests to it, we could see the impact of increased requests, how the website was performing, and adjust our resources accordingly. That gave us an idea of what the regular requests were and set that baseline. The con with this approach is that those regular requests also included bots, and that will be incorporated into the baseline. We don’t think the goal is to get to 0 bot requests, but to manage the flood of requests.
Here are some example statistics on the number of ‘hits’ per day:
- From June 2024 to February 2025 we averaged about 3.2 million requests to the firewall per day with about 0.6 million requests making it to Drupal.
- In February, after we did a big rollout of the new website, we went to 6.7 million requests to the firewall per day with 0.8 million requests reaching Drupal per day through the rest of the year.
- The aggressive scans from China and other regions at the end of the year saw a spike to 26 million requests per day to the firewall until we blocked it. We ended the year with 10 million requests to the firewall per day and a fraction of that reaching Drupal.
Caching
Caching is a common and recommended step for any website. You want all pages to be cached, ideally for a long time, especially if you have an easy, automated way for the content editor to clear the cache when they update the content. Serving cached content is going to be cheaper and also faster for users who visit your website (even bots).
On our website, we had a 30-minute cache on our pages. We were able to increase it to an hour after improving our automatic cache-clearing functionality. Our caching was implemented in multiple layers: we used the AWS CloudFront content delivery network (CDN) with caching enabled in front of the Drupal website, and Drupal caching was enabled as well. Our automatic cache-clearing functionality would clear the Drupal cache when content is updated and communicate with AWS CloudFront to purge the content’s URLs from its cache.
Our website received requests, including bot traffic, from many regions, so we also enabled CloudFront Origin Shield to improve cache performance. We collaborated with AWS technical staff to ensure the cache content is available in more edge regions, so users in those regions can serve requests faster than waiting for the request to reach the Drupal website. After implementing it, we saw an improvement in cache hits.
Serving cached content to users and bots has worked because it reduces the number of requests Drupal must process and serve. Users get quicker page load times, which is very important. Cache has been relatively easy to configure and to improve iteratively, such as increasing the time, setting the cache in Drupal and CloudFront, and automating cache clearing at both levels. The majority of our website is cached, but the downside of caching is handling requests that have multiple query parameters. With query parameters, different parameters are treated as individual pages, causing each one to need to be cached: if you have a lot of combinations, that can be a lot of pages.
One approach we are looking into is allowing only the query parameters we want and ignoring the rest. This should help reduce the risk that different query parameters cause more requests to bypass the cache. Another approach is to make our websites more static and move the filtering to the client side, rather than the server side. That will be another way to reduce the impact on Drupal (known as Decoupled Drupal). Both approaches require resources and are architected differently, so it is not easy to switch automatically from dynamic to static sites. However, it is worth exploring as a way to manage a lot more requests than a dynamic site.
Load Testing
Load testing is a useful tool for assessing how your website performs under heavy load, simulating bot traffic. We use an open-source load-testing tool called Locust to regularly test our website’s performance and identify infrastructure bottlenecks.
Our website has many layers: an Elastic Kubernetes Service (EKS), a Relational Database Service (RDS), a search appliance, storage, networking appliances, and more. Running a load test lets you see where networking, memory, and CPU are spiking; which pages are taking a long time to load and why; and whether errors are being thrown, where they occur, and what the error message is. Doing that gives you an idea of where you need to increase resources to serve the requests you want to.
We used Locust to determine that with auto-scaling Drupal pods in EKS, our search appliance was reaching its connection limit before Drupal or RDS reached their performance limits, so we focused on increasing it to support more connections. The downside of not load testing, or of load management in general, is that serving a large number of requests comes at a cost.
Load testing can show that after a certain number of requests, you will see errors or other issues. Increasing resources to support that number will mean higher costs. If 50% of the requests are bots, should we — or our customers — bear the cost to pay to serve bots? Probably not. We recommend using load testing to ensure your current target request volume is supported by the resources, and use the result to set a threshold for anything that exceeds it, with restrictions applied.
What’s next?
In addition to the exploring rate-limiting and bot identification techniques from part 1, we are also looking forward to exploring:
- How to continue to improve our caching
- Pushing our websites to deliver even more static content
- Use load testing and testing strategies to ensure our site’s bottlenecks or performance issues are removed
Thank you to Marissa Fox, Sharjeel Aziz, Tamara Maniscalco, Bob Schmitt, Eric Napier, and Scott McGee for your support, thoughts, and feedback.