Record of a website traffic surge issue (possibly caused by DataforSEO crawler)

# My Website Traffic Skyrocketed - The Crawler Incident and Resolution

This article documents an issue where my website traffic experienced an abnormal surge, doubling or tripling almost daily. While the GA console appeared normal, the final investigation revealed it was caused by a crawler. I hope this incident provides some experience for you. If you've encountered something similar, feel free to read on.

## Problem Summary

Recently, my game resource website "[MainGamers](https://www.maingamers.com/)" experienced a traffic surge problem. The continuous increase in visits lasted for several days. As shown in the GA console below, a few days ago, the average daily visitor count was less than 2000. In just four days, it climbed to over 200K!

![](https://res.lodenhu.com/media/media-cmjzyhfcz000001pa5iyq1ha7-1791888843005.webp)

When the traffic first started to climb on the first and second day, I thought it was a traffic explosion from a trending keyword. However, upon closer inspection, I realized something was wrong. The visitor surge didn't correspond to any actual increase in logged-in users, which didn't make sense. I also noticed that the traffic was primarily concentrated on the landing page!

## Hacking Attack?

This led me to suspect a hacking attack. I started thinking about defense measures.

During this period, I tried to analyze the "hacker's" "attack" methods and techniques to devise countermeasures. I also implemented some practical defense measures on Cloudflare, such as rate limiting, bot interception, and enabling interactive verification by region, among other basic protection methods. Initially, these measures seemed to help, with real-time traffic dropping by about half. However, after a day, the total traffic continued to climb!!!

I started to question reality... Then I thought, with such massive traffic, my already strained server remained unharmed. Was this "hacker" too weak? Are there such weak hackers? My rational mind told me no. I began to feel lost...

## Discovering the Truth

Since my code didn't actively log detailed information like visitor UserAgents or IP addresses, I had no data to effectively troubleshoot. I previously considered these logs to be overkill for my website.

Then I remembered that while configuring Cloudflare's defense measures, I had seen visitor IP data. I quickly checked if I could find any clues, and indeed, I found useful information, as shown below:

![](https://res.lodenhu.com/media/media-cmjzyhfcz000001pa5iyq1ha7-1791889674764.webp)

The monitoring graph showed that a single IP address made over 10K visits! Then, by checking the IP overview, I could see associated domains for that IP, meaning this IP was likely managed by an organization.

I then researched the domain `dataforseo.com` and discovered it belonged to an organization called DataforSEO, a commercial SEO company that uses its own crawlers to scan data across the web.

At this point, I was almost certain that this organization's crawler was "attacking" me! However, I didn't understand why. Through discussions with AI, I realized my website was missing a `robots.txt` file... The AI suggested: "Perhaps due to the absence of `robots.txt` and your website's login redirection mechanism, the crawler's logic entered a dead loop, leading to this situation..."

## Attempting and Resolving

Alright, I added a `robots.txt` file and upgraded my code. However, several hours after implementing these changes, when I checked the GA console, the real-time data hadn't improved. I reluctantly contacted DataforSEO directly through their website to try and resolve the issue.

My request was roughly:

> "I've noticed your crawler is frequently crawling my login page. I'm unsure how to stop it. Please investigate, and if there's an issue, I hope you can resolve it."

DataforSEO's response was quick. They informed me via email:
> "Our crawlers are beneficial. If you wish for them not to crawl xxx, please declare it in your robots.txt. If adding it doesn't work, you can try resubmitting your website on the xxx page to trigger a refresh."

In essence, it was a very common reply. With a "let's give it a try" attitude, I proactively submitted my domain on their page. Indeed, a few hours later, the data displayed on the GA panel showed a significant improvement. Real-time visitors dropped from over 4000+ to within 1000 in 30 minutes, as shown in the initial graph. However, through real-time monitoring, I could see that sometimes there were still over 100 visitors within a minute. This indicates it hasn't been completely resolved, but at least the data is no longer as alarming.

## Conclusion

This incident is worth documenting because it was quite troublesome. The reasons for the trouble are as follows:

1.  The website's `robots.txt` was incomplete. According to SEO best practices, login pages should ideally not be crawled by crawlers (according to AI).
2.  DataforSEO's crawler IPs were too human-like... GA couldn't distinguish them.
3.  DataforSEO's crawler was too "weak" and easily entered dead loops...