Tell HN: Amazonbot aggressively scraping my website and ignoring robots.txt

At the beginning of the year I decided to set up a scraping and LLM honeypot on one of my personal websites which included a fake git repo with code containing fake HTTP endpoints. The address to this repo was hidden in a public page inside a comment.

About three weeks ago IP addresses from Amazon Searchbot attempted to make requests to the fake endpoints included inside a shell script.

My robots.txt explicitly includes Amazonbot.

I am honestly surprised that this is coming from Amazon. Is this kind of behavior legal?

20 points | by pera 2 days ago

11 comments

leros 2 hours ago
It's not abnormal for people to disrespect robots.txt
Meta not only ignored my robots.txt but they hit me with 1k requests per second sometimes.
BingBot is doing something weird. Based on the logs I see, it feels like their crawler is trying to reverse engineer my public facing APIs and scrape those directly.
toomuchtodo 2 days ago
Respecting robots.txt is not a legal obligation. If you don’t want the traffic, ban the offending IP block or ASN, or put the origin behind Cloudflare for their crawler blocker.
You could go down the rabbit hole and try to find someone at Amazon to tell their crawler to chill. Don’t expect success but would make a fun blog post.
https://www.cloudflare.com/learning/ai/how-to-block-ai-crawl...
[-]
- pera 2 days ago
  My concern is more in relation to trying to use an endpoint from non-public source code, it seems negligent to randomly try endpoints like this
  [-]
  - toomuchtodo 2 days ago
    Whether it’s negligent isn’t terribly relevant. Is it illegal? It is not, they aren’t bypassing access controls. No different than using Shodan, scanning public IPs, crawling open directories, etc.
    It would be different if they were attempting to brute force credentials to access an endpoint, but they aren’t.
    [-]
    - rovr138 22 hours ago
      > they aren’t bypassing access controls.
      Weev went to jail for accessing public api's, https://en.wikipedia.org/wiki/Weev#AT&T_data_breach
      > The flaw was part of a publicly-accessible URL, which allowed the group to collect the e-mails without having to break into AT&T's system.
      It was argued that he didn't circumvent, but it didn't stop them from putting him in jail initially.
      [-]
      - toomuchtodo 16 hours ago
        His conviction was vacated and Amazon has deep pockets and diffusion of internal liability. The illicit state drug charges did not help his case.
schappim 1 day ago
It is not just Amazon (although they're #4 on my list of bad actors)[1].
1. https://files.littlebird.com.au/bad-scrapers.png
[-]
- pera 1 day ago
  Yeah it's not just Amazon, but so far Amazon was the only one specifically looking for URLs in source code. Interestingly it ignored URLs in the fake markdown docs.
  Another detail: the scraper did not attempt to access the endpoints immediately (as it did for hrefs in htmls) but it did it on the day after, twice.
  [-]
  - schappim 1 day ago
    Was it actually an Amazon bot IP[1] or someone pretending to be on AWS?
    1. https://developer.amazon.com/amazonbot/searchbot-ip-addresse...
    [-]
    - pera 1 day ago
      Yup, already mentioned in a comment below: the IPs are in that list.
      They also show up in AbuseIPDB with multiple reports.
- cyanydeez 1 day ago
  Perhaps you need a TOS that requires a usage fee be mailed to a PO Box that has a reasonable "free" limit of like 100$ and when they exceed it, you automatically mail them a copy of the TOS, the logs and an invoice.
  Just make sure you have a good lawyer.
  [-]
  - moomoo11 1 day ago
    is there a law they’re breaking?
    because idk i could be wrong but some small project vs a 2.5T market cap company is gonna need more than “a good lawyer”
    [-]
    - psd1 7 hours ago
      It would fall under contact law, and the case would hinge on either the ToS is binding, and whether the bot operator is deemed to have accepted the ToS, and whether access inflicts a cost.
      In the UK, you would likely win in court, and be awarded £1 for your increased hosting costs. There isn't the concept of punitive damages for contract law here. YMMV.
      However. People report that you can get paid by sending a plausible invoice to a tech company. The culture disincentivises verifying purchase orders.
    - psd1 8 hours ago
      Edit: oops, double-posted.
ipaddr 1 day ago
Ban Singapore that solved my Amazon traffic issues and probably others.
KellyCriterion 2 days ago
Ah, if its only Amazon, then you are in a good position :-))
I hosted a small website with some newfeeds and it got killed by all the AI scrapers in the end.

Bender 2 days ago

Assuming your site has no dependency on Amazon EC2:

    fetch_aws()
    {
     curl -s \
     -H "Accept-Charset: UTF-8" \
     -H "Accept-Encoding: gzip, deflate" \
     -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
     -H "Accept-Language: en" \
     -H "User-Agent: Mozilla/5.0 (Windows NT 11.1; rv:102.0; ) Gecko/20100101" \
     -H "Cache-Control: 0" \
     -H "Host: ip-ranges.amazonaws.com" \
     -H "Connection: keep-alive" \
     --referer https://ec2.amazon.com/ \
     -o /dev/shm/aws.json \
     --url "https://ip-ranges.amazonaws.com/ip-ranges.json"
    grep ip_prefix /dev/shm/aws.json | awk -F "\"" '{print $4}' | sort -n | uniq
    rm -f /dev/shm/aws.json;
    };

Make sure that gives you CIDR blocks, then add it to a script and use a for loop to

    for AWS in $(fetch_aws);do ip route add blackhole "${AWS}" 2>/dev/null

Google and Cloudflare if you need them:

    get_google()
    {
    for line in $(dig +short txt _cloud-netblocks.googleusercontent.com | tr " " "\n" | grep include | cut -f 2 -d :)                                                           
    do                                                                                                                                                                          
            dig +short txt "${line}"                                                                                                                                            
    done | tr " " "\n" | grep ip4 | cut -f 2 -d : | sort -n | uniq
    }
    
    get_cloudflare()
    {
    curl -A Mozilla "https://api.cloudflare.com/local-ip-ranges.csv"|grep -Ev "::|/32"|awk -F "," '{print $1}'|sort | uniq
    }

This assumes you have no need to connect to or get connections from Amazon on your web server. If your server is an instance in Amazon that will break DNS resolution unless you are using something outside of AWS for DNS. The gateway should still work just fine. Obviously test from an out of band console if that is an option. This is easier to maintain if you remove DNS records for IPv6 and eventually disable IPv6 listeners.

If you paste a couple of lines of the bots from your access logs I can offer more suggestions in the event they try from outside of Amazon.

If you want to have some fun, add a hidden link only the bot will see that points to http://cpanel.yourdomain.tld/ after adding a DNS record for cpanel that points to 169.254.169.254 so they start scraping the AWS cloud-init IP.

[-]

pera 2 days ago
Thanks, the Amazonbot IPs that accessed my honeypot are in this other list:
https://developer.amazon.com/amazonbot/searchbot-ip-addresse...
By the way, they used false user-agents.
az09mugen 1 day ago
Thanks ! I've been looking into a simple solution for blocking aws ips.

Era24UK 2 days ago
Which website platform are you using and hosting provider to support your need?
bellowsgulch 1 day ago
I haven't personally seen it, but I think us operators need to take entire registered ranges and hook 'em up to fail2ban. Maybe maintain lists of offenders and document it and block these orgs.
That is, I haven't personally seen a bad actors list that gets used in a fail2ban-like setup.
bruce_xu 1 day ago
[flagged]
deadcatfound 1 day ago
[dead]
jvondev 12 hours ago
[dead]