13 February 2002
Cryptome noticed the FBI bot in December 2001 but it may have come before. We would welcome information on other sites being mirrored by the FBI. Send to: jya@pipeline.com
We have no objection to the FBI mirroring or by anyone else, but would appreciate configuring bots so that only the new files listed on the home page are downloaded or checked each day, that is, do not redundantly check the links to other pages.
Excuse the panhandling, but we offer a CD of the 8,000-file, 600MB, bountiful archive for a mere $100 contribution. See the homepage.
Date: Wed, 13 Feb 2002 08:46 -0500 To: bdastur@fbi.gov From: jya@pipeline.com Subject: Help Please Dear FBI Administrator, For several days our site cryptome.org has been attacked by machines from 65.207.53.xxx, every morning beginning at about 3 AM and running for about a half hour, generating thousands of hits most of which are duplicated in succeeding days. These attacks are causing denial of service to other users, and we ask that they cease, perhaps by reconfiguring what appears to be an automatic program to download only new files and to avoid duplicating earlier accesses. See logfiles of attacks: http://cryptome.org/fbi-log020802.htm http://cryptome.org/fbi-log020902.htm http://cryptome.org/fbi-log021202.htm http://cryptome.org/fbi-log021302.htm Thank you very much, John Young Administrator Cryptome.org
http://www.arin.net/cgi-bin/whois.pl?queryinput=NETBLK-UU-65-207-53 MAS (NETBLK-UU-65-207-53) 935 Pennsylvania Ave NW Washington, DC 20535 US Netname: UU-65-207-53 Netblock: 65.207.53.0 - 65.207.53.255 Coordinator: Dastur, Brian (BD680-ARIN) bdastur@fbi.gov 202-324-6124 Record last updated on 17-Aug-2001. Database last updated on 12-Feb-2002 19:56:22 EDT.
Date: Wed, 13 Feb 2002 15:20:46 -0500 Subject: cryptome.org Cc: bdastur@fbi.gov To: jya@pipeline.com From: Stephen E. Schmidt <sschmidt@fbi.gov> Hi John! Brian passed me your messages regarding our web cache mirroring processes. He told me you were concerned about what appeared to be duplicative GET's of files already taken by the cache. Sorry if our cache has caused you any concern. From the logs you have on your website, I see many many HEADs (which should be there), and only a few GET's (indicating actual transfers of the file). Since the HEAD's indicate that our cache was stat'ing the files to see if they had changed from the last day, rather than transferring any files, it shouldn't be overly loading your system. In order to spare you any further pain until we get this worked out, I've temporarily removed your site from our cache system. If you agree that I'm correct about the above (GET=file xfer, HEAD=get status of file to determine if file should be downloaded), I'll re-enable things. If you believe I'm wrong, please correct me. Thanks, Steve Schmidt
Date: Wed, 13 Feb 2002 16:00 -5:00 To: Stephen E. Schmidt <sschmidt@fbi.gov> From: jya@pipeline.com Subject: cryptome.org Stephen, Thanks for your prompt response. You are reading the logfile correctly. We've observed that when your program (bot) is active it takes over the site for a period of time, usually about 24-25 minutes as it cycles (HEADS) through the links to check the old stuff and to GET the new files. Where the overload occurs is when the bot cycles through the backup page of the home page -- the backup page inherits listings from the home page as files age. The home page has a relatively few files compared to the backup page. A number of bots perform like yours, and we've asked users to try to set the bot to check for new files that are added daily to our home page and to not cycle through the backup page because the backup page does not receive original material. It is recycling through the backup page and linked pages that generate most of yours and other bot's logfiles, and while files are not GETed, the bots shut out others' access while active. That is probably due to the modest capacity of our server compared to more powerful machines. I'm no wizard on this but I've been told most bots can be configured to check only the home page for new files. If yours cannot gear down for this limited search, I have no problem with it continuing as it has been. Regards, John