WebSavvy
03-09-2004, 23:30/11:30PM
Many search engines and directories are being listed in automated rank checking software against the wishes of the search engine or directory owners.
My own directory has become a target of ranking checking software as well.
I've placed a robots.txt DISALLOW to my search form. Any bot that obeys protocol would have obeyed it, but NOT Ranking Manager (http://www.websitemanagementtools.com/).
The bot identifies itself when it's scraping the pages, so banning it through robots.txt should have worked -- if it was a compliant bot.
I've sent email notification to Michael Lange of Sophtware Inc., developer of this noncompliant software, and requested that my directory be removed from his software.
He's been contacted three separate times and to no avail. He has not complied with the request to remove my property from his software.
My permission to add it to his software was never obtained, nor was it ever sought. We have a published TOS (http://www.websavvy.cc/TOS.php) which specifically prohibits automated query (data mining) of our server and data.
Three days ago, I changed the URL of our search page. Mr. Lange followed by fixing his software to again query our server. This very act alone shows willful and deliberate disregard for our published TOS.
Blocking this software at the server level is not possible as it's ran from the user PC and shows the IP of the user.
So, I've written a script that will change the variables on our search page at a regular interval. The code for doing so, I will make open source.
Any search engine owner or directory owner who would like to protect their data and stop the harassment of automated software violating their server, may PM me and ask for the codes.
At this point, I'm not sure publishing the code openly would be a wise idea, although, even so, it wouldn't help the scum software producers anyway.
My own directory has become a target of ranking checking software as well.
I've placed a robots.txt DISALLOW to my search form. Any bot that obeys protocol would have obeyed it, but NOT Ranking Manager (http://www.websitemanagementtools.com/).
The bot identifies itself when it's scraping the pages, so banning it through robots.txt should have worked -- if it was a compliant bot.
I've sent email notification to Michael Lange of Sophtware Inc., developer of this noncompliant software, and requested that my directory be removed from his software.
He's been contacted three separate times and to no avail. He has not complied with the request to remove my property from his software.
My permission to add it to his software was never obtained, nor was it ever sought. We have a published TOS (http://www.websavvy.cc/TOS.php) which specifically prohibits automated query (data mining) of our server and data.
Three days ago, I changed the URL of our search page. Mr. Lange followed by fixing his software to again query our server. This very act alone shows willful and deliberate disregard for our published TOS.
Blocking this software at the server level is not possible as it's ran from the user PC and shows the IP of the user.
So, I've written a script that will change the variables on our search page at a regular interval. The code for doing so, I will make open source.
Any search engine owner or directory owner who would like to protect their data and stop the harassment of automated software violating their server, may PM me and ask for the codes.
At this point, I'm not sure publishing the code openly would be a wise idea, although, even so, it wouldn't help the scum software producers anyway.