AV Spidering

[Great-1]

Is anyone else being absolutely murdered by Scooter???

It has spidered over 80,000 pages in 12 hours, and is using the vast majority of our bandwidth. Everyone in the office is having difficulty getting out to the internet, and people are having problems getting to our site.

[ihelpyou]

Sounds to me like Scooter has gone mad! :) You may have to block him as something may have gone wrong with the robot.

[Great-1]

I've excluded them in my robots.txt file, but they have, once again blatently ignored it. They did the same last time I had this problem.

[ihelpyou]

Must have gone awry then. It will have to scooter on back to AV to get tweaked and fixed.

[Mel]

Hi Great1

Now that you mention it scooter has visited one of my sites which has only 40 pages and so far has in three visits spidered 280 pages.

[JuniorHarris]

Scooter should be pulled for DUI (defying universal intelligence)!~ Scooter has been all over our domain as well, I think he actually performed three complete deep-level index sessions in one day!~ Now how's about an index update? :rolleyes:

[bkztx]

Scooter has been going nuts on my site as well! Using up tons of bandwidth and my hosting company sends me nasty messages. I wouldn't mind if AV would do an update. As of now, I can't even find my site.

[ihelpyou]

Well actually, I have seen many changes in the index since yesterday. Some kind of update must have happened.

[markymark]

I've noticed a little movement at AV also. Lost 30 places (from 17-54 on my main keyword) but I can't see any new sites in there.

[ihelpyou]

What it looks like to me is that AV has switched indexes once again. They seem to be rotating two different indexes every month. I do not see new pages in either, just a rotation change.

[highman]

They are in the process of merging all the regional databases into one, this is prob. what you are seeing.

A side note, I recently talked one of our clients into AV paid inclusion, using my normal optimisation routine the site is doing very well (#4) most of our other sites (not paid) are buried..... off the scale.

A boost for paid inclusion?

[ihelpyou]

I would be shocked to know that AV would be giving a boost to pages who "paid". That would do a certain amount of undermining to the AV index. IMO

But your find is interesting. Hope it is not true, however.

[Great-1]

Just a quick update:

Scooter has spidered my site like a spider possesed for the last 6 weeks or so, and it appears as though 2 pages have been added to the index.

My message to AV, if you're gonna kill my bandwidth, make it worth my while, other wise, naff off (or words to that effect).

[MsSearch]

:green: LOL....

[JuniorHarris]

>My message to AV, if you're gonna kill my bandwidth, make it worth my while, other wise, naff off (or words to that effect).

ditto that!~

[bkztx]

I got an e-mail from AV telling me that they could not spider my site because I have a Robots.txt file excluding fetch. I do not have such a thing. Any ideas or comments?:confused:

[ihelpyou]

I am not sure as I do not know what some of this means. Maybe others can help. This is your robots.txt file right now for your site:

User-agent: *
Disallow: /websys/
Disallow: /lib/
Disallow: /ackfiles/
Disallow: /lib.bk/
Disallow: /temp.bk/
Disallow: DSN.txt
Disallow: site.ini

One of the bottom two is telling AV to not spider any of your site.

[MakeMeTop]

>A boost for paid inclusion?

I've seen the same thing and suggested immediate 'aging' for paid inclusion. It is as if pages have been in the AV index for 6 months and worked up through the ranks.

[ihelpyou]

You could be right but would be very bad for AV. To rule out giving all the free sites and nonprofit sites out there a fair shake in ranks would be their demise.

[JuniorHarris]

Forget the ranks, let's talk traffic!~:rolleyes:

[MazY]

<Wakes up> Is AV still around then? :D

I'm with JH (again) if it don't produce traffic then what really is it worth?

Now for the seamless link - talking of traffic - since changing my sites, boy have I suffered! Good job I have a bizzare sense of humour that laughs at such things!

Luckily, I was about to change keyphrases anyway as my experience tells me that "web site promotion" (certainly in the UK at least) is about as useful as a fishnet window in mid-winter! :D

On to phrase number 34958359

[bkztx]

Junior,
I thought you had to have ranks to get traffic. If you know a better way, I'm open to suggestions. :)

[ihelpyou]

I think what JH meant is that even if you have good ranks in AV, it does not equate to "good" traffic. :)

[bkztx]

Now I get it! Thanks, ihelpyou. ;)

[MazY]

Anyway, JH comes from an old Malawian Tribe where mysticism is rife. You'd be amazed at what he can do with a golf ball and a candle, let alone traffic without rankings!

[Advisor]

I for one would like to see the golf ball and candle trick.

J

[MazY]

You are a sick puppy, Mrs. Whalen! :)

[Advisor]

That's MS. not Mrs.

And if I'm a sick puppy, it's probably due to the company I keep!

J

[MazY]

lol. Can't mean me, I'm an innocent don't ya know.....

[JuniorHarris]

Moni!~ :hi:

Well I suppose now is the best time to share the biggest secret of all, Immaculate Traffic Inception! This is a blend of old world secrets with new world technology. It includes the candle, a Kampango, and the URL of a virgin listing. It is an intricate process, which if not executed properly can have disastrous side, side, side-effects.

However, not to worry as I have included all the necessary artifacts and detailed instructions in my new:

Malawian Tribe mysticisms search engine optimization kit

This kit includes everything necessary to unlock over 20 mystic optimization techniques. Learn how to increase traffic with Immaculate traffic inception. Increase sales with mystical marketing chants, and even learn how to apply banishment hexes against competitors.

All this and 17 other do-it-yourself mystic techniques, but act fast as supplies are going quick. (and the fish are starting to smell). PM me for your order today, and the first 10 responses will receive a complete restock kit at no cost!

[JuniorHarris]

bkztx, I was being sarcastic. Yes high ranks (for competitive or well searched words) are necessary for traffic. But as Doug suggested, even with high positions AV does not drive much traffic. Just compare the traffic volume for a given keyword across engines. I would imagine that 10th position on google would/could drive more traffic then the same listing as number one on AV!~ :eek:

[bkztx]

I want to know more about this Malawian Tribe optimization kit! Where do I sign up?:up:

[Advisor]

I think I saw an infomercial on it the other day at about 4 in the morning on my local UHF station...looks like a good deal...and at only 26 payments of $19.99...how can you go wrong?

Jill

[bkztx]

I just need to know one thing-- What is a kampango???

[Advisor]

I don't know what kampango is either, but it sounds kinda kinky! A cross between a kangaroo and a mango, perhaps??? :D

Jill

[manwah]

Apparently it's like a Mlamba, only not as common. :)

M

[ihelpyou]

ol' JH is going to get a few PM's now about this. :)

[JuniorHarris]

Kampango" (Bagrus meridionalis) is a famous catfish in Malawi. (http://ecology.kyoto-u.ac.jp/~yuhma/FLM_html/FLM_E_1.html)

[Great-1]

Originally posted by JuniorHarris
Kampango" (Bagrus meridionalis) is a famous catfish in Malawi. (http://ecology.kyoto-u.ac.jp/~yuhma/FLM_html/FLM_E_1.html)

Didi you find that doing a search on AV?? :D :D

[MsSearch]

Oh JH....thank you so much for sharing your secrets...and here I've been putting all this hard work into optimization...Sign me up!!! :D

[JuniorHarris]

LOL!~ I thought a little humor would brighten the day...

[bkztx]

Much appreciated, Junior. We all need a little humor to stay sane these days.:)

[Great-1]

Sorry to open this discussion up again, but Scooter is once again going mental.

I am getting (insert expletive here) annoyed with these morons. I have had 14 people call up in 3 hours saying my site is too slow to load. I am losing customers and money.

Is there any legal issues involved with this??

[ihelpyou]

I don't think so. I guess you could exclude scooter with a robots.txt file.

It's one thing if AV gave some traffic, but another since they do not.

[markymark]

Scooter seems to ignore robots.txt most of the time. Have you tried denying it access using a .htaccess file ?

[Great-1]

They do ignore the robots.txt file.

How do I go about creating a .thaccess file???

[markymark]

An .htaccess file is a file that goes right at the root of your server. If you create it as a .txt file, then upload it, you should the rename the file .htaccess (thereby removing the .txt), at which point it disappears from sight, but is still there.

The file should include something like this

AuthName "Spider Access Blocked"
AuthType Basic
<Limit GET POST>
order allow,deny
allow from all
deny from 202.412.45.1
</limit>

Obviously you will have to include IPs for Scooter to do this effectively. There may be an easier way to do this and I'm no expert at messing around with servers, but this should work, unless one of the real techies here has a better idea.

[markymark]

Oh, incidentally this only works with Unix/Linux servers, not Windows.

[Great-1]

Unfortunately we are on a Windows server so that's a no go.

We have had a major breakthrough though, we've got hold of a phone number of someone in Crawl Support at AV, and told him off for spidering us. I also asked him why it took about 4 months to refresh there data, and whether it would be full of crap when it does refresh. He declined to answer that one.

1-0 to the little people :D :D :D

[lots0cash]

I’m already to go catfishing, anyone know where Malawi is?

[markymark]

LOL - that will teach them to go one on one with the Great-1, won't it ?

[Kal]

Originally posted by Great-1
They do ignore the robots.txt file.
Hey Great-1, AltaVista claim they support the Robot Exclusion Standard so just create a "disallow" tag aimed at Scooter.

See their tutorial for "Avoiding the Index" here:
http://help.altavista.com/adv_search/ast_haw_avoiding

[ihelpyou]

Obviously, they do not have very good control of the ScooterMeister then. :)

[Great-1]

Unfortunately, MarkyMark, it didn't teach them. Scooter is still murdering me. AAAAAGGGGGGHHHHHHH.

Kal, AltaVista sent me that link in an email when I first complained to them. My robots file was coded on the specifications given to me by a member of AV last time I had this problem, alas they just ignore it.

Today I've sent a sample of my log file to AltaVista for them to look at. I'll keep everyone up to date on the situation.

[Great-1]

Update:

I blackmailed the guy at AltaVista by telling if he didn't pull his finger out, I'd give his name, email address and telephone number out on all the search engine foums I know, causing him to be inundated with angry SEO's.

This tactic worked, as I am now expecting a call from the Vice President of Alta Vista Europe.

Does anyone have questions they want me to ask him (apart from why is your search engine so pants)?

[Kal]

Originally posted by Great-1
Does anyone have questions they want me to ask him (apart from why is your search engine so pants)?

Yeah I have one. Why don't you ask them how successful their new "Listing Enhancements" (pictures next to listings) has been? :green: :green: :green:

[Great-1]

Well, unsuprisingly the Vice President of AltaVista Europe didn't call me, but one of the guys from their NOC did. I told him to stop the spidering or I would post his private mobile number on forums (blackmail worked once, so I tried it again). He stopped the spidering, and told me to ring his private mobile number if it started again. I might ring him just to get him out of bed at 4 in the morning on Thanks Giving :D :D .

I also discussed compensation, stating that I'd lost 3 days business and wanted to compensated. I suggested that they make my site top of their rankings for some suitable keywords. He's looking into it :D . I was only half serious.

If I get my way, and they put me top, I'm gonna step on some peoples toes. People that have worked hard to get top rankings, but hey, it's a dog eat dog industry, and if you want to go 1 on 1 with the Great-1, just bring it :D :D