Spider Script Question

[rocwalker1]

Hi Gang, haven't been here for a while, hope all is well with the old timers and the new one's too. I have a question.

We are preparing to launch a public use search engine later this year. URL submission will be at no charge. Sites submitted should be included withinin 48 hours and there will be paid listings.

Any SEO's here have thoughts about paid listings? We are planning on allowing a limited number per keyword bid and make it plain to the searching reader that the first results are paid. What would you do if you were us, paying god money for this project, wanted to make money, but also be fair to users and website owners?

Thanks ROC


MyGa.net (http://MyGa.net)

[Christian_SEO]

What you are thinking about doing is pretty well accepted in general, so it should be ok.

But if it were me, I might do something like let them pay for enhanced listings that appear where they may, in the search results, but allows them to call attention to THEIR listing.

This would provide the most relevent content to the searcher, but give an edge to the lister who wants to pay to have bold text highlighting the search phrase, or a box around the listing, or a small graphic like a logo in the listing...

so you would have:

Aardvarks unlimited
Portal site for aardvarks, both domestic and exotic

Aardvark Grooming of Toronto
Struggling with matted hair? Profession aardvak wranglers will..

***********************************
* Aardvark World *
* Fashion and accessories for your pet! *
***********************************

Thanks,
Christian

P.S. Please sign me up to be notified when you launch. I'll try to help get you listed in sites that lists search engines.

[rocwalker1]

Hi Christian SEO

***********************************
* Aardvark World *
* Fashion and accessories for your pet! *
***********************************

Good idea, an enhancement would draw attention to a web URL and entry. This bears looking into. Are you saying let the website fall where it may in the ranks but enhance it for a fee?

My only concern is and I will have to figure this out, What if a paid enhanced site is temporarily down during a weekly dead links check and gets dropped automatically without our knowledge or the owners knowledge. The dead link check program is very aggressive toward bad links and shows no mercy. Maybe we can script a notification in somehow. Anyway now I am talking to myself.

I will put you on our list for a notification prior to launch. Hope publication will take place within 60 days.

Best Regards, ROC

[Kal]

Welcome back ROC :hi: What you've got planned sounds fair to me, but I don't think I'd pay to "enhance" my listing if it is like the current AltaVista type enhancements. Why not introduce Paid Inclusion or Category Sponsorship?

Good luck with the launch and put me on your email list for publicity too.

[Christian_SEO]

Hey Roc,

You're not talking to yourself here... I'm sure that there is more than just me that is very interested in being able to provide suggestions for anyone building or operating a search engine or directory.

And yes, a modest fee makes your listing stand out from the others just like in the Yellow pages. The user's search query produces the results they want (hopefully) and the more serious advertisers can put their money on the line to compete for attention. It's fair, since the listings are not their because they are paid, but because the filled the request for information.

Another option is to "rotate" listings that fall within the same "search results score". That way everyone gets a turn at the head of the line for the content they provide.

Thanks,
Christian

[ihelpyou]

Good to see you again Roc! :hi:

I'm simply not interested in any ppc engines or otherwise. Just burnt out on all the new ones. I have never paid any of them. I am sure that Christian and others have many ideas.

Is your engine going to crawl around and index pages?

[rocwalker1]

Hi and thanks for your comments,

DOUG, As far as the crawl on our search engine, we will designate through admin which sites to crawl out of the user submitted listings. Sites with content get crawled, period. Sites without content simply get included ( 1 Page)

We will have 2 options in designating a crawl and how we want the program (spider) to perform it basic instructions. But thats all I can say about that.

For Example:

# 1 BillyBob's home page with pictures of his Dawg gets only the submitted URL included. No crawl.

# 2 A site about Mountain Crafts with several pages, solid content, pictures, stories, etc would get crawled and may have several pages listed under the submitted category.

This will be a content driven engine. Good for business and good for other web site offerings.


I appreciate all of your comments, it helps us to direct or even rethink our focus when I see what others perceive and are thinking.. I will put each of you (KAL included) on our notify list.

Now for a biggie question; Anyone care to comment on spiders instructed to disregard your robots tag? Just curious about this and how prevalent of a practice you perceive it to be?

[ihelpyou]

A good spider will obey all robots tags.

BTW, simply notify us through here. Many of us get our fair share of emails and yours would get deleted just like all the rest of them do if they are not a client. :)

Oh, okay. How about dynamic url's like these forums? I'd say we have some fairly good content in here. :)

[rocwalker1]

Hi Doug, Good question and I am glad you asked it.


Yes there is some excellent information in this forum. I have used information in this forum to bring a website from 25 visitors a day to half a million a year. So yes this is one of my favorite places on the net.

And as you know dynamic pages present a whole individual set of problems for many spiders. The worst problem being the ? (question mark) and the & . Some spiders stop at databsae designations, ie; ASP, PHP, etc. Since most dynamic pages are created by user input, well you know the rest. I have heard of spiders lost for days in endless URL creating dynamic pages. This puts a heck of a drain on resources.

Scripting for our spider will not include dynamic page crawls at first. I would suggest that sites with dynamic generated content also have a few , at least 3, static pages to lead the way and of course a site index or map page. This way the site can get a good listing and still bring the dynamic content to it's readers..