Problems with INEEDHITS/Inktomi

[aps]

First Post,

I have a problem with Ineedhits/Inktomi. My site has been fine for over 7-weeks and has been spidered successfully up until a few days ago. I made absolutely “NO” changes to the site (in profile). All of a sudden I am getting an error 7000 which means that your robots.txt file is blocking Slurp spider. I do “NOT” have a robots.txt file nor do I have any robots META data. Everyone else has spidered successfully and so has Inktomi up until a few days ago. I have gone to great lengths and used sim-spiders to check the site and all is in order. I even deleted the entire site and reposted that INK spidered successfully before. I have filled out the Ineedhits support form until I’m blue in the face and they are unresponsive. My logs show that slurp has successfully spidered but yet I still get this error. Does anyone, anywhere have any idea what the problem could be.

Thanks in advance!

[ihelpyou]

Welcome to the forums aps! :hi:

Since I do not pay, I cannot help you.

If slurp is crawling and ineedhits says no, then the problem is with ineedhits.

[aps]

I paid for submission and when you log in to your account it show the status of the url's. I also get an email kicked out everyday stating I have a robots.txt file blocking them. I stated before I do not even have a robots.txt file. Does anyone know how to contact INK directly as ineedhits support seems useless and unresponsive.

Thanks

[MsSearch]

:hi: aps and welcome...

You really need to contact ineedhits as it is their service that is giving you the error. If Slurp is coming around and crawling but in your ineedhits logs, you are getting an error, then the problem lies with ineedhits service...

It could just be a temporary bug/glitch in their system...How long has it been since you contacted ineedhits?

I don't think contacting Inktomi will solve the problem....as Slurp is able to crawl your site...

[aps]

MsSearch,

Thanks for your reply. This is the following message (acct info snipped).

========================
CUSTOMER SERVICE
ineedhits.com
Inktomi Search/Submit
========================

Dear xxxxx

Account number: xxxxx

There has been an error in the submission of one or more URLs to the Inktomi Database.

The affected URLs are:

URL: http://www.aps-remodeling.com/
Error: ERROR_DISALLOW: Crawling Disallowed by robots.txt
Link: http://ink.ineedhits.com/inclusionerrors.asp#7000


To see an explanation of the above error(s) please click on the error number.

To correct or change the URL address please login to your account and make the necessary corrections.
https://ink.ineedhits.com/secure/login.asp

If you have any further problems please contact us again through our Contact Us or FAQ page on the site.
http://ink.ineedhits.com


Kindest Regards,

Customer Service
------------------------------------------------------------------------------------

The problem is that my site is now dropped from INK/Partners because of the above. I am gone completely! Ineedhits is completely unresponsive and I have filled out their service request form everyday for the last 4 days. I just keep getting autoresponses. :(

To say I am displeased with their support is understated!

[MsSearch]

Have you tried calling them?

From their website:
If you would prefer to speak with one of our ineedhits.com Customer Service Staff over the phone, please feel free to call us on one of the numbers below:

Calling from the US: 011 61 8 9244 7066
Calling from Australia: ( 0 8 ) 9244 7066
Calling from elsewhere in the world: +618 9244 7066

If you can't get a response through email then try calling them, at least you can try speaking to a 'real' person as long as they don't keep transferring you from person to person....

[aps]

I have seen these Phone #'s but I'm sure the call would cost an extremely large amount. I have contacted Ink directly and they replied with a tech email address that I have send the information to. I have learned a valuable lesson and am glad our company values our customers more than is shown by iNeedHits. I am very displeased with their service.

Thanks for the reply,
Dave

[Alan Perkins]

Hi Dave

If the site is the one in your profile the problem is probably that you have recently installed a 404 handler and have no robots.txt file. So your handler is picking up the access to robots.txt and serving an illegal file. Inktomi is choking on it.

I recommend putting a robots.txt file on your server, in the root directory. It should just contain the following two lines:

User-agent: *
Disallow:

After you have published it, visit it with a browser to ensure you can see it. If you can, your problem should be solved next time Slurp visits. :)

You can find more details on robots.txt at www.robotstxt.org

Alan

P.S. Send my bill to ineedhits, would ya? :D

[aps]

Alan,

I have not thought of that - giving it something it is asking for. I have had the 404.html for a while though. I have set it up my account for spidering again so if it works, I owe you. Do you use PayPal? :)

[Alan Perkins]

Just my little joke. It's on the house. :) I thought it was odd that something so straightforward couldn't be answered by Ink or ineedhits, both of whom you have paid!

I see you've published a robots.txt now. That looks fine. I would guess your problem will quickly go away.

[aps]

Alan,

Thanks Again! I will report back tomorrow with the results.

I think the problem with some businesses is that once they have your money, they don't really care. I find everywhere I turn the lack of customer service is what's going to bring down a lot of companies that feel they can't be hurt. I have had quite a few people as of late speaking to me about how so many of the search engines “results” return irrelevant information. Very few of the internet users realize that the PPC results are listed first or that they even exist. I have turned quite a few people into Google users once they find out about PPC results.

I’m rambling now :)

Thanks Again (Sincerely),

Dave

[aps]

Alan,

Of course, Ink/Slurp did not try to spider the site last night :(

It still shows queued for inclusion in my account. Good news is iNeedhits finally emailed me and told me "I did indeed have a robots.txt file". I replied and told them I "just" put it up on the suggestion of someone (you) else. Ink usually crawls this site at about 2:30 AM EST so I am not sure why it did not last night (checked my logs). I guess I will have to wait another day/night. It may have got tired of trying to crawl :(

Ineedhits stated they will be using toll-free support in the near future which would be nice.

Sincerely,
Dave S.

[aps]

Bad News- Just received another:

========================
CUSTOMER SERVICE
ineedhits.com
Inktomi Search/Submit
========================

Dear David xxxx,

Account number: xxxx

There has been an error in the submission of one or more URLs to the Inktomi Database.

The affected URLs are:

URL: http://www.aps-remodeling.com/
Error: ERROR_DISALLOW: Crawling Disallowed by robots.txt
Link: http://ink.ineedhits.com/inclusionerrors.asp#7000


To see an explanation of the above error(s) please click on the error number.

To correct or change the URL address please login to your account and make the necessary corrections.
https://ink.ineedhits.com/secure/login.asp

If you have any further problems please contact us again through our Contact Us or FAQ page on the site.
http://ink.ineedhits.com


Kindest Regards,

Customer Service

----------------------
ineedhits.com
Inktomi Search/Submit
http://ink.ineedhits.com
----------------------
"Keeping Websites Alive"(TM)
Ineedhits.com Pty Ltd

[aps]

Just to post an update,

iNeedHits has been keeping in close touch and has regained my faith :)

They even phoned from Australia and said there is a problem on Inks end and they are talking with Techs there to sort out what the problem is. With Google's index/results going haywire and now the problem with Ink, it has not been a good week :(



Thanks to all,

Dave S.

[aps]

Update- end of thread,


Well my site was crawled last night. iNeedHites wrote and said the problem was invalid syntax in my robots.txt file. I wrote back and said that was impossible because I "NEVER" had a robots.txt until days after the problem began. It's working so I'm happy. Thanks to all that let me cry on their shoulder.


Dave S.

[Alan Perkins]

Hi Dave

Glad it's all sorted. The ineedhits analysis was correct - you did have an illegal robots.txt file. It was the file thrown out by your 404 handler.

Your 404 handler sends the following response:

HTTP/1.1 302 Found
...
Location: http://www.aps-remodeling.com/404.html

I'm not sure whether Slurp would choke on the 302 (a redirect), or follow it and retrieve 404.html and try to treat it as a robots.txt file. Either way, that was almost certainly the source of your problem, and putting a valid robots.txt file in place has almost certainly fixed it. The response for that file is now:

HTTP/1.1 200 OK

If you want to test the theory, just remove the robots.txt file again and see if the problem re-occurs! (joke).

[aps]

Alan,

Thanks for your reply. I still don't understand, since my 404.html was in place for weeks and they never had the problem before. I want to sincerely thank you for all of your knowledgeable advice.


Thanks Again,

Dave S.

[Alan Perkins]

Sounds like either you have recently changed something or Inktomi have.

You may have had 404.html for weeks, but have you had the 404 handler that redirects to it for weeks?
Have you recently changed the way the handler works?

OTOH, Inktomi may have recently started following the redirect or specifically rejecting it, whereas before they treated it as a 404 i.e. the behaviour you wanted.

Who knows? It's fixed now, anyway. :)

[aps]

Alan,

I had not changed anything with the 404 since it's inception. The only thing I have added has been original/information content and changed the page(s) design a little. The strange thing was my logs/stats were showing that it crawled successfully.
I am not sure what caused it but like you said it is fixed now.

Thanks Again,

Dave S