Quote:
Originally posted by jazzy_rosez
I was wondering does spiders or robot count as users towards your bandwidth usage? How should you setup you refresh option for a page and/or site? is the refresh option really needed?
|
Yes they do. However, they don't produce gigs of traffic, most people are happy, when they get spidered. If you are not, there certain META tags to deny them.
"refresh" is only necessary if you are dealing with a lot of dynamic (ever-changing) content. If not, don't bother.
Code:
<META NAME="Revisit" CONTENT="After 10 days">
By the way, if you got access logs, you can see what spiders came to your site under browsers. They act just like a visitor, only difference is, that instead of sending IE or Netscape, they send "Google", "Metacrawler" and such.