Gort_Firing How much do you know about the Robots.txt file?  It can be a handy tool for any webmaster to know more about.  Here are some of the most frequently asked questions I get about this special little file.

What is the robots.txt file used for?

In web site development, the robots.txt file is used as a special file that can talk back to the search engine spiders and crawlers to tell them what to do. Here is a little more about them from

Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.

How do you create a robots.txt file?

All you have to do is create a basic text file (with the name robots.txt) in your public_html folder inside of your hosting account. You can create that from within your control panel or you can create it on your desktop, and upload it via an FTP client.

Can a robot ignore my robots.txt file?

Yes, but this does not happen that often.  Most good main-stream  robots will not do this.  So is there a way to block just the bad robots?  Kind of.  You can only control robots that obey the robots.txt file.  If that doesn’t work, you might also be able to block the IP address the robot is coming from (if they are coming from a single IP address). 

What should I put in my robots.txt file?

This would depend on what you want to do.

If you would like to make sure all crawlers and spiders make their way in, insert this into your robots.txt file:

User-agent: *

If you want to keep the spiders and bots away from your content in a particular folder, use this:

User-agent: *
Disallow: /privatestuff

With "privatestuff" being the folder you wish to protect.

How can I check to see my robots.txt file has been created successfully?

It is easy to check if your have an account with Google Webmaster Tools. To analyze a site’s robots.txt file:

  • Sign into Google Webmaster Tools with your Google Account.
  • On the Dashboard, click the URL for the site you want.
  • Click Tools, and then click Analyze robots.txt.

How can I figure out where this robot or crawler came from?

You can find a good list of just about all the different robots out there here:

That would be a good list to check first, when you have a robot crawling your site, and you are unaware where it came from.

How do you know you are being visited by a robot?

He will mumble something about klatu verata nikto.  No wait, that was my buddy Gort.  Often the sign that your being visited by a robot is that in your server logs you see many documents have been retrieved in a very short period of time. 

Hope that helps in your robot taming ways.  If you have any other questions, feel free to ask away.

Related Tips and Tricks:

© Lunarpages Web Hosting – Also, don’t forget to follow @lunarpages on Twitter!

Free Fonts to Use!

Looking for a modern looking font for your next design?  Look no further than the Contemporary – Modern font.  This wonderful creation (via MyFox on deviantART) is a great simple font face to use for anything from logos to ad text in your next graphic masterpiece. 

This is font free for public or commercial work.  If you use their font, be sure to link back to them though (or at least tell them thanks).  You can pick it up over on the DeviantART web site.

Related Tips and Tricks:

© Lunarpages Web Hosting – Also, don’t forget to follow @lunarpages on Twitter!

Create a Google Profile!

Have you setup your Google profile?  A while back, Google started to show profiles in their search results, so if you want to make sure somebody is able to find you – it might be a good idea to set up or update your own Google profile.

To get started, go to Google’s create a profile page, and click on the create my profile button.  From there, you can list your name, upload a photo, give a description about yourself, list your super powers (what, you don’t have super powers?), list your web site links and so much more.  The above picture is an example of my own profile on Google.

Once you are done, it might take a while, but you should start showing up as a result at the bottom of your Google search for your own name.  Here is mine, as another example:

Mitch in the Google Search Results

So there you have it.  That is one more easy way to make sure people find you and your web sites when searching via Google.

Related Tips and Tricks:

© Lunarpages Web Hosting – Also, don’t forget to follow @lunarpages on Twitter!

Free Images for Your Web Project

Stock photography has always been a little scary for me.  Mainly cause most of it you find out there, the people look so creepily happy.  For the most part, they remind me of the kids in the Children of the Corn movies.  Blank stares aside, I have found one great place to grab a few free stock photos – photoXpress.

If you are tired of paying for your stock photography, then this is the place for you.  They have hundreds of free photos to choose from.  The catch here is that you are only allowed one photo a day.  If you want +4 daily downloads, all you have to do is become a fan of them on Facebook.  The more information you give them, the more downloads you can get.  Sounds like a pretty good deal to me.

So if you are in need of soem free stock photography, check out

Related Tips and Tricks:

© Lunarpages Web Hosting – Also, don’t forget to follow @lunarpages on Twitter!

Web Hosting Newsletter

Be sure to go re-subscribe (or subscribe for the first time) to our new Lunarpages Newsletter feed.  It has been recently changed, to make it a little more memorable – as well as so we can add some interesting new features to it in the upcoming weeks and months ahead.  Here is the NEW RSS FEED ADDRESS:

To help you get the job done, here are a number of RSS feed chicklets you can click on to subscribe using your favorite method of choice (or to subscribe to multiple places, so you never miss a post).   

Add to Google Reader or Homepage  Add to My AOL Subscribe in NewsGator Online Subscribe in Rojo Add to netvibes Subscribe in Bloglines Add to The Free Dictionary  Add to Plusmo Subscribe in NewsAlloy Add to netomat Hub Add to Excite MIX Add to fwicki Add to Webwag Add to Attensa Receive IM, Email or Mobile alerts when new content is published on this site. Add to Pageflakes

Just one more Lunartastic thing to get excited about on this fantastic Friday morning.

Related Tips and Tricks:

© Lunarpages Web Hosting – Also, don’t forget to follow @lunarpages on Twitter!

Next Page »