Cry for help
I had a cry for help recently from an old friend who had moved to Canada or USA not sure which lets just call it the land of Hummingbirds. Her husband had a web site and they had been trying to get it listed on Google they’d been frustrated with this issue for a number of months and when they posted a comment on one of my Facebook updates I offered to have a look.
Two pronged attack
Generally I review a site from two angles, the first is what I used to consider the most important and it deals primarily with the content and in particular if the structure and copy will give the best result from Google. Once I’ve reviewed the content I move onto the site set up to make sure an updated site map is attached and create a relevant robots.txt file.
Robots dot what file?
This was the last thing I checked on my friends site and I wish it had been the first. The robots.txt file is a file used to tell bots which areas of your site they are allowed or disallowed to visit/crawl. My friends site was built on a free hosting package and they had no control over the sitemap.xml file or more importantly the robots.txt file, the contents of their robot.txt file is below.
# Block all bots from the core Homestead site
User-agent: *
Disallow: /~site/
You may have guessed or worked out that the code above will block all bots from visiting/crawling this site so any work done on any other areas of the site will be absolutely pointless.
