Robots.txt, Robots Meta Tag, .htaccess mod_rewrite

There are three commonly supported methods for instructing/requesting internet indexing spiders/bots/robots what to scan and what to skip. Each of these methods are complimentary in usefulness to each other, but none are not equal in effect.

1. robots.txt
2. Robots Meta Tag
3. .htacess and mod_rewrite

Summary:

To really protect and enforce rules for any specific user agent that is visiting your website you will have to constantly analyze website traffic analytics, bandwidth reports and visiting IP addresses and geographic locations, known pubilc or private proxy servers, and the specific methods and tactics of EVERY unwanted program and visitor and be able to implement new means to thwart their new methods on a regular basis.

Block Unwanted Visitors by IP Address or UserAgent in Apache using mod_rewrite

Use .htaccess rules to block unwanted bots, spiders and other UserAgents that don’t fetch, or that fetch and ignore robots.txt.

Blocking visitors by IP address filtering in .htaccess file:

# deny specific IP addresses, and allow all others
order allow, deny
deny from 123.45.6.7
deny from 123.45.6.8
deny from 123.45.6.9
allow from all

Block specific UserAgent using mod_rewrite

   # Block Google Images Bot from Indexing your Copyrighted Images
   # Hopefully someday Google will publish a "supported way" of
   # Disallowing the Google Image Bot when necessary, but until then...
   RewriteEngine on
   RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Image
   RewriteRule ^(.*)$ http://images.google.com/

The catch-22 with this method is that “sneaky” program developers can simply masquerade as “normal” visitors by using common web browser user agent strings. Reinforcing the fact that all three of these methods are USEFUL, but in no way a complete or secure solution even with the precise use of all three.

Also see:

1. robots.txt
2. Robots Meta Tag

Robots Meta Tag

Use an embedded meta tag on a specific page to instruct search engine spiders and robots what to index and disallow:

1. Pages including “noindex, nofollow” indicate that they are NOT to be index, NOT to be included in listings, and NOT to be scanned for reciprocal links.
2. Pages including “index, nofollow” indicate that they are to be indexed and listed, but not scanned for reciprocal links.
3. Pages including “index, follow” indicate that they are to be fully index and scanned for all reciprocal links and included in all applicable listings.

DO NOT index, DO NOT include in listings, and DO NOT follow reciprocal links

<input name="robots" content="noindex, nofollow" />

Index, include in listings, but DO NOT follow reciprocal links

<input name="robots" content="index, nofollow" />

Index, include in listings, and follow reciprocal links

<input name="robots" content="index, follow" />

Also see:

1. Robots.txt
2. .htacess and mod_rewrite

Robots.txt

Robots.txt is a plain text file that is implemented in the root directory of a URI as a configuration file used by some search engine spiders and internet robots/bot programs to help direct them to what you want to be indexed and what you don’t. Although many robots will read and follow your instructions in the “/robots.txt” file, many ‘less compliant’ programs may actually ignore this file completely.

Here are a few examples of robots.txt file (plain text):

Ask all search engines to NOT index or follow links on the entire website:

#asks all search engines to NOT index and NOT follow any pages or links on the entire website
User-agent: *
Disallow: /

Allows all search engines to index and follow links on the entire website by Disallowing nothing:

#allows search engines to index and follow all pages and links on the entire website by Disallowing nothing
User-agent: *
Disallow:

Disallows specific folders and files from indexing and following:

User-agent: *
Disallow: /uploads/ # since this folder may contain secure, private, cached or temporary files, we should disallow this entire folder from being indexed.
Disallow: /tmp/ # since this folder may contain cached or temporary files, we should disallow this entire folder from being indexed
Disallow: /page.php

Also see:

1. Robots Meta Tag
2. .htacess and mod_rewrite

Search Engine Optimization (SEO)

SEO is an acronym for “search engine optimization” or “search engine optimizer.” Deciding to hire an SEO is a big decision that can potentially improve your site and save time, but you can also risk damage to your site and reputation. Make sure to research the potential advantages as well as the damage that an irresponsible SEO can do to your site. Many SEOs and other agencies and consultants provide useful services for website owners, including:

• Review of your site content or structure
• Technical advice on website development: for example, hosting, redirects, error pages, use of JavaScript
• Content development
• Management of online business development campaigns
• Keyword research
• SEO training
• Expertise in specific markets and geographies.

Keep in mind that the Google search results page includes organic search results and often paid advertisement (denoted by the heading “Sponsored Links“) as well. Advertising with Google won’t have any effect on your site’s presence in the search results. Google never accepts money to include or rank sites in the search results, and it costs nothing to appear in the organic search results. Free resources such as Webmaster Tools, the official Webmaster Central blog, and the discussion forum can provide you with a great deal of information about how to optimize your site for organic search. Many of these free sources, as well as information on paid search, can be found on Google Webmaster Central.

Additional Resources for Webmasters, Developers and Search Engine Optimizers:

• Check out Google’s Search Engine Optimization Starter Guide
• Read more on the Google Help page:  Search Engine Optimization (SEO)

Next Page »