Block Unwanted Visitors by IP Address or UserAgent in Apache using mod_rewrite
Use .htaccess rules to block unwanted bots, spiders and other UserAgents that don’t fetch, or that fetch and ignore robots.txt.
Blocking visitors by IP address filtering in .htaccess file:
# deny specific IP addresses, and allow all others order allow, deny deny from 123.45.6.7 deny from 123.45.6.8 deny from 123.45.6.9 allow from all
Block specific UserAgent using mod_rewrite
# Block Google Images Bot from Indexing your Copyrighted Images
# Hopefully someday Google will publish a "supported way" of
# Disallowing the Google Image Bot when necessary, but until then...
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Googlebot-Image
RewriteRule ^(.*)$ http://images.google.com/
The catch-22 with this method is that “sneaky” program developers can simply masquerade as “normal” visitors by using common web browser user agent strings. Reinforcing the fact that all three of these methods are USEFUL, but in no way a complete or secure solution even with the precise use of all three.
Also see:
Bookmark to:
Robots Meta Tag
Use an embedded meta tag on a specific page to instruct search engine spiders and robots what to index and disallow:
- Pages including “noindex, nofollow” indicate that they are NOT to be index, NOT to be included in listings, and NOT to be scanned for reciprocal links.
- Pages including “index, nofollow” indicate that they are to be indexed and listed, but not scanned for reciprocal links.
- Pages including “index, follow” indicate that they are to be fully index and scanned for all reciprocal links and included in all applicable listings.
DO NOT index, DO NOT include in listings, and DO NOT follow reciprocal links
<input name="robots" content="noindex, nofollow" />
Index, include in listings, but DO NOT follow reciprocal links
<input name="robots" content="index, nofollow" />
Index, include in listings, and follow reciprocal links
<input name="robots" content="index, follow" />
Also see:
Bookmark to:
Sitemap Generators: Automating Google Sitemap Creation for Search Engine Optimization
Server-side Programs
- Enarion phpSitemapsNG (PHP)
- Google Sitemap Generator (Linux or Windows, 32/64bit, open-source)
- GS Master (Polish, PHP/Perl)
- Outil en PHP (French, PHP)
- Perl Sitemap Generator (Perl)
- Python Sitemap Generator (Python)
- Simple Sitemaps (PHP)
- SiteMap XML Dynamic Sitemap Generator (PHP) $
- XML Sitemap Generator (PHP) $
CMS and Other Plugins
- ASP.NET – Sitemaps.Net
- ClickCartPro $
- Coppermine
- DotClear (Spanish)
- DotClear (2)
- Drupal
- ECommerce Templates (PHP) $
- Ecommerce Templates (PHP or ASP) $
- LifeType
- mnoGoSearch
- OS Commerce
- PHPNuke (German)
- PHPNuke – NukeSEO Sitemaps Generator
- phpWebSite
- Plone
- RapidWeaver
- Textpattern
- vBulletin
- Wikka Wiki (PHP)
- WordPress
Downloadable Tools
- CoffeeCup Sitemapper (Windows) $
- Devintelligence SitemapGenerator (Windows)
- GSiteCrawler (Windows)
- GWebCrawler & Sitemap Creator (Windows)
- G-Mapper (Windows)
- Inspyder Sitemap Creator (Windows) $
- IntelliMapper (Windows) $
- KeyLimeTie Sitemap Generator (Windows) $
- Map-IT Sitemapper (Windows, Mac) $
- MediaCET Diadematus (Windows)
- Microsys A1 Sitemap Generator (Windows) $
- Rage Google Sitemap Automator $ (OS-X)
- Site Magellen (Windows)
- Site Map Pro (Windows) $
- Sorrowmans Sitemap Tools (Windows)
- Vigos Gsitemap (Windows)
- Web-design-pros.ca Sitemap.xml.gz Generator (Java Webstart Application)
Online Generators/Services
- AuditMyPc.com Sitemap Generator
- AutoMapIt
- Autositemap $
- Enarion phpSitemapsNG
- Free Sitemap Generator
- Neuroticweb.com Sitemap Generator
- Safe Area SitemapGenerator
- ScriptSocket Sitemap Generator
- ResourceSEO Online Sitemap Generator
- SitemapDoc
- Sitemapspal
- Smart-IT-Consulting Google Sitemaps XML Validator
- XML-Sitemaps Generator
Code Snippets / Libraries
Bookmark to:
A List of Major and Minor Search Engines with PageRank and Alexa Rank
|
|
Bookmark to:
Free Search Engine Submission Services
AddPro.com
AddPro.com provides a free service that performs a basic submission of your site to 18 of the top search engine providers:
| Abacho | FindOnce | Splatsearch |
| Acoon | Fybersearch | Surfgopher |
| Aewi | Burf | |
| Cipinet | Jayde | Walhello |
| Entireweb | Towersearch | WotBox |
| ExactSeek | ScrubTheWeb | Searchit |
AddMe.com
AddMe.com also provides a free search engine submission service that auto-submits your website to the following Search Engines:
| Scrub The Web | FyberSearch | |
| Burf.com | Alexa | BigFinder |
| LifeTips | ExactSeek | Nerdworld |
| InfoTiger | FocusLook | Walhello |
| Jayde | EntireWeb | AdmCity |
| SpiderMonkey | MixCat | Abacho |
| SearchSight | WhatUSeek | ESiteSecrets |
| Tower Search | TyBit | Burf.org.uk |
FreeWebSubmission.com
FreeWebSubmission.com provides free manual and auto submission to the highest-rated, Free Internet Search Engines and Directories.
| Yahoo! Search | Bing | |
| Open Directory | ScrubTheWeb | Cuil |
| ExactSeek | SearchSight | EntireWeb |
| LII | Web World | VieSearch |
| A1 Web Directory | Skoobe | Info Listings |
| IllumiRate | Dramba | Directory Storm |
| Arakne Links | Wikidweb | LinkSite |
| MavicaNet | Link Centre | My Directory Live |
| Domaining | Pedsters Planet | BusinessSeek |
| Ablaze Directory | Info Tiger | What U Seek |
| OneMillionDirectory | SurfSafely | World Site Index |
| Search Web World | AMRAY | CyberWebSearch |
| Finest 4 | NewWebDirectory | IS |
| IWD | The Living Link | SonicRun |
| Wondex | Cipinet | Mardox |
| NetInsert | 01WebDirectory | Web Directory |
| Clock Tower Studio | TowerSearch |
Bookmark to: