How to Make my Web Site Spider Friendly

written by: Joseph White; article published: year 2006, month 08;



In: Categories » Internet » Search engines and SEO » How to Make my Web Site Spider Friendly

Getting into the Google index is largely a waiting game, in which preparation, persistence, and patience are the tools of success. However, a number of techniques incline Google’s spider to look on you more favorably:

  • Place important content outside dynamically generated pages: A dynamic page is one created on-the-fly based on choices made by the site visitor. This method of page generation works fine when the visitor is a thinking human. (Or even a relatively thoughtless human.) But when an index robot hits such a site, it can generate huge numbers of pages unintentionally (assuming robots ever have intentions), sometimes crashing the site or its server. The Google spider picks up some dynamically generated pages, but generally backs off when it encounters dynamic content. Weblog pages do not fall into this category — they are dynamically generated by you, the Webmaster, but not by your visitors.

  • Don’t use splash pages: Splash pages, (which Google calls doorway pages) are content-empty entry pages to Web sites. You’ve probably seen them. Some splash pages employ cool multimedia introductions to the content within. Others are mere static welcome mats that force users to click again before getting into the site. Google does not like pointing its searchers to splash pages. In fact, these tedious welcome mats are bad site design by any standard, even if you don’t care about Google indexing, and I recommend getting rid of them. Give your visitors, and Google, meaningful content from the first click, and you’ll be rewarded with happier visitors and better placement in Google’s index.

  • Use frames sparingly: Frames have been generally loathed since their introduction into the HTML specification early in the Web’s history. They wreak havoc with the Back button, and they confuse the fundamental format of Web addresses (one page per address) by including independent page functions within one Web page. However, frames do have legitimate uses. Google itself uses frames to display threads in Google Groups. But the Google crawler turns up its nose when it encounters frames. That’s not to say that framed pages necessarily remain out of the Web index. But errors can ensue, hurting both the index and your visitors — either your framed pages won’t be included, or searchers are sent to the wrong page because of addressing confusion. If you do use frames, make your site Google friendly (and human friendly) by providing links to unframed versions of the same content. These links give Google’s diligent spider another route to your valuable content, and give us (Google’s users) better addresses with which to find your stuff. And your visitors get a choice of viewing modes — everybody wins.

  • Divide content topically: How long should a Web page be? The answer differs depending on the nature of the page, the type of visitor it attracts, how heavy (with graphics and other modem-choking material) it is, and how on-topic the entire page is. Long pages are sometimes the result of lazy site building, because it takes effort to spin off a new page, address it, link to it, and integrate into the overall site design. From Google’s perspective, and in the context of securing better representation in the index, breaking up content is good, as long as it makes topical sense. If you operate a fan page for a local music group, and the site contains bios, music clips, concert schedules, and lyrics, Google could make more sense of it all if you devote a separate page to each of those content groups. Google also likes to see page titles relating closely to page content. Keeping your information bites mouth-sized helps Google index your stuff better.

  • Keep your link structure tidy: Google’s spider is efficient, but it’s not a mind-reader. Nor does it make up URL variations, hoping to find hidden content. The Google crawler is a slave to the link. If you want all your pages represented in the index, make sure each one has a link leading to it from within your site. Many site-building programs contain link-checking routines and administrative checks to diagnose linkage problems. Simple sites might not warrant such firepower; in that case, check your navigation sidebars and section headers to make sure you’re not leaving out anything.

legal disclaimer

1) Our website is not responsible for the information contained by this article as well for any and all copyright infringements by authors and writers. E-articles is a free information resource. If you suspect this article for any copyright infringements, please read the Terms of service and contact us to investigate the problem.
2) The E-articles directory team is not responsible for inaccuracies, falsehoods, or any other types of misinformation this tutorial may contain and will not be liable for any loss or damage suffered by a user through the user's reliance on the information gained here. Please read the Terms of service

Useful tools and features

Translate this article to...    Send this article to you or to a friend

Link to this article from your page   
If you like this article (tutorial), please link to it from your web page using the information above. Linking to this page, this is the only way to help us improve our service, the same time providing your visitors with a way to improve their online experience.

related articles

1. Google and U.S. Government Searches
Arguably, the most useful of Google’s specialty search areas is that devoted to the U.S. government. Actually, this distinct search engine is both larger and smaller than the name implies. This engine is global in reach. At the same time, it reaches below federal government sites to the state and municipal level. You might think that this entire search engine merely replaces the site:.gov operator:keyword combination. Not so. In fact, site:.gov remains quite useful in the UncleSam search because the results pages dish up a hear...

2. Google Search Operators
Google allows you to search using search operators, special words and symbols that make it easy to get search results that match as closely as possible the information for which you're looking. You combine search operators with search terms to form a query, like this: zeppelin "Led Zeppelin" That search would bring back pages that had the word zeppelin on them, but did not have the term Led Zeppelin on them. Here are the common operators you can use with Google: ...

3. Recommended Searching Strategies
Everyone has her own strategies for better searching, so it's always a good idea to ask others for their ideas. But here are a few hints that should help you with Google searches: Be specific The Web is an enormous place, containing literally billions of pages. Almost any search you do brings back far too much information. So make sure that your search is as specific as possible. If you're looking for a history of the making of the album Blonde on Blonde, by Bob Dylan, don't just search for "Blonde on Blonde&q...

4. Searching Google Catalogs
As in Froogle, Google Catalogs presents a topical directory and keyword searching. After you get into the directory, you can limit further searching to that directory category or launch a global Catalogs search. Start at the Google Catalogs home page. catalogs.google.com The directory tempts by listing a few mail-order companies in each main category. Feel free to leap into the directory by clicking either a catalog or a topic on the home page.Google maintains an archive of past catalogs, which can gum up th...

5. Google`s Approach to Online Shopping
The main difference between Google’s shopping services and those in other major portals is that Google doesn’t get its hands on the money. You don’t buy anything through Google. Both Froogle and Google Catalogs function purely as directories to products, sending you elsewhere to get your hands on the goods. Google has no revenue-sharing association with e-commerce retailers (in Froogle) or mail-order companies (in Google Catalogs). The search results you get in both services are pure; preferred placement in the search resul...

6. How to Download and Install Google Tools
Many Google tools live on the Web and don't require you to install any special software. For example, Gmail, Froogle, and Blogger are all websites; to use them, you only need to point your browser to the right page. But other tools are software programs that live on your PC, and they require that you download and install them before you can use them.All are installed in the same basic way, as outlined in the following steps. The Google Toolbar is used as an example of how to install software. (To see other downloadable tools, go to ...

7. Finding the Freshest Google Search Results
Google is not particularly strong at letting you determine the freshness of search results. The vagueness surrounding page freshness is due to several reasons: Google uses more than spiders to crawl the Web, and more than one type of spider.These crawlers operate at different speeds and different depths. It’s possible for a newly created Web page to go undetected by one crawl and then turn up in the index two weeks later after a deeper crawl. Google uses more than one server (Internet compute...

8. `Search by Numbers` Google`s Feature
You can type many numbers into Google to find information. Here's a list: Type a product's UPC code, and you are sent to the UPC Database, which gives you manufacturer information about any product. Type a Federal Express tracking number, and you are sent to a FedEx page that supplies tracking information. Type an U.S. Postal Service tracking number, and you are sent to a page with tracking information. You can only do it on packages that you can also track through the U.S. Postal...

9. How to Do University Searches Using Google
High school seniors take note: Google has your search engine. The University specialty searches let you rummage through a single university’s Web site with the power of Google’s search algorithms and operators. University search operates differently than the other specialty searches. Google does not aggregate many university sites for searching. And this is not a search engine for getting information about universities in general. Instead, Google has actually created dozens of small search engines, each dedicated to a sin...

10. Why Use Google
Let's start off with the most basic question: Why bother to use Google? After all, there are other search sites out there. It's true that there are other search sites, but none are as fast or effective as Google. Google doesn't necessarily scan more of the Web and index more websites than any other search engine. That's not really what's important. What it does better than any other site is deliver more accurate results. At times, the results seem so uncanny that you almost feel as if Google is reading your mind. ...