WWW Index Server:
Frequently Asked Questions

Questions:

1. How does the system build its index?
2. What is a Web robot?
3. How do I identify a particular robot?
4. What sites are in the robot's target list?
5. How many Web pages/URL's have the system visited?
6. How do I register my site/page?
7. How can I tell robots to stay away from my site?
8. Your robot causes problems, what should I do?
9. How often does the system update its index?
10. Why does my pages get hit all the time?
11. Your robot never gone past my homepage, why?
12. Can I register a site outside HK?
13. What kind of data does the robot collect?
14. Which words in a page are indexed?
15. What search engine does the system use?
16. Is wildcard/word-pattern search supported?
17. Why do I need a Boolean query construct?
18. How do I search for phrases?
19. Why do I get results not relevant to my query?
20. Why does my query return nothing?
21. How does the system search on Chinese texts?
22. Why do I want to save my query?
23. Does the system index gopher, ftp, telnet sites?
24. Is source code available?
25. What code library is used in the implementation?
26. Any online documentation?
27. Where do I send comments/enquiries?

Answers:

1. Q: How does the system build its index?
   A: It runs a Web robot which automatically retrieve Web pages, traverses hyperlinks and collects information.

2. Q: What is a Web robot?
   A: Essentially, it is a user-less Web browser/client-program. Its internal algorithm decides what information to retrieve and where to go. Today, there are many Web robots out there on the net. The majority of them are designed for collecting index data.

3. Q: How do I identify a particular robot?
   A: If you have access to your site's Web server access-log and full logging mode is set, a (well behaved) robot can be identified from its "User-Agent" field. Here is our robot's identification data. Robots which honor the Proposed Standard for Robot Exclusion always make an attempt to retrieve a plain text file named robots.txt before sending further access requests (unless the robots.txt disallows them to do so.)

4. Q: What sites are in the robot's target list?
   A: Our service covers only Web servers/sites in Hong Kong territory. Here is a list of Web servers visited by our robot.

5. Q: How many Web pages/URL's have the system visited?
   A: See our weekly summary.

6. Q: How do I register my site/page?
   A: You can register your site or Web page by filling out our URL registration form. Pages linked to the registered page will be automatically traversed by the robot, so they need not be registered.

7. Q: How can I tell robots to stay away from my site?
   A: Assuming all robots honor the Proposed Standard for Robot Exclusion, you can prepare a text file named robots.txt in your site's root directory containing the following:

   
   User-agent: *
   Disallow: /
   

   disallows any robot to retrieve any page from your site. The "Disallow:" field can be used to prevent robots from retrieving pages under the specified directory path. For example:

   
   User-agent: *
   Disallow: /webstat/
   

   disallows accesses to pages under "/webstat/" directory. Multiple "Disallow:" fields is permitted. Empty "Disallow:" means any request is allowed. The "User-agent:" field can be used to specify a particular Web robot, by naming the robot's User-Agent identification name. If no robots.txt file is found, by convention, a robot can assume that any access request is allowed.

8. Q: Your robot causes problems, what should I do?
   A: Please feel free to report any problem our robot may have caused to our robot maintainer.

9. Q: How often does the system update its index?
   A: Once in every two weeks we run an index maintenance process. For every URL/page in the index, this process checks if the URL/page is valid (exists) and if it has been modified since the last time it was visited by the robot. Such a probing mechanism is supported by the HTTP (Hypertext Transfer Protocol) by means of a special request called "HEAD request." A page is then revisited if its last modification date is later than the last time it was visited by the robot.

10. Q: Why does my pages get hit all the time?
    A: On some systems, weekly file-system backup automatically resets the last modification date of the whole file system. As a result, all Web files in the file system are thought of by HTTPD (the Web server program) as have been changed on that date. Based on this information, our system automatically runs the robot to re-index the pages.

11. Q: Your robot never gone past my homepage, why? A: This usually happens when pointers to your other pages are accessible only through an image-map. For an obvious reason, our robot can not construct access requests from image-maps.

12. Q: Can I register a site outside HK?
    A: To avoid any possible redundant coverage with other Web indexing systems out there, we restrict our coverage to Hong Kong based sites only. Our main purpose is to facilitate browsers outside Hong Kong (as well as those inside Hong Kong) with an easy way to locate information provided by the Hong Kong Internet community.
    One way to make an off Hong Kong site keyword-searchable, albeit indirect, through our Index Server is to have its URL listed somewhere in some local Web page.

13. Q: What kind of data does the robot collect?
    A: Our robot collects URL's (Web addresses), page titles, hypertext anchors, hyperlink data, and keywords.

14. Q: Which words in a page are indexed?
    A: Our robot collects words in page titles, hypertext anchors, headings, list items, and words in bold/italic/emphasized type fonts. Function words or common words are discarded.

15. Q: What search engine does the system use?
    A: Our search engine employs the so called vector space model [Salton, G., Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer, Addison-Wesley, Reading MA, 1989.] to search and rank documents (URL's) based on keywords specified by the user. Basically, the algorithm works based on word distributions within a document and across documents.

16. Q: Is wildcard/word-pattern search supported?
    A: No, because our search engine uses hashing search method. However, word stemming is done to remove suffices (for example `s' suffix for plural nouns) from the query and index words. Also, the search is also case insensitive (disregards upper/lower cases.)

17. Q: Why do I need a Boolean query construct?
    A: You can use Boolean construct to specify multiple queries (search statements) in one line. For example, if you are interested in information about hotels, instead of using query: "hotel information guide", you can get a better result using query: "hotel (information | guide)." Unlike the former query, the latter query gives less scores to "information guide" with missing word "hotel."

18. Q: How do I search for phrases?
    A: You can use a - (hyphen) to connect two words of a phrase so that the words are not treated independent of each other. For example, query: "local-area-network."

19. Q: Why do I get results not relevant to my query?
    A: The search engine returns N documents (URL's) with relevance scores higher than 0 (N is user-specified with a default value of 40.) However, having a score higher than 0 does not guarantee that a page is relevant with respect to the query (let alone to your real information need.) If you find an irrelevant URL in the result list, you can be sure that other URL's further down the list (with lower scores) are also irrelevant.

20. Q: Why does my query return nothing?
    A: First, make sure that the spelling is correct. If perfect spelling does not help, consider using the word's synonyms. If that fails, the keyword(s) may not be in the index. Our system does not index Web-specific common words (among others) such as "homepage", "home", etc.

21. Q: How does the system search on Chinese texts?
    A: Our system searches on Chinese texts based on character-sequence statistics. The version of the Chinese search engine that we are running now is still very crude, but works reasonably well.

22. Q: Why do I want to save my query?
    A: You may save your query for later reference (save you time for fine tuning the query), or for others to share your discovery. Some people save failed queries with a hope that somebody somewhere will provide the unavailable information online.

23. Q: Does the system index gopher, ftp, telnet sites?
    A: Yes, as long as they are listed on some Web pages somewhere.

24. Q: Is source code available?
    A: Not at this moment. Partly because we are short in staff.

25. Q: What code library is used in the implementation?
    A: Our entire system is written from scratch in C language, except for the index database management module which is implemented using the Gnu Database Management (GDBM) library package.

26. Q: Any online documentation?
    A: We currently have two papers submitted for publication. One paper, "Search and Ranking Algorithms for Locating Resources on the World Wide Web" (postscript format, 174KB), describes and evaluates a number of document ranking/search algorithms used by the earlier version of this server. The other paper, "A WWW Resource Discovery System"", describes our prototype of an integrated system consisting of the indexer robot, the search engine and an enhanced user interface to the search engine.