|
|
| Line 3: |
Line 3: |
| | A list of existing crawlers can be found at: [[Integrated_Search/Supported_data_sources|supported data sources]] resp. [[repo_config.xml]]. | | A list of existing crawlers can be found at: [[Integrated_Search/Supported_data_sources|supported data sources]] resp. [[repo_config.xml]]. |
| | | | |
| | + | Note that for most web-based sources, the generic [[Repo_config.xml#Web_ressources_.28.22web.22.29|web crawler]] can be sufficient for a first shot solution. However, the web crawler does index complete HTML files and might thus result in a lower search quality (e.g. due to indexing navigation menues etc.). |
| | + | |
| | + | == Pull vs. Push Crawlers == |
| | There are two general strategies for crawling data sources with TeamWeaverIS: | | There are two general strategies for crawling data sources with TeamWeaverIS: |
| - | * "Pull"-Crawlers are included in TeamWeaverIS and allow to crawl/extract data by accessing external sources/system. Pull-crawlers are fairly easy to realize, but have the disadvantage, that the index might not be accurate, since TeamWeaverIS does only learn about data changes, when a crawl is executed. | + | * "Pull"-Crawlers are implemented/deployed within TeamWeaverIS and allow to crawl/extract data by accessing external sources/system. '''Pull-crawlers are fairly easy to realize''', but have the disadvantage, that the index might not be accurate, since TeamWeaverIS does only learn about data changes when a crawl is executed. Typically, there is a script file to update the TeamWeaverIS index regularly (e.g. every night). Index accuracy thus depends on the frequency of crawls. |
| - | * "Push"-Crawlers are implemented on the side of the client system which includes the data. They proactively update the index and thus have a direct connection to the TeamWeaverIS backend. Accordingly, push-crawlers are more difficult to implement, but can reflect changes in the data more rapidly in the index. | + | * "Push"-Crawlers are implemented on the side of the client system which includes the data to be indexed. They proactively update the index and thus need to have a direct (network) connection to the TeamWeaverIS backend. Accordingly, '''push-crawlers''' are more difficult to implement, but can '''reflect changes in the data more rapidly in the index'''. |
| | | | |
| | == Writing a "Pull"-Crawler == | | == Writing a "Pull"-Crawler == |
| Line 24: |
Line 27: |
| | * tbd. | | * tbd. |
| | * Push-Crawlers have to call the TeamWeaver backend's [http://svn.polarion.org/repos/community/Teamweaver/Teamweaver/trunk/org.teamweaver.is.api/src/main/java/org/teamweaver/is/api/IndexService.java IndexService] | | * Push-Crawlers have to call the TeamWeaver backend's [http://svn.polarion.org/repos/community/Teamweaver/Teamweaver/trunk/org.teamweaver.is.api/src/main/java/org/teamweaver/is/api/IndexService.java IndexService] |
| - | * You might want to look at our [[Woogle]] code for an example implementation of a push-crawler | + | * You might want to look at our [[Woogle4MediaWiki]] code for an example implementation of a push-crawler. The particular place to look at is the "WoogleRemote" addon [https://waves1.fzi.de/svn/waves/trunk/Woogle4MediaWiki/addons/Remote/]. |