|Reported by:||user-A||Owned by:|
|Component:||programming||Keywords:||seo, sitemaps, search|
Hi, I'm partiticipating to the opensource distributed searchengine YACY. Thus I like to encourage you, to offer webspiders an sitemap.xml file (and refer them in the robots.txt): https://en.wikipedia.org/wiki/Sitemaps
This allows the bot to just walk trough the flat list and he doesn't has to get the webstructure out of the parsed HTML pages. This avoids that the searchengine misses some content and speeds up the analysis process.