id,summary,reporter,owner,description,type,status,priority,milestone,component,resolution,keywords,cc,parents 802,sitemap.xml generation,user-A,,"Hi, I'm partiticipating to the opensource distributed searchengine [http://yacy.net/ YACY]. Thus I like to encourage you, to offer webspiders an sitemap.xml file (and refer them in the robots.txt): https://en.wikipedia.org/wiki/Sitemaps This allows the bot to just walk trough the flat list and he doesn't has to get the webstructure out of the parsed HTML pages. This avoids that the searchengine misses some content and speeds up the analysis process.",defect,closed,major,,programming,wontfix,"seo, sitemaps, search",,