To hide or not to hide 25.08.2002
Had a peek at Adrian Holovatys post about how news sites populate their robots.txt, so I decided to take a peak on some Danish sites, and their robots.txt.
- Politiken disallows all agents, and all content.
- Berlingske doesn't use a robots.txt-file
- Ekstra Bladet disallows all agents, and all content.
- Jyllands-Posten doesn't use a robots.txt-file
- Information doesn't use a robots.txt-file
- WeekendAvisen doesn't use a robots.txt-file
- Børsen disallows all agents to some of their directories.
- BT doesn't use a robots.txt-file
- Danish Radio Broadcast doesn't use a robots.txt-file
- TV2 doesn't use a robots.txt-file
I am not sure if these observations gives me a reason to conclude any mindbending new universal facts, about how Danish news sites filter out web robots from skimming their content. After the big lawsuit against Newsbooster i would have thought that they all had a huge robots.txt-file to keep search engines from indexing their content.
I guess they trust the law then.