Had a peek at Adrian Holovatys post about how news sites populate their robots.txt, so I decided to take a peak on some Danish sites, and their robots.txt.

• Politiken disallows all agents, and all content.
• Berlingske doesn't use a robots.txt-file
• Ekstra Bladet disallows all agents, and all content.
• Jyllands-Posten doesn't use a robots.txt-file
• Information doesn't use a robots.txt-file
• WeekendAvisen doesn't use a robots.txt-file
• Børsen disallows all agents to some of their directories.
• BT doesn't use a robots.txt-file
• Danish Radio Broadcast doesn't use a robots.txt-file
• TV2 doesn't use a robots.txt-file

I am not sure if these observations gives me a reason to conclude any mindbending new universal facts, about how Danish news sites filter out web robots from skimming their content. After the big lawsuit against Newsbooster i would have thought that they all had a huge robots.txt-file to keep search engines from indexing their content.

I guess they trust the law then.