%% Good answer to question. WSJ tried to get me to resubscribe, so i asked them if they have a free 10 day trial, like IBD does/did?? If they ever outlaw truth like IBD ;may read WSJ, for free.WSJ dose have some helpful info info on takeovers, but that is mostly bear markets LOL
Does the robots.txt file not work anymore for search engines, or are the over-paid pimply-faced know-nothings not doing their jobs on how to keep their website out of engine search caches with the latest technologies? I bet the latter. 50 sim cookies! (Chocolate-based).
robots.txt keeps the page from being indexed at all. If you want people to pay to get behind your paywall you want your articles indexed, you just only want people to actually be able to read the first couple paragraphs. I'm not aware of Google caching full WSJ articles, I think the OP might have been referring to sites that are actively scraping their content and displaying it, knowing that they can get away with it for a short time before the takedown and not caring that they will be taken down because they'll just pop up again.
Does anyone stop and calculate the opportunity cost of time spent trying to get around blocks or getting partial or non standardized news feed versus paying a few bucks for a reliable, consistent source of news flow ? Further more what % is that few bucks of your total cost of doing business ?
What news feed is reliable? I would guess the AP is the only one left. When they fall, we're all screwed, and we'll get our most accurate news from the dead relatives we buried.