FYI 4 web 2.0, YMMV February 13, 2007 3:54 PM Subscribe
Google Reader is pretty crappy at updating certain feeds, especially MeFi and del.icio.us feeds (I'm pretty sure it's been discussed before and chalked up to whatcanyoudo). On a lark, I tried making a yahoo pipe for ask.me that doesn't actually do anything to the feed, just grabs it in, and splurts it back out. And guess what, its been updating like a champ all day. Your mileage may vary, but I thought I'd share this TOTALLY AWESOME and buzzword compliant "hack" with youse guys.
It works on my del.icio.us network and for feeds too, which were updating maybe once every couple weeks, if you care.
That's cool. If any other Google Reader users are wondering why mefi sites update infrequently, it's because years ago I had the googlebot turned down from "psycho-grabber of 100,000 pages every hour" to "moderate fetching at a few points in the day, only new stuff" and as a result, the feeds are only fetched like 3-4 times in 24 hours. That means nothing for 8 hours then bam, 40 new questions and 10 new mefi posts.
posted by mathowie (staff) at 4:32 PM on February 13, 2007
posted by mathowie (staff) at 4:32 PM on February 13, 2007
Hmm, does that apply to yahoo-bot? Cause then this would just be some kind of horrible short-lived pipe dream.
posted by 31d1 at 4:35 PM on February 13, 2007
posted by 31d1 at 4:35 PM on February 13, 2007
how does one turn down the googlebot?
posted by Dave Faris at 4:46 PM on February 13, 2007
posted by Dave Faris at 4:46 PM on February 13, 2007
it does not apply to yahoo bot.
Back in 2002 or so, I was getting crushed by the googlebot. 75% of the server's entire traffic was one googlebot gone wild each day and I begged them to stop.
Eventually, I had a long email exchange with a search engineer that wrote code for the googlebot who insisted the problem was me not sending correct http headers in my application and I kept coming back with how unreasonable the googlebot was acting, no matter what I'm doing on my end (I had no idea how to fix it). So they put in place a permanent "go easy on *.metafilter.com" rule somewhere that still applies today.
posted by mathowie (staff) at 4:55 PM on February 13, 2007
Back in 2002 or so, I was getting crushed by the googlebot. 75% of the server's entire traffic was one googlebot gone wild each day and I begged them to stop.
Eventually, I had a long email exchange with a search engineer that wrote code for the googlebot who insisted the problem was me not sending correct http headers in my application and I kept coming back with how unreasonable the googlebot was acting, no matter what I'm doing on my end (I had no idea how to fix it). So they put in place a permanent "go easy on *.metafilter.com" rule somewhere that still applies today.
posted by mathowie (staff) at 4:55 PM on February 13, 2007
how does one turn down the googlebot?
Matt seems to have a custom job, but now anyone can do it by setting up Google's Webmaster Tools. Once you've verified that you own the site, you can change the crawl rate from "Normal" to "Slower."
Faster becomes an option only under certain circumstances, which don't seem to be explained on the site. Mine just says, "At this time, crawl rate is not a factor in your site's crawl. If it becomes a factor, the Faster option below will become available."
posted by Partial Law at 5:57 PM on February 13, 2007
Matt seems to have a custom job, but now anyone can do it by setting up Google's Webmaster Tools. Once you've verified that you own the site, you can change the crawl rate from "Normal" to "Slower."
Faster becomes an option only under certain circumstances, which don't seem to be explained on the site. Mine just says, "At this time, crawl rate is not a factor in your site's crawl. If it becomes a factor, the Faster option below will become available."
posted by Partial Law at 5:57 PM on February 13, 2007
I tried to use Yahoo Pipes, but I think it should come with some kind of user manual.
/blonde
posted by Brittanie at 5:58 PM on February 13, 2007
/blonde
posted by Brittanie at 5:58 PM on February 13, 2007
how does one turn down the googlebot?
how does one not turn down the googlebot? it's so damn cute.
posted by loquacious at 6:10 PM on February 13, 2007
how does one not turn down the googlebot? it's so damn cute.
posted by loquacious at 6:10 PM on February 13, 2007
Brittanie: "I tried to use Yahoo Pipes, but I think it should come with some kind of user manual.
/blonde"
Do you perchance require assistance with that there pipe, ma'am?
posted by 31d1 at 6:14 PM on February 13, 2007
/blonde"
Do you perchance require assistance with that there pipe, ma'am?
posted by 31d1 at 6:14 PM on February 13, 2007
I set up my own CGI script that just grabs the feed and passes it on to me at Bloglines. About 1/2 the time the main feed is [!] there, mine works. The other half, both break.
posted by mendel at 6:28 PM on February 13, 2007
posted by mendel at 6:28 PM on February 13, 2007
Do you perchance require assistance with that there pipe, ma'am?
Bow chicka bow bow.
posted by Brittanie at 6:37 PM on February 13, 2007
Bow chicka bow bow.
posted by Brittanie at 6:37 PM on February 13, 2007
So they put in place a permanent "go easy on *.metafilter.com" rule somewhere that still applies today.
oh maybe thats why google seems to suck when searching for metafilter threads
posted by petsounds at 6:45 PM on February 13, 2007
oh maybe thats why google seems to suck when searching for metafilter threads
posted by petsounds at 6:45 PM on February 13, 2007
awesome. thanks dude!
posted by fishfucker at 7:32 PM on February 13, 2007
posted by fishfucker at 7:32 PM on February 13, 2007
I made a few Metafilter-related pipes yesterday, too, and was wondering how kosher it would be to show them off with my own Metatalk thread. Been wanting a universal MeFi feed for ages.
posted by brownpau at 7:36 PM on February 13, 2007
posted by brownpau at 7:36 PM on February 13, 2007
brownpau, I really wanted your pipes to work, but I only saw ten old mefi projects when imported to Google Reader. Does it eventually catch up with the present across all sites?
posted by mathowie (staff) at 8:45 PM on February 13, 2007
posted by mathowie (staff) at 8:45 PM on February 13, 2007
I combine all six feeds, then tell it to sort them by pubDate, and it still insists on sorting them by feed first. Apparently they haven't worked out date sorting to perfection just yet.
posted by brownpau at 5:05 AM on February 14, 2007
posted by brownpau at 5:05 AM on February 14, 2007
So they put in place a permanent "go easy on *.metafilter.com" rule somewhere that still applies today.
Would it be possible to ask Google to up it to a slightly more reasonable number than 4 times a day?
posted by Chrysostom at 5:41 PM on February 14, 2007
Would it be possible to ask Google to up it to a slightly more reasonable number than 4 times a day?
posted by Chrysostom at 5:41 PM on February 14, 2007
You are not logged in, either login or create an account to post comments
posted by cortex at 4:30 PM on February 13, 2007