Youth social networking researcher danah boyd has observed that many people presume the way they use social networks is the way everyone uses them. "I interviewed gay men who thought Friendster was a gay dating site because all they saw were other gay men," she says. "I interviewed teens who believed that everyone on MySpace was Christian because all of the profiles they saw contained biblical quotes. We all live in our own worlds with people who share our values and, with networked media, it's often hard to see beyond that."

Now picture our perspective leaving our own experiences, zooming out and up until we can see how all the different groups are interacting on a worldwide social network. That bird's-eye view could be both beautiful and horrible if the resolution was clear enough. That's what a Ramen-eating, ex-Apple engineer named Pete Warden is about to release to the public this week.

Sponsor

This Wednesday, Warden will make Friend, Fan page and name data from hundreds of millions of Facebook users available to the academic research community. It's a move that Facebook has to have seen coming, a move that many in the data-centric community have been calling on the company itself to do for years, and an event that's been complicated by Facebook's recent privacy policy changes, which have muddied the waters of right and wrong but rendered even more data available for outside analysis.

If what people call Web 2.0 was all about creating new technologies that made it easy for everyday people to publish their thoughts, social connections and activities, then the next stage of innovation online may be services like recommendations, self and group awareness, and other features made possible by software developers building on top of the huge mass of data that Web 2.0 made public. It's a very exciting future, and Warden is about to fire one of the earliest big shots in that direction.

Nerds in Space: Social Graph Analysis For Solving Large-Group Problems

Warden studied Computer Vision in college in the U.K., then got into game development. After moving to L.A., he spent six years building graphics drivers for the original Playstation and the XBox. Then he started his own independent business, where, thankfully, he open-sourced much of his work (something he's still doing today).

When he found out that starting his own business wasn't going to work with his immigration status, he was very fortunate to have also caught Apple's eye with the software he had been releasing to the public. Apple bought his company in order to bring him on board. The proceeds of that small sale are now sustaining his next project after going independent again.

After spending five years at Apple struggling to navigate the maze of people and connections and types of expertise in order to get the information he needed, Warden decided to go independent and build a company that solved exactly that kind of problem. "I can't think of a better big company to work for, but it was still a big company," he says. "It was hard to find the right people to talk to, whether for particular expertise or for contacts at external companies." And so Warden left Apple to build a company that would use social graph analysis to solve problems like that. He called the company Mailana.

We've written here a number of times about Mailana's tool that analyzes the social graph of any Twitter user. Enter the username of someone on Twitter and Mailana will show you which 20 other people the user has exchanged the largest number of reciprocal public @ replies with. Find someone interesting or important? Mailana's Twitter analyzer will tell you who they most regularly interact with. See, for example, The Inner Circles of 10 Geek Rockstars on Twitter.

Pulling Down the Facebook Social Graph

Now Warden is about to unveil a much larger project along the same vein. For the past six months he's been crawling public profile pages on Facebook. He now has more than 215 million of them indexed and updated about once a month. When he began he was using the Web crawling service 80legs, but over time he had to build his own crawling infrastructure.

When I talked to him this afternoon, he had already begun uploading 100 GB of user data onto his server to make it available for academic research starting on Wednesday. Warden says he's removed identifying profile URLs but kept names, locations, Fan page lists and partial Friends lists. All those fields of data are just waiting to be analyzed and cross referenced. That's one very rich resource.

Yesterday Warden posted some of his own initial observations from the data on his personal blog. Those included:

• In almost every state in the Southern U.S., God is number one most popular Fan page among Facebook users. Among people in the L.A., San Francisco and Nevada regions? "God hardly makes an appearance on the fan pages, but sports aren't that popular either," Warden writes. "Michael Jackson is a particular favorite, and San Francisco puts Barack Obama in the top spot." In the Oregon and Idaho region? Starbucks is number one.
• In the Mormon-influenced areas of Utah and Eastern Idaho, the most popular Fan pages are The Book of Mormon, Glen Beck and the vampire book Twilight, which was authored by a Mormon.
• The bulk of Warden's posted analysis yesterday was about location networks. People in the western U.S. tend to have Facebook friends all over the country; people in the southern U.S. tend to mostly be friends with people who have remained in the same area.

Taking a Deeper Look

These observations are interesting, but they are only the beginning of what's possible. Name, location, friends and interests are great data points to analyze. Warden has written a program that will estimate gender as well, based on names. All these data points can be cross-referenced with outside data, too. Members of Facebook's own staff did this kind of analysis when they compared user last names to U.S. Census data, which allowed them to estimate changes in Facebook's racial composition over time based on the likelihood of people with particular last names to report a particular racial backgrounds.

"I'm mostly thinking 'What do I try first?'," Warden says. "There's so many interesting ways to slice the data - especially as I'm starting to get changes over time. I'm also trying to map out political networks in aggregate; how polarized the fans of particular politicians are - so how likely a Sarah Palin fan is to have any friends who are fans of Obama, and how that varies with location too. One of my favorite results is that Texans are more likely to be fans of the Dallas Cowboys than God."

Warden says he hasn't talked to anyone from Facebook since he started crawling the site, but he did get an email from someone on the security team asking him to take down instructions he'd posted that exposed a security hole that made harvesting peoples' email addresses easy. So the company is paying attention. "I'd love to see them put me out of business by putting decent data out there," Warden says. He says his Amazon Web Services bill was over $5,000 last month.

Why is he indexing all this content and why is he going to hand it over to the academic world later this week? "I am fascinated by how we can build tools to understand our world and connect people based on all the data we're just littering the Internet with," Warden says.

"Nobody thinks about how much valuable information they're generating just by friending people and fanning pages. It's like we're constantly voting in a hundred different ways every day. And I'm a starry-eyed believer that we'll be able to change the world for the better using that neglected information. It's like an x-ray for the whole country - we can see all sorts of hidden details of who we're friends with, where we live, what we like."

For a great example of the kind of social impact that data analysis can make, Warden points to some of the fascinating ways that GIS data is illuminating the intersection of race and public services. Data has shed light on social injustices for decades, and measurable information about the interactions of hundreds of millions of people every day on Facebook offers opportunities to discover both good and bad news about the contemporary human condition.

Warden says he's not yet been able to interest any investors in his ideas for businesses based on this data, so his girlfriend Liz Baumann, a former insurance actuary, stepped in to help and is now running much of the crawling. He says he's now focused on "working on ways of presenting all this information in a form that answers questions for people willing to pay." His first experiment along those lines is the very interesting FanPageAnalytics.com.

What does Pete Warden hope for from this week's public release of all this Facebook data? "Hopefully I'll get to see a bunch of interesting [academic research] papers come out of it, worst case. And I'd like to be the guy people turn to when they need stuff like this."

Already well-respected among a fringe group of bleeding-edge geeks, we hope that Warden's work on social graph analysis will end up impacting a far larger number of people than may ever know his name.

The iPad is not the transformational device so many Apple enthusiasts were hoping for. It won't turn all the content industries upside down, it won't be your primary computing device, and it's not even a bigger, better iPhone.

Apple CEO Steve Jobs introduced the iPad as a device to fill the gap between smartphones like the iPhone and high-end laptops like the MacBook and MacBook Pro. He said there needs to be a middle device, but it needs to be better than the alternatives at what it does. Netbooks currently fill the void, but according to Jobs, netbooks aren't better at anything. He and his colleagues at Apple believe that the iPad is.

Apple's website and promotional video call the iPad magical. We're told the iPad is the best way to experience the web, email, photos, and videos. Hands down. But it's not it's not even close. It's mighty cool, it's super convenient, and it's very sexy, but it's not even better than a netbook at some of those things.

This isn't the middle device folks have been waiting for because and I'm using Steve Jobs's own criteria here it's not better at anything than any other device on the market. It's a step in that direction, but the day hasn't come yet. Here are just a few of the ways the iPad isn't as magical as Apple claims.

It's Not the Best Way to Browse the Web

Steve Jobs said it needs to be a better web device than the alternatives. The Apple website says it's the best way to experience the web. Some variation of that phrase is repeated several times in the promotional video Apple has released. But it's just not true.

It might be one of the best ways to browse the web on a mobile device, but laptop and desktop computers even netbooks are still better. Most current websites were designed to be experienced on those devices with a mouse and a keyboard. Maybe the mouse isn't necessary, but you don't have to pop up a software keyboard to type in URLs on a netbook or laptop. Even if you lug around the keyboard dock, it will be a tad awkward moving between the keys and the screen to interact. You're sacrificing some usability for simplicity on the iPad.

Most importantly, the iPad's browser does not support Adobe Flash, the foundation of rich media on the web today. Adobe is planning to make it possible for Flash developers to develop apps, but it won't work on the web.

I'll admit that the decision not to support Flash is a logical one if you start at the right premises; Flash is responsible for countless reported crashes on Macs, and Apple can't control it to ensure quality of experience. Apple is banking on a transition to HTML5 and CSS 3 for rich web content. While that transition has already begun, it hasn't fully happened yet. Until it does, it's ridiculous to call this device the best way to experience the web when one of the most ubiquitous and essential web technologies is not supported.

It's an Unprecedented Win for Closed Computing

Many of the software restrictions that drive people mad when they're using the iPhone are going to be just as frustrating on the iPad. All the device's content apps, songs, TV shows, movies, books, you name it can only be processed through Apple's iTunes Store.

You won't be able to drag and drop or share files with other computers like you can with your laptop on your home network. You won't be able to download a program or music file from the web and play it on the spot. You won't be able to use any application that doesn't meet Apple's strict approval guidelines. It's closed computing at its most extreme.

Unfortunately we've come to expect that from our smartphones. For a larger device that's supposed to replace your netbook as a complete portable computing solution, though, this is almost unprecedented at least from a device that's likely to have a great deal of influence on the market and on the design of future devices. That's bad news no matter how you spin it.

It's Not Really a Competitive eReader

The Kindle owns the eReader landscape right now, and the greatest expectation for the iPad was that it would bury the Kindle. While the iPad's reader interface is indisputably sweet-looking and the list of participating publishers is promising, there are several ways it just won't beat the Kindle.

The most important issue is the price. The Kindle costs $260; so do Barnes & Noble's Nook and the comparable Sony Reader. The Kindle even comes bundled with free 3G network access, though it admittedly can't do anywhere near as much with it as the iPad can.

But if you are considering the iPad primarily as a reader, that price difference is a big problem. Also a big problem: The lack of an e-ink display. E-ink doesn't wash your face in eye-strain-inducing light like the displays on the iPhone, the iPad, and laptop computers do. It's meant to be a soft experience, just like reading a book. Without e-ink, you might not be able to tolerate spending four straight hours reading Stephen King's latest on a regular display, cool IPS tech aside.

Finally, as impressive as 10 hours of battery life is for a multi-purpose device like the iPad, the Kindle can run in reading mode for a week without recharging longer if Wi-Fi is disabled. Because it's trying to do everything, the iPad isn't the best at anything.

It's Not Worth It If You Have a Smartphone and Laptop

If the iPad isn't a good option as a middle device, it ought to at least be attractive to power users and enthusiasts who already have other devices. Unfortunately, it's not.

It's not significantly better at anything than either your iPhone or your MacBook. It can't be used as your daily workhorse computer on the go, because just like the iPhone's OS 3.1.2 the iPad's OS 3.2 doesn't multitask. And if you already have an iPhone, you can do basic information gathering, mapping, and so on while you're on the go without spending an additional $29.99 per month for 3G service.

Further, your laptop or netbook very likely has a web cam for video conferencing, and your cell phone probably has a camera (or even video camera) for capturing images. The iPad has neither.

Since the interface is graceful and satisfying, you might want to buy it as an extra device just for the experience, but at between $499 $829, that's not practical for most consumers.

The Anti-Hype

The iPad isn't going to be a phenomenon with either netbook users or power users. It's not better than existing devices at anything, and it's too expensive for most people to use it as a secondary device. I might have said something different if the rumors that the iPad would be all about a new push in the content marketplace were true, but that didn't happen. Instead, we got a cool toy.

Tags: apple, Apple Tablet, ereader, ipad, Kindle, Opinion

By Senior Editor Kris Smith (@croncast)

You need a pile-o-data fast and you got nowhere to get it other than surf, bookmark and beg for interns to copy and paste for you. Where do you turn? Your IT department? Your hackery skills and your shared GoDaddy hosting account for bandwidth? Nah.

80Legs is ready to run a couple miles with your pile of data on their shoulders. You get to pick it up and work with it as you see fit.

Did I mention that they are now offering this as a free service? Well, up to a certain point it is free but for the many is plenty of room to get what they're looking for.

80Legs offers a unique service that will crawl the internet on your behalf and gather data from the links that you provide. They then take this unstructured data and make it available for further refinement to the customer.

Their value proposition lies in the ability to deliver this service efficiently and affordably. Like I said earlier, it would be difficult if not impossible for an individual run a service to crawl 100,000 pages quickly. 80Legs is offering this as a free service now and it's all powered by a 50,000 computer network.

The ability to put the data collection into another companies hands allows developers to think about what to do with the data. By freeing up developers more can be done with the data that is returned to them as they have time to think about new algorithms to run across the dataset.

An example of this would be simple search. Developers with more time could work on creating new layers to search that make it more valuable to the end user. Whether it is integrating advanced search functionality or returning results contextually depending on the page that a user is currently searching from.

If you're interested, the free Basic specs are below. Plus and Premium are listed on their blog.

80Legs Basic Plan:

• Free to use
• Normal crawling speed (up to 1 request/second/domain)
• Access to 80legs Web Portal
• 1 job running at a time
• Up to 100K crawled pages per job
• Low priority in 80legs job queue
• No recurring jobs allowed

[Via VentureBeat]

Related articles by Zemanta

• GoDaddy Referral URLs and ISC codes (startups.com)
• How to Use TipTop for Real Time Market Research (growmap.com)
• Sony Trademarks Qrisoity, Possibly New Premium PSN Service? (1up.com)
• Got Data? How Changing My Social Sharing Workflow Is Making Me Smarter (I Hope). (techstartups.com)

80Legs, 50k Computers and a Web Crawler is a post from: TechStartups.com

By Senior Editor Kris Smith (@croncast)

On my way to the Audience Conference this morning I was fortunate enough to have the bus drop me right in front a Verizon store. The balloons were out and I knew that is meant one thing . . . Droid.

At 7:30 a.m. there was no line and I was able to go straight in the store where the associates outnumbered the patrons. That is saying a lot because in New York mobile stores are always busy.

It was obvious though that the people in the store were nerds of the first order, though. One glance at the khaki pants, bad leggings and dishevled fauxhawks let me know the IT department had arrived. My people.

I was able to avoid ending up on a list to buy the heralded Droid and step right up and start fondling it with my geek gloves. Geek gloves, btw, are similar to kid gloves but they hold a special reverence and care for gadgetry.

After a hands on I can tell you that the device is gorgeous, lighter than you would expect and blazing fast on Verizon's network.

Here's how it went down: I immediately flipped it open to expose the QWERTY keyboard to get my hot thumbs typing in URLs to load up some internet goodness. For sites with Android style sheets it loaded them right up without more than a 2 second delay. For full sites it took a bit longer but seemingly quicker than the iPhone. I know, I should have done some AB testing but the overly chatty associates might have asked me to move it along.

The screen is clear, bright and has a large surface area that is welcome to fat fingers like mine. I was able to press icons with the phone with ease and get them open quickly. Which was a surprise to see them load so quickly.

Since this is a cursory overview of the phone I can say that I have only one gripe the keyboard is shifted left. Which forces the use to type mostly with the left hand. I want two hand glory on this sucker to rip out blog posts.

All in all, I was happy testing it out and probably will pick one up in the next few weeks.

DISCLOSURE OF MATERIAL CONNECTION: http://cmp.ly/0

The Droid Doth Be Here Initial Review is a post from: TechStartups.com

By Senior Editor Kris Smith

Startups, it's time to begin shortening and tracking your own URL's. Taking control of your data is the simplest way of measuring your engagement online next to direct conversations with users. Look at it like taking a survey without asking a single question.

So what?

There is a wealth of data to be collected from that shortened URL that can help you make your product(s) better. Take for instance if your job is to write blog posts and then tweet about them. If you're tracking the data yourself you can measure the most active click-through times and tweet then. Another example might be your desire to track other click-throughs and track the geo-location of your biggest fans.

Other great information to get from this one-click survey is what site did they use to click-through, also known in the biz as referrer and what some might call the holy grail of necessary data for digital product development, user-agent. For when you need to know the application or platform, such as mobile, to better your products. So that's so what.

So how?

There are a couple of ways to get it done. If you want the data and tracking that I described above without having to parse your log files you can use the code samples that I will add below. If you simply are looking for a way to brand your URL shortener and use it on Twitter, then you can follow the discussion and directions in this forum thread for using .htacess on Apache [http://www.webmasterworld.com/forum92/2545.htm].

For those of you ready to get it on with a kick start and build on top of it, here we go. I should mention that this is for those on a LAMP box. Not sure what LAMP means? This isn't for you then. If you do and are on another platform, feel free to follow along for the concepts.

First Get a short URL

The best choice doing this the right way is to figure out something that comes close to your regular URL. Say your site is goldfishbowlpirate.com and you found that gbp.me was available. Then you would snatch it up and your short URL's would look like http://gbp.me/1234. Make sure you host the account on a LAMP server.

Second Edit the .htaccess file

Simple copy and past job into your .htaccess file will get you on the road to tracking your click-throughs like a champ.

code: RewriteEngine On
RewriteBase /
RewriteRule ^[0-9] /

Third MySQL table

You'll need to setup a couple of tables: one to store the URL's that need to be redirected and one to store the click-through data.

The base table

CREATE TABLE IF NOT EXISTS `url_base` (
`url_id` int(11) NOT NULL auto_increment,
`url_key` varchar(6) NOT NULL default '',
`url_value` varchar(255) NOT NULL default '',
`bookmarklet` tinyint(1) NOT NULL default '0',
`user_host` varchar(255) NOT NULL default '',
`create_time` timestamp NOT NULL default CURRENT_TIMESTAMP,
PRIMARY KEY  (`url_id`),
KEY `url_key` (`url_key`)
) ENGINE=MyISAM  DEFAULT CHARSET=latin1;

The clicks table

CREATE TABLE IF NOT EXISTS `url_clicks` (
`hit_id` int(11) NOT NULL auto_increment,
`url_id` int(11) NOT NULL default '0',
`hit_host` varchar(255) NOT NULL default '',
`hit_time` timestamp NOT NULL default CURRENT_TIMESTAMP,
`hit_referer` varchar(255) NOT NULL default '',
`hit_agent` varchar(255) NOT NULL default '',
PRIMARY KEY  (`hit_id`),
KEY `hit_agent` (`hit_agent`)
) ENGINE=MyISAM  DEFAULT CHARSET=latin1;

Fourth Code it up

This is where the magic truly happens. We need code to create shortened URL's, handle the redirects by looking up the shortened URL in the url_base table and track the click-through in the url_clicks table.

I've created a zip file that has all the necessary files in it and the SQL above for download. Also Included in the zip are the .htaccess file and four PHP scripts.

1. config.php A configuration file
2. index.php To act as the index incase there is an error with the redirect
3. makeit.php To make shortened URL's. It can be called by other programs like an API
4. makeit_b.php To make shortened URL's for a Bookmarklet for easy testing and copy and paste. To use the Bookmarklet simply right-click on your bookmarks toolbar and paste the code below into the Location' section. Make sure to swap out yourshorturl' with your own!javascript:(function(){var%20a=window,b=document,c=decodeURIComponent,d=a.open('http://yourshorturl/makeit_b.php?url='+c(b.location),"bkmk_popup","left="+((a.screenX||a.screenLeft)+10)+",top="+((a.screenY||a.screenTop)+10)+",height=145px,width=200px,resizable=1,alwaysRaised=1");a.setTimeout(function(){d.focus()},300)})();

Now that you've got all the code you need to begin shortening and tracking your own URL's you can take a look at a sample of a very simple admin page. I've been using this system of shortening for nearly two years for all of my Twitter URL's.

A special thanks to Mike Marusin for writing the first version of this code for his own personal URL shortener back in 2007.

DISCLOSURE OF MATERIAL CONNECTION: http://cmp.ly/4

Post from: TechStartups.com

Tweet This Post

UPDATE: For the latest on this story, see Confirmed: Digg Just Hijacked Your Twitter Links

The social news site Digg found itself criticized earlier this year after the release of the DiggBar and Digg short URLs, which some said stole traffic and pagerank from publisher sites to increase Digg's pageviews.

Now Digg faces a new accusation: that it has, either accidentally or on purpose, changed the behavior of these URLs to send logged-out users to Digg.com in preference to the publisher sites. We were able to verify that Digg is indeed redirecting Digg URLs to its own site.

DiggBar Controversy

First, a little background. At launch, Digg URLs provided an alternative to popular URL shorteners like bit.ly and TinyURL (commonly used to save characters on Twitter), except that the links loaded publisher sites in framed pages on Digg.com. Some claimed that this was a way to build traffic to Digg while hurting the search engine traffic provided to publishers.

After a firestorm in the SEO community that lead to some sites adding framebreakers to prevent Digg framing their sites, Digg relented and decided to only frame pages if the user was logged in to Digg at the time.

New Digg URL Behavior: Redirects Traffic to Digg.com

This week Digg users have noticed an odd change in the way Digg URLs work: for logged out users, they no longer go to the site they link to. Instead, the links go to the Digg.com page for that story, provided it has already been submitted to Digg. The result? The thousands of short links that people are trying to create to their favorite websites are instead redirecting their followers to a Digg landing page.

The blog EndofWeb, which appears to have spotted the issue first, calls this a bait and switch operation.

You can try it out for yourself: choose any webpage URL and place Digg.com/ at the front of it to create a Digg URL. While this used to create a link that redirects to the original story, it now simply directs visitors to the Digg.com landing page for that story.

Intentional Change or Mistake?

If intentional, the move is likely to sour Digg's relationships with publishers: Digg became popular based on its ability to drive traffic to publisher sites, but the DiggBar showed Digg's intent to retain more of that traffic on its own site.

If a mistake, it's likely to damage trust in URL shorteners: users want to be sure that when they create a link, it'll send visitors to the intended destination. When that process fails, it hurts confidence.

We've reached out to Digg via email for more information.

What do you think? Are your Digg URLs directing to Digg.com?

Fever and the Future of Feed Readers

Time was, every self-respecting geek lived and died by his feed reader (or aggregator, if you prefer). Just several years ago, the number of subscriptions in your RSS-chomping tool of choice made for bragging rights. 200? Oh, I can get through 500 feeds a day. More subscriptions meant you were more in the know. Really good lists of subscriptions were traded amongst friends, but cautiously, just as one might hold back a recommendation to a superb but little-known restaurant.

At the time, the only real debate was around the best way to present all this information. Some preferred a river of news, others preferred their content categorized and neatly filed, like sections in a newspaper. But everyone was in agreement: having all this fresh content collected for you in one place was a boon. It was a change in mindset, and it seeded the demand for what is now being called the Real-Time Web. (Incidentally, the Real-Time Web is next year's Web 2.0. If you'd like to appear cool and aloof, start disdaining the expression now).

Today, at least in the web-tech echo chamber, feed reading is quickly falling out of fashion. Too many sites producing too many feeds of dubious quality means information overload, and a creeping sense of obligation to keep up with a torrent of questionably relevant content. Some have gone back to checking a handful of bookmarked sites, as we did in the early days of the web. Others rely on social aggregation sites like Reddit, Digg, and Hacker News to show them what's worth reading. Both strategies are highly manual and, to me, distressingly unoptimized.

Abdicating Aggregation

Another camp all but eschews the idea of trying to keep up with feeds. Chris Wanstrath, co-founder of the superb social coding site GitHub, is one of the more visible advocates of this approach, saying in a tech conference keynote last year:

Stop using Google Reader or NetNewsWire or whatever the kids are using these days. It's not worth your time. [L]et other people do the filtering for you. Use your time for other things.

This statement initially rings true. We're in the age of social networking, after all. I've told social sites about my friends, and my friends are always talking about things, so just show me what my friends are talking about and I'll always be in the loop, right? Then I can focus on my own interests and projects. Sounds great.

The problem with abdicating your content consumption to other people, though, is other people. Perhaps it's overestimating my ability to find interesting things to read, but I don't trust my friends and the Internet at large to educate and entertain me. In the venn diagram of my interests and my friends', there may be 80% overlap, but most of the content that I'm going to find deeply engaging is probably in the leftover 20% at the margins.

There's also a sort of collective danger to the strategy of exclusively consuming information through social osmosis: if everyone does it, who's going to find the interesting stuff? Who takes the reigns as the editors, the arbiters of taste? Going back to a post I wrote in 2003, who will be our cool shit aggregators?

If everyone took Wanstrath's advice, nobody would do any filtering and nobody would consume anything. Realistically, we're in no danger of that, but we're also not seeing a radical improvement in the way we consume information on the web. Surely someone's investigating another strategy?

Blending Subscriptions with Social Data

Google Reader is, as evidence of the slowly dying field of feed reading, pretty much the only regularly-updated, widely-used aggregator left on the web. Bloglines has been gasping for air for over a year, and NewsGator is positioning itself towards the enterprise, presumably trying to scrape some money out of the generally unprofitable business of aggregation.

Reader has been something of a playground for Google, and one of the products for which the behemoth has been most responsive to public feedback. When Reader launched, its interface was nigh-unusable. It was updated, improved, and gradually became the only feed reader worth using and not just on the web, something it pains me to say as the owner of licenses for multiple desktop aggregators that eventually had their price driven down to free, and have since seen little attention from their developers.

Today, Google seems hellbent on cramming its otherwise clean and speedy products with cumbersome, poorly conceived social features. Presumably they see social networks as a threat to their valuable side business of, uh, completely free products, and this is their ham-fisted response. In Reader's case, the user response has been one of confusion and derision.

Seeing content filtered through my social lens seems like the marriage of traditional feed reading to Wanstrath's more osmotic approach. Reader's implementation doesn't prove this to be a happy union. The tool is now cluttered with smilie faces indicating content that my friends liked, only Google has fairly incomplete view of who my friends are because they've yet to create a social experience that encourages me to share that information. Reader's myriad competing ways to share, vote on, annotate, and remember items further detract from its former appeal.

I've given up on Reader, but I'm not ready to give up on feed reading just yet. I wanted to try one more experiment.

Enter Fever

Fever is a feed reader designed and built by Shaun Inman, the developer behind the popular Mint web traffic analytics product. Like Mint, Fever is $30 (USD) and runs on your server a ballsy proposition in an age of free software running in the proverbial cloud. It is unapologetically for power users.

Fever's proposition is straightforward: supply it with the feeds you always want to read, and supplement those with feeds that you only want to read the juicy bits of. Fever will then show you a sort of personal Techmeme or Google News, pulling together stories that reference common URLs. Fever's precise formula for this isn't discussed on the product's relatively curt homepage. Take it or leave it.

I forked over my money, spun up a virtual server, and have been using Fever for several days now. Installation was as straightforward and slick as you could hope for given that Fever is a self-hosted web application. Special features aside, it handles the basics well imagine Google Reader before all the social bloat and with a far more attractive design. Fever's design is not perfect, but it's easy on the eyes and pleasant to use. Put another way, Fever doesn't make it harder to read feeds much as you always have.

The $30 question, though: does Fever really float the best, most relevant content to the top in a personalized way? Can it dig through all the noise on the web and show you what you need/want to know at a glance? The free answer: sort of.

For starters, it's easy to pollute your corpus of signal feeds, which Fever calls sparks. Fever needs sparks that contain a lot of links. If you put top feeds from Digg, Reddit, and the like into Fever, you'll basically just end up with your own dim, mostly irrelevant slice of the web. Fever really needs folks like Waxy, Laughing Squid, and Trivium to keep churning out link blogs full of references to good content. Without those sort of quality, URL-rich feeds, your Fever's view of what's hot is going to be lukewarm.

For this reason, Fever is just fine for floating good techie content to the top, but poor for most any other subject. I'd love it if Fever could find me good posts from the set of minimal techno or cocktail blogs I subscribe to, but link blogs and, indeed, linking outside one's own site just aren't as prevalent in those communities. Fever did similarly poorly given a number of sparks for top world news; a paucity of URLs means Fever can't replace Google News for figuring out what's on the front pages of the world's newspapers.

It's disappointing that I can't depend on Fever to be a one-stop shop for my daily information intake. With my current heavily-curated collection of subscriptions, I can rely on Fever to be a sort of no-bullshit Techmeme, but little more. For the topics of world news, music, art, culture, humor, food, and drink, I still need to read a number of feeds entry-by-entry.

Given Fever's initial cost, plus the ongoing cost of hosting a server on which to run it, I can't imagine that it's a tool that will last long in my tool belt. I already regret the time I spent setting it up and tuning my feeds, and I can't really justify keeping it around for the sole purpose of being a less-encumbered Google Reader.

The Future of Feed Readers

I'm not sure what the solution is here. Feed readers as we've known them are dying, but it's as yet unclear what will take their place. Filtering feeds for relevance algorithmically seems all but fruitless; filtering through the social graph is only a slight improvement, but misses the rare content that may only strike a chord with a small audience.

If there's one thing I'm convinced of at the end of this exploration, it's that there's more work to be done, and more businesses to emerge in this field. Social networks alone aren't focused enough tools to bubble up and share quality content. My hope is that a surplus open data of the sort we're trying hard to share at Twitter will help spawn a new generation of tools to manage the flood of content. I don't think it's a problem that Twitter, or any other pipeline for information, can solve on its own.

With all that said, perhaps the right approach really is to abdicate one's consumption of content to whatever you're passively exposed to, and to occupy your mind with other things. The act of creation is almost always self-affirming, and the act of consumption so rarely is.

Reader recently launched liking (and a bunch of other features). One of the nice things about liking is that it's completely public*. It would therefore make sense to be pretty liberal with liking data, and in fact Reader does try to expose liking in our feeds. If you look at my shared items feed you will see a bunch of entries like:

<gr:likingUser>00298835408679692061</gr:likingUser>
<gr:likingUser>11558879684172144796</gr:likingUser>
<gr:likingUser>07538649935038400809</gr:likingUser>
<gr:likingUser>09776139491686191852</gr:likingUser>
<gr:likingUser>02408713980432217881</gr:likingUser>
<gr:likingUser>05429296530037195610</gr:likingUser>

These are the users that have liked. Users are represented by their IDs, which you can use to generate Reader shared page URLs. More interestingly, you can plug these into the Social Graph API to see who these users are.

Liking information isn't just limited to Reader shared item feeds. If you use Reader's view of a feed, for example The Big Picture's, you can see the <gr:likingUser> elements there too. This means that as a publisher you can extract this information and see which of your items Reader users find interesting.

For now liking information that is included inline in the feed is limited to 100 users, mainly for performance reasons. That number may go up (or down) as we see how this feature is used.

* I've seen some wondering what the difference between liking, sharing and starring is. To some degree that's up to each user, but one nice thing about liking is that it has less baggage associated with it. We learned that if we try to redefine existing behaviors (like sharing) users get upset.

My recent computer woes led to some corruption that makes python no longer run on my MacBook. This means that I can't use Juice as my podcatcher anymore. To be honest, I've been using Juice for years without ever liking it but without much of an alternative since I refuse to use iTunes as my podcatcher. In a way, losing python was a positive because it forced me off the fence and into looking for a better alternative.

Luckily, I found it first try. I decided to try out Linc Fessenden's bashpodder. It's a 50 line bash script that takes a simple text file of feed URLs and fetches them. No muss, no fuss, no BS. RSS feeds in, podcasts out. I like that. There are now many variations as hackers have fiddled with the functionality, but I'm running the core vanilla mainline version. This one collects together shows into a date based directory. Because of the way it is using wget to fetch the actual files, in most cases it preserves the timestamp of the server version of the file. This actually helps me out a lot in my attempts to listen to shows in chronological order. I did make my own little hack to it, changing where it does the logging of a show URL to the history. The original script does it unconditionally, I have it check the exit code of wget and only put it in the history if that was successful. This way, a failed download will retry later.

Switching from one podcatcher to another is always a bit dicey at first. Since some of these feeds do the insane thing of keeping hundreds of episodes in them, if you aren't careful bashpodder will fetch every one of those and fill up your hard drive. Here's how I handled the transition. It was a bit labor intensive and required me watching it, but after the first run everything was perfect. The thing to be aware of is that there are two files podcast.log and temp.log. The first is the permanent list of fetched files, the second is a working copy and at the end of the run the two are combined, duplicates filtered and the whole thing resaved to podcast.log. As files are fetched, it checks to see if an URL is in podcast.log and if it is, bashpodder skips it.

I ran the script from my MacBook in a terminal window. I ran it via:

sh -x bashpodder.shell

so that it was outputting all of its variables as it worked. When it would get to a new feed, it would splat out the list of file URLs that were parsed out of the RSS feed. I'd copy the files from the list I didn't want downloaded and just put them directly into podcast.log via a file editor. You can be somewhat sloppy with this. When in doubt I let it fetch the file and I'd delete it later. If the URL goes into podcast.log more than once, no problem. It will get taken care of later. This required me riding the script for 45 minutes or so, but I mostly got the old shows into podcast.log manually. After the first run succeeded, I ran the script one more time. It fetched a few at the edges that I missed but then was completely caught up. I deleted files that I knew I had already listened to and away I went.

Now when I run it, I get only the new files. They go into that day's directory, they sort themselves out somewhat by timestamp. I set up a cronjob to run this at 5 AM and now I'm in business. All the scripts that I use to put the files on my Insignia MP3 player work fine with the new directory structure and I'm back in business. Thanks Linc. This workflow is better than what I had, I no longer have Juice bogging down my machine and eating a lot of memory to do this simple task, and the whole thing runs in a simple bash process that I'm comfortable modifying if I want to. Right on.