Youth social networking researcher danah boyd has observed that many people presume the way they use social networks is the way everyone uses them. "I interviewed gay men who thought Friendster was a gay dating site because all they saw were other gay men," she says. "I interviewed teens who believed that everyone on MySpace was Christian because all of the profiles they saw contained biblical quotes. We all live in our own worlds with people who share our values and, with networked media, it's often hard to see beyond that."

Now picture our perspective leaving our own experiences, zooming out and up until we can see how all the different groups are interacting on a worldwide social network. That bird's-eye view could be both beautiful and horrible if the resolution was clear enough. That's what a Ramen-eating, ex-Apple engineer named Pete Warden is about to release to the public this week.

Sponsor

This Wednesday, Warden will make Friend, Fan page and name data from hundreds of millions of Facebook users available to the academic research community. It's a move that Facebook has to have seen coming, a move that many in the data-centric community have been calling on the company itself to do for years, and an event that's been complicated by Facebook's recent privacy policy changes, which have muddied the waters of right and wrong but rendered even more data available for outside analysis.

If what people call Web 2.0 was all about creating new technologies that made it easy for everyday people to publish their thoughts, social connections and activities, then the next stage of innovation online may be services like recommendations, self and group awareness, and other features made possible by software developers building on top of the huge mass of data that Web 2.0 made public. It's a very exciting future, and Warden is about to fire one of the earliest big shots in that direction.

Nerds in Space: Social Graph Analysis For Solving Large-Group Problems

Warden studied Computer Vision in college in the U.K., then got into game development. After moving to L.A., he spent six years building graphics drivers for the original Playstation and the XBox. Then he started his own independent business, where, thankfully, he open-sourced much of his work (something he's still doing today).

When he found out that starting his own business wasn't going to work with his immigration status, he was very fortunate to have also caught Apple's eye with the software he had been releasing to the public. Apple bought his company in order to bring him on board. The proceeds of that small sale are now sustaining his next project after going independent again.

After spending five years at Apple struggling to navigate the maze of people and connections and types of expertise in order to get the information he needed, Warden decided to go independent and build a company that solved exactly that kind of problem. "I can't think of a better big company to work for, but it was still a big company," he says. "It was hard to find the right people to talk to, whether for particular expertise or for contacts at external companies." And so Warden left Apple to build a company that would use social graph analysis to solve problems like that. He called the company Mailana.

We've written here a number of times about Mailana's tool that analyzes the social graph of any Twitter user. Enter the username of someone on Twitter and Mailana will show you which 20 other people the user has exchanged the largest number of reciprocal public @ replies with. Find someone interesting or important? Mailana's Twitter analyzer will tell you who they most regularly interact with. See, for example, The Inner Circles of 10 Geek Rockstars on Twitter.

Pulling Down the Facebook Social Graph

Now Warden is about to unveil a much larger project along the same vein. For the past six months he's been crawling public profile pages on Facebook. He now has more than 215 million of them indexed and updated about once a month. When he began he was using the Web crawling service 80legs, but over time he had to build his own crawling infrastructure.

When I talked to him this afternoon, he had already begun uploading 100 GB of user data onto his server to make it available for academic research starting on Wednesday. Warden says he's removed identifying profile URLs but kept names, locations, Fan page lists and partial Friends lists. All those fields of data are just waiting to be analyzed and cross referenced. That's one very rich resource.

Yesterday Warden posted some of his own initial observations from the data on his personal blog. Those included:

• In almost every state in the Southern U.S., God is number one most popular Fan page among Facebook users. Among people in the L.A., San Francisco and Nevada regions? "God hardly makes an appearance on the fan pages, but sports aren't that popular either," Warden writes. "Michael Jackson is a particular favorite, and San Francisco puts Barack Obama in the top spot." In the Oregon and Idaho region? Starbucks is number one.
• In the Mormon-influenced areas of Utah and Eastern Idaho, the most popular Fan pages are The Book of Mormon, Glen Beck and the vampire book Twilight, which was authored by a Mormon.
• The bulk of Warden's posted analysis yesterday was about location networks. People in the western U.S. tend to have Facebook friends all over the country; people in the southern U.S. tend to mostly be friends with people who have remained in the same area.

Taking a Deeper Look

These observations are interesting, but they are only the beginning of what's possible. Name, location, friends and interests are great data points to analyze. Warden has written a program that will estimate gender as well, based on names. All these data points can be cross-referenced with outside data, too. Members of Facebook's own staff did this kind of analysis when they compared user last names to U.S. Census data, which allowed them to estimate changes in Facebook's racial composition over time based on the likelihood of people with particular last names to report a particular racial backgrounds.

"I'm mostly thinking 'What do I try first?'," Warden says. "There's so many interesting ways to slice the data - especially as I'm starting to get changes over time. I'm also trying to map out political networks in aggregate; how polarized the fans of particular politicians are - so how likely a Sarah Palin fan is to have any friends who are fans of Obama, and how that varies with location too. One of my favorite results is that Texans are more likely to be fans of the Dallas Cowboys than God."

Warden says he hasn't talked to anyone from Facebook since he started crawling the site, but he did get an email from someone on the security team asking him to take down instructions he'd posted that exposed a security hole that made harvesting peoples' email addresses easy. So the company is paying attention. "I'd love to see them put me out of business by putting decent data out there," Warden says. He says his Amazon Web Services bill was over $5,000 last month.

Why is he indexing all this content and why is he going to hand it over to the academic world later this week? "I am fascinated by how we can build tools to understand our world and connect people based on all the data we're just littering the Internet with," Warden says.

"Nobody thinks about how much valuable information they're generating just by friending people and fanning pages. It's like we're constantly voting in a hundred different ways every day. And I'm a starry-eyed believer that we'll be able to change the world for the better using that neglected information. It's like an x-ray for the whole country - we can see all sorts of hidden details of who we're friends with, where we live, what we like."

For a great example of the kind of social impact that data analysis can make, Warden points to some of the fascinating ways that GIS data is illuminating the intersection of race and public services. Data has shed light on social injustices for decades, and measurable information about the interactions of hundreds of millions of people every day on Facebook offers opportunities to discover both good and bad news about the contemporary human condition.

Warden says he's not yet been able to interest any investors in his ideas for businesses based on this data, so his girlfriend Liz Baumann, a former insurance actuary, stepped in to help and is now running much of the crawling. He says he's now focused on "working on ways of presenting all this information in a form that answers questions for people willing to pay." His first experiment along those lines is the very interesting FanPageAnalytics.com.

What does Pete Warden hope for from this week's public release of all this Facebook data? "Hopefully I'll get to see a bunch of interesting [academic research] papers come out of it, worst case. And I'd like to be the guy people turn to when they need stuff like this."

Already well-respected among a fringe group of bleeding-edge geeks, we hope that Warden's work on social graph analysis will end up impacting a far larger number of people than may ever know his name.

By Senior Editor Kris Smith (@croncast)

I'm hesitant to use the term brand' since it is overused right now and a bit nebulous, especially when it comes to humans. I guess if your name was a brand then it would fit . . . but there thousands of brands with the same name, so maybe not.

Over the last few years with the rise of social everything online many people have arrived without the experience of the growing pains that made the market ready for these sites. With that, the newbies have arrived at the front gate to deliver gems on Facebook like, In hell right now, that would be work! Way to go genius. Don't get run over by the forklift you rode in on.

It has been my desire for years that there were be some licensing that has to happen before people could hop online and make the same mistakes that people were making a decade ago. An education center would be fitting like community annex night courses. Even cabbies gotta go to school.

I thought that since tomorrow is Thanksgiving and you might engage face to face with some people that aren't so savvy, that you could share some words of internet wisdom with them. It. Is. Your. Job. To save them. From. Themselves. Share these of course after you have helped them remove 30 GB of kitty photos from the desktop of their Windows Me eMachine. Okay?

1. Don't use Facebook or MySpace or Twitter at work
2. Don't talk about your boss on the computer
3. Don't ever upload pictures of yourself or Mom drunk
4. Don't put pictures of grandma drunk on the internet
5. Don't take photos of anything naked and upload it
6. Don't threaten the bully from your 8th grade class on Facebook. We know he didn't deserve to succeed in life but he did. Let it go.
7. Don't contact anyone that you hooked up back in the day that Mom knows. That goes for you too Mom. Forget they exist and use your memories remember their sweet smell and freak moves.
8. Don't tell your neighbor that his spouse is cheating on Facebook. Just don't.
9. Don't type anything into your computer that would cause Santa to leave you a lump of coal
10. Don't evah, evah, evah upload a photo of your cat to the internet and caption it. You will never get hired for a job again especially if I am the head of HR

That's it. Now you can spend your Thanksgiving in peace after you lay down the law to your in-laws about Managing Yourself Online'. Cuz the internet is a job.

DISCLOSURE OF MATERIAL CONNECTION: http://cmp.ly/0

Managing Yourself Online: A Thanksgiving Guide is a post from: TechStartups.com

By Senior Editor Kris Smith (@croncast)

Beyond joining the cool kids at the table again, Facebook is ready to begin stepping on the toes of its biggest competitors, Google and Twitter. And they are positioning themselves to put more nails in the MySpace coffin.

Facebook has been rejuvenated by the release of new products, interface improvements and acquisition of FriendFeed. A service much loved by brand name nerds that were able to grok the arcane interface and discussion method.

Facebook is now positioned (with some tweaks) to stunt Twitter's growth, enter real time search as the dominate player and extend its platform much further than Google or Twitter through its API.

Stunting Twitter's Growth

Facebook and its audience are maturing. And this maturation makes it more tolerable for the cool kids to come back to the service. Cool kids in this case being those that were first to jump on board and when the unwashed digital masses began showing up they jumped ship to avoid those newbies getting Dooced for moronic updates.

The audience maturation shouldn't overshadow the growth of Facebook as a more robust platform for messaging. With changes in chat, fan pages, the release of Facebook Lite and snatching up FriendFeed they are acting on a strategy to better control ads that provide a higher ROI for advertisers, removing excess interface components to run a lean set of products like Lite and integrating the core of an API that can power a massive real time search engine.

All of these capitalize on growing an audience by word of mouth and new services build on top of the Facebook platform. With both this combined maturation it will be harder for Twitter to justify the lack of features that it offers as a core service. No payloads of video or audio, chat or multiple user pages for the same account. Today's users expect these features and more like social search.

Real time Social Search

Facebook has done a great job of making recommendations of people to friend and groups to join. They track every bit of activity that you perform on the site, analyze it and then offer up customized recommendations. Add the FriendFeed engine to this and you have an extremely powerful real time social search utility.

Users on Facebook are heading to the front lines, where the cool kids have been for years, to fight the battle against an ever expanding delta of information. The best weapon to use in this battle is created by harnessing the power of each users friend base to narrow incoming river of information . . . to begin with. By using this select group of trusted confidants the delta begins to shallow and offer more valuable focused results.

When users are selecting who they are friends with, they have assigned trust of some varying level in the individual and the type information that they will be sharing. Many times friends or colleagues or members of other hobby groups will share information accordingly to their interests in updates and on their account pages. Real time social search depends on this trust to deliver the most relevant items to a searcher. Facebook has this built in as a core feature.

The I/O of Data, Ads and Opportunity

It is hard to know where to begin. Facebook recently updated their Connect feature which simplified their API somewhat but it still relies on Facebook Markup Language (FBML) a proprietry list of namespaces and commands that interact with Facebook's platform.

I fully expect that Facebook will be releasing a version of their API that conform with the ease of use that existed with FriendFeed's. The FriendFeed API was more akin to that of Twitter than their own. The Twitter API is simple to use, open very powerful.

The release of a simpler API that delivers data in JSON or XML to satisfy developers on different platforms would strengthen all third party tools. It could open Facebook's platform to a greater influx of user generated data to power robust social search and ad inventory. Which is what will keep the ship afloat.

There has been some discussion recently about charging for API usage versus paying developers for using your API. Evidence suggests that the latter is the winning choice to build a strong developer network. Facebook is in a position to drive this type of revenue stream and turn it into the AdSense of the developer world.

Conclusion

Facebook is in a position like no other company to capitalize on paying developers a slice of the advertising pie for including targeted ads in their applications. And if they can be delivered with the content that Facebook is sending developers in a way that allows devs to enhance or choose better, more targeted ads for their applications audience, Facebook would be walking into El Dorado with a smelter the size of the Rhode Island on their back.

For now they have the ability to step on some toes, be it pretty hard. In the coming months and long-term, look for them to begin breaking backs by acquiring properties to freeze out competitors and adding new features for social search.

DISCLOSURE OF MATERIAL CONNECTION: http://cmp.ly/0

Facebook Ready To Step On Toes is a post from: TechStartups.com

Database normalization is a technique for designing relational database schemas that ensures that the data is optimal for ad-hoc querying and that modifications such as deletion or insertion of data does not lead to data inconsistency. Database denormalization is the process of optimizing your database for reads by creating redundant data. A consequence of denormalization is that insertions or deletions could cause data inconsistency if not uniformly applied to all redundant copies of the data within the database.

Why Denormalize Your Database?

Today, lots of Web applications have "social" features. A consequence of this is that whenever I look at content or a user in that service, there is always additional content from other users that also needs to be pulled in to page. When you visit the typical profile on a social network like Facebook or MySpace, data for all the people that are friends with that user needs to be pulled in. Or when you visit a shared bookmark on del.icio.us you need data for all the users who have tagged and bookmarked that URL as well. Performing a query across the entire user base for "all the users who are friends with Robert Scoble" or "all the users who have bookmarked this blog link" is expensive even with caching. It is orders of magnitude faster to return the data if it is precalculated and all written to the same place.

This is optimizes your reads at the cost of incurring more writes to the system. It also means that you'll end up with redundant data because there will be multiple copies of some amount of user data as we try to ensure the locality of data.

A good example of a Web application deciding to make this trade off is the recent post on the Digg Blog entitled Looking to the Future with Cassandra which contains the following excerpt

The Problem

In both models, we're computing the intersection of two sets:

1. Users who dugg an item.
2. Users that have befriended the digger.

The Relational Model

The schema for this information in MySQL is:

CREATE TABLE `Diggs` ( `id` INT(11), `itemid` INT(11), `userid` INT(11),
`digdate` DATETIME, PRIMARY KEY (`id`), KEY `user` (`userid`), KEY `item` (`itemid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;   CREATE TABLE `Friends` ( `id` INT(10)
AUTO_INCREMENT, `userid` INT(10), `username` VARCHAR(15), `friendid` INT(10), `friendname`
VARCHAR(15), `mutual` TINYINT(1), `date_created` DATETIME, PRIMARY KEY (`id`), UNIQUE
KEY `Friend_unique` (`userid`,`friendid`), KEY `Friend_friend` (`friendid`) ) ENGINE=InnoDB
DEFAULT CHARSET=utf8;

The Friends table contains many million rows, while Diggs holds hundreds of millions. Computing the intersection with a JOIN is much too slow in MySQL, so we have to do it in PHP. The steps are:

1. Query Friends for all my friends. With a cold cache, this takes around 1.5 seconds to complete.
2. Query Diggs for any diggs of a specific item by a user in the set of friend user IDs. This query is enormous, and looks something like:

   SELECT
   `digdate`, `id` FROM `Diggs` WHERE `userid` IN (59, 9006, 15989, 16045, 29183, 30220,
   62511, 75212, 79006) AND itemid = 13084479 ORDER BY `digdate` DESC, `id` DESC LIMIT
   4;

   The real query is actually much worse than this, since the IN clause contains every friend of the user, and this can balloon to hundreds of user IDs. A full query can actually clock in at 1.5kb, which is many times larger than the actual data we want. With a cold cache, this query can take 14 seconds to execute.

Of course, both queries are cached, but due to the user-specific nature of this data, it doesn't help much.

The solution the Digg development team went with was to denormalize the data. They also went an additional step and decided that since the data was no longer being kept in a relational manner there was no point in using a traditional relational database (i.e. MySQL) and instead they migrated to a non-RDBMS technology to solve this problem.

How Denormalization Changes Your Application

There are a number of things to keep in mind once you choose to denormalize your data including

1. Denormalization means data redundancy which translates to significantly increased storage costs. The fully denormalized data set from the Digg exampled ended up being 3 terabytes of information. It is typical for developers to underestimate the data bloat that occurs once data is denormalized.

2. Fixing data inconsistency is now the job of the application. Let's say each user has a list of the user names of all of their friends. What happens when one of these users changes their user name? In a normalized database that is a simple UPDATE query to change a single piece of data and then it will be current everywhere it is shown on the site. In a denormalized database, there now has to be a mechanism for fixing up this name in all of the dozens, hundreds or thousands of places it appears. Most services that create denormalized databases have "fixup" jobs that are constantly running on the database to fix such inconsistencies.

The No-SQL Movement vs. Abusing Relational Databases for Fun & Profit

If you're a web developer interested in building large scale applications, it doesn't take long in reading the various best practices on getting Web applications to scale such as practicing database sharding or eschewing transactions before it begins to sound like all the advice you are getting is about ignoring or abusing the key features that define a modern relational database system. Taken to its logical extreme all you really need is a key<->value or tuple store that supports some level of query functionality and has decent persistence semantics. Thus the NoSQL movement was borne.

The No-SQL movement is a used to describe the increasing usage of non-relational databases among Web developers. This approach has initially pioneered by large scale Web companies like Facebook (Cassandra), Amazon (Dynamo) & Google (BigTable) but now is finding its way down to smaller sites like Digg. Unlike relational databases, there is a yet to be a solid technical definition of what it means for a product to be a "NoSQL" database aside from the fact that it isn't a relational database. Commonalities include lack of fixed schemas {TODO}. Below is a list of some of the more popular NoSQL databases that you can try today along with a brief description of their key qualities

1. CouchDB: A document-oriented database where documents can be thought of as JSON/JavaScript objects. Creation, retrieval, update and deletion (CRUD) operations are performed via a RESTful API and support ACID properties. Rich querying is handled by creating Javascript functions called "Views" which can operate on the documents in the database via Map/Reduce style queries. Usage: Although popular among the geek set most users seem to be dabblers as opposed to large scale web companies.

2. Cassandra: A key-value store where each key-value pair comes with a timestamp and can be grouped together into a column family (i.e. a table). There is also a notion of super columns which are columns that contain whose values are a list of other key-value pairs. Cassandra is optimized to be always writable and uses eventual consistency to deal with the conflicts that inevitably occur when a distributed system aims to be always writable yet node failure is a fact of life. Querying is available via the Cassandra Thrift API and supports fairly basic data retrieval operations based on key values and column names. Usage: Originally developed and still used at Facebook today. Digg and Rackspace are the most recent big name adopters.

3. Voldemort: Very similar to Cassandra which is unsurprising since they are both inspired by Amazon's Dynamo. Voldemort is a key-value store where each key value pair comes with a timestamp and eventual consistency is used to address write anomalies. Values can contain a list of further key value pairs. Data access involves creation, retrieval and deletion of serialized objects whose format can be one of JSON, strings, binary BLOBs, serialized Java objects and Google Protocol Buffers. Rich querying is non-existent, simple get and put operations are all that exist. Usage: Originally developed and still used at LinkedIn.

There are a number of other interesting NoSQL databases such as HBase, MongoDB and Dynomite but the three above seem to be the most mature from my initial analysis. In general, most of them seem to be a clone of BigTable, Dynamo or some amalgam of ideas from both papers. The most original so far has been CouchDB.

An alternative to betting on a speculative database technologies at varying levels of maturity is to misuse an existing mature relational database product. As mentioned earlier, many large scale sites use relational databases but eschew relational features such as transactions and joins to achieve scalability. Some developers have even taken that practice to an extreme and built schema-less data models on top of traditional relational database. A great example of this How FriendFeed uses MySQL to store schema-less data which is a blog post excerpted below

Lots of projects exist designed to tackle the problem storing data with flexible schemas and building new indexes on the fly (e.g., CouchDB). However, none of them seemed widely-used enough by large sites to inspire confidence. In the tests we read about and ran ourselves, none of the projects were stable or battle-tested enough for our needs (see this somewhat outdated article on CouchDB, for example). MySQL works. It doesn't corrupt data. Replication works. We understand its limitations already. We like MySQL for storage, just not RDBMS usage patterns.

After some deliberation, we decided to implement a "schema-less" storage system on top of MySQL rather than use a completely new storage system.

Our datastore stores schema-less bags of properties (e.g., JSON objects or Python dictionaries). The only required property of stored entities is id, a 16-byte UUID. The rest of the entity is opaque as far as the datastore is concerned. We can change the "schema" simply by storing new properties.

In MySQL, our entities are stored in a table that looks like this:

CREATE TABLE entities ( added_id INT NOT NULL AUTO_INCREMENT PRIMARY
KEY, id BINARY(16) NOT NULL, updated TIMESTAMP NOT NULL, body MEDIUMBLOB, UNIQUE KEY
(id), KEY (updated) ) ENGINE=InnoDB; 

The added_id column is present because InnoDB stores data rows physically in primary key order. The AUTO_INCREMENT primary key ensures new entities are written sequentially on disk after old entities, which helps for both read and write locality (new entities tend to be read more frequently than old entities since FriendFeed pages are ordered reverse-chronologically). Entity bodies are stored as zlib-compressed, pickled Python dictionaries.

Now that the FriendFeed team works at Facebook I suspect they'll end up deciding that a NoSQL database that has solved a good story around replication and fault tolerance is more amenable to solving the problem of building a schema-less database than storing key<->value pairs in a SQL database where the value is a serialized Python object.

As a Web developer it's always a good idea to know what the current practices are in the industry even if they seem a bit too crazy to adoptyet.

Further Reading

• How FriendFeed Uses MySQL to Store Schema-less Data by Bret Taylor
• Looking to the future with Cassandra Digg Blog

Now Playing: Jay-Z - Run This Town (feat. Rihanna & Kanye West)

This is part 3 of a three-part series on the fundamental characteristics of the real-time Web.

In part 1 and part 2, we looked at how the real-time Web is a new form of communication, creates a new body of content, is real time, is public, and has an explicit social graph associated with it. A final characteristic of the real-time Web is that it carries with it an implicit model of federation.

Sponsor

A number of sources both generate and consume real-time streams. As a result, many of these new companies are becoming communication carriers, passing their users' real-time threads through their networks to other networks. This is more than simply being open (i.e. more than allowing data to be imported and exported). Just as in shipping and transportation and other communication industries before it (telephone, Internet packets, and email, to name a few), the real-time Web is developing a federated model of transmission whereby companies formally or tacitly agree to facilitate transmission and perform actions on behalf of end-users within the eco-system.

It's hard to say whether this model has arisen because of a conscious strategic effort to build a new industry, or because building a fully closed world would have required just too many resources, or because of a collective effort among business friends and acquaintances to develop open products and open interactions so that cool new things could be created. It's probably a combination of all three, but considering the history of the people at Twitter and FriendFeed (Paul Buchheit, one of FriendFeed's founders, is credited with coining Google's unofficial "Don't be evil" slogan), the open and cool factors are probably a big part of the equation.

At this point, there seems to be a general willingness to accept and transmit messages from outside sources (carrying costs are not significant, and transmission is automated via APIs, and so overhead is minimal). That said, infrastructure costs are bound to increase, competition will heat up, illegitimate companies will spot opportunities, and monetization strategies will be devised, which will all put strain on this truly open exchange.

As in the past, formal carrier agreements could be set down, governments could decide to regulate markets, or other forces could come into play that would transform what is now essentially a free-for-all bazaar into a marketplace with hierarchy. All the same, the expectation of openness and transparent transmission will be difficult to counteract or stop. So, new companies that enter the space, even bigger and better funded ones, will have to adhere to the same model of federation that these pioneering companies have established.

Summary

Whether Twitter will remain the focal point of the real-time Web or be supplanted by another or several companies (as happened in the social network space, first with Friendster, then MySpace, and now LinkedIn and Facebook) is unclear. The underlying characteristics of the real-time Web, however, are defining the next major stage of the Internet and will spread throughout its infrastructure in years to come.

Broader trends on the Web point to users having discrete data and services follow them as they move around the Web. Fred Wilson, a principal of Union Square Ventures, has called this the "de-portalization of the Web," and John Borthwick, CEO of betaworks, has co-opted Chris Anderson's phrase "small pieces, loosely joined" to describe the fast-moving risk-taking small companies that work in the space. Both individuals are leading investors in Twitter and other real-time Web companies.

The Internet is shifting from discrete units of websites and Web pages to discrete units of information (e.g. people, organizations, articles and videos, product offerings, store listings, and blog posts) and associated meta data (e.g. images, addresses, reviews, ratings) that move seamlessly around the Web, being slotted where appropriate. These units of information can be organized in ways that are relevant and personal to each individual, using data gleaned from social graphs as well as recommendation and personalization services that allow users to set their preferences.

In some cases, locations are integrated into these units as supplementary information. For example, Google and Yahoo now include map locations and reviews as part of their search listings. Their search engine algorithms read markup formats in the form of microformats and RDFa that are embedded on Web pages. These formats contain tags denoting names of people and organizations, geo-locations, and ratings and reviews. Both companies report great results from the inclusion of this data, both in increased click-through rates and reduced bounce rates. Support for other structured data is almost sure to follow. Reading tags on a page and doing something useful with them in a search result is not a novel concept, but the rapidly growing support of these tags across the Web is a clear sign that data is becoming much more identifiable and actionable.

This trend towards open and accessible data is even more obvious when you consider the real-time stream for all of the reasons mentioned above: atomic real-time messages, public accessibility, attached social graphs. In a sense, this is similar to the vision of the semantic Web. Tim Berners-Lee said at the TED conference in the fall of 2008, "Twenty years ago, I asked everyone to put their documents on this Web thing... Now I want you to put your data on the Web." The difference is that the effort to make data accessible and more actionable on the real-time Web is being made through methods and interactions not necessarily prescribed by the W3C.

Tim Berners-Lee and the W3C use the term "linked data" to refer to the latter's initiative to expose data and make it accessible. "Actionable data" might be a better term to use for the real-time Web because it doesn't imply a particular approach but merely refers to the concept of making data more identifiable and independent. Linked data refers specifically to using RDF and other W3C protocols to link important concepts, a prescription that is overly complex and not likely to address many of the usage cases on the Web.

The real-time stream is a massive body of continously created and authentic content that by itself would be significant. But when it is added to and integrated with other information on other sites, and when derivatives can be created in a number of dimensions, this concept of actionable data reaches the tipping point. In non-Silicon Valley business circles, Twitter is criticized for not having a solid revenue model. Those on the inside (investors and advisers), however, believe the criticism is short-sighted. As with most communication platforms, the value of the network increases exponentially as the size of the network increases.

By having a low barrier to adoption, the network is able to grow quickly. Only after a critical mass has been reached, and after other companies and communities of interest have helped shape how the platform is used, will it become clear what people are willing to pay for. While they may not have a solid grasp yet of exactly how to make money, those who are building companies and investing in the space do know there will be opportunities. In their minds, the real-time stream is at an early stage in its cycle, one that will likely last 5 to 7 years.

If the real-time Web and its fundamental characteristics are widely understood, its benefits and opportunities can extend throughout the Internet and across all industries.

Read part 1 and part 2 of this series.

Guest author: Ken Fromm is a serial entrepreneur who has been active during both the Internet and Web 2.0 innovation cycles. He co-founded two companies, Vivid Studios, one of the first interactive agencies, and Loomia, one of the top recommendation, discovery, and personalization companies. He has worked at the leading edge of recommendations and personalization, interactive development, e-commerce and online advertising, semantic technologies and information interoperability, digital publishing, and digital telephony. He is currently advising a number of startups and looking at the next big thing in Web 3.0. He can be found on Twitter at @frommww.

Social networking is for lonely, psychotic shut-ins. Or at least that's the upshot of the jokes in the attached clip from Adam Sandler vehicle Funny People. And still MySpace apparently cooperated with the filmmakers; its co-founder and logo appear.

The video clip above, from YouTube, is grainy, but TechCrunch's Mike Arrington assures readers it's in the final movie. I hadn't seen the film myself, unaware it touched on social networking, but Arrington writes that MySpace takes up a solid five minutes of the movie.

The treatment is brutal. Early in the clip, MySpace co-founder Tom Anderson asks Sandler if he actually uses the product. The star's reply: "No, no no. I fuck girls, Tom. I don't have time for that." When he goes on stage, the comic greets the MySpace crowd as "nerds" and then trashes their users: "They say the more friends you have on MySpace the less friends you have in real life." .

Sure, MySpace's competitors are insulted, too. But companies like Silicon Valley-based Facebook are fighting hard to avoid Hollywood; Facebook trashed Ben Mezrich's book about the company, The Accidental Billionaires, and by extension the Aaron Sorkin movie based on that book, calling it inaccurate.

But MySpace is based in Beverly Hills, close to Hollywood, and seems to have a better handle on the big picture: Being on the silver screen, in any context, means you're culturally relevant. Why not embrace the opportunity to make your virtual community a lot more real? (Via TechCrunch.)

Macolm Gladwell may have taken issue with Wired magazine editor-in-chief Chris Anderson's assertion that the price of digital goods naturally drops to zero, but Trent Reznor who has successfully practiced the theory for years couldn't agree more.

Some fans objected to Reznor's claim that Topspin Media (video interview) got it right with its re-release of the Beastie Boys album Ill Communication, which offers a wide array of merchandise in just about every conceivable format at a wide variety of prices. It's become a well-worn criticism of the independent distribution model that fledgling bands need a helping hand in order to make it in the music business. Not so, says Reznor. According to him, giving away digital music while charging for scarce, premium edition is the best way forward for artists of all stripes not just Radiohead and his own band, Nine Inch Nails.

Forget thinking you are going to make any real money from record sales, wrote Reznor on his message board. Make your record cheaply (but great) and GIVE IT AWAY [as DRM-free MP3s] Collect people's e-mail info in exchange (which means having the infrastructure to do so) and start building your database of potential customers. Then, offer a variety of premium packages for sale and make them limited editions / scarce goods.

It's a play straight out of Anderson's playbook (and, in fact, Anderson cites Nine Inch Nails as an example of a business that understands Free).

To put it into practice, Reznor advises that bands distribute through Amazon, TopSpin or Tunecore; set up a simple, Flash-free site outside of MySpace (which he says is dying and reads as cheap / generic); never abuse their mailing list; use free tools from Twitter, Flickr, Vimeo, YouTube and SoundCloud; and give people a reason to keep coming back to their site (Reznor's own forums are an example of this strategy).

However, Reznor says the strategy of giving away music in return for e-mail addresses, then marketing pricey box sets and other premium goods to those e-mail addresses only makes sense if a band wants to keep all its money and stay in control of its image.

If you are looking for mainstream super-success (think Lady GaGa, Coldplay, U2, Justin Timberlake), your best bet in my opinion is to look at major labels and prepare to share all revenue streams / creative control / music ownership. To reach that kind of critical mass these days, you'll need old-school marketing muscle, and that only comes from major labels.

Good luck with that one.

See Also:

• Free! Why $0.00 Is the Future of Business
• TopSpin Lets Bands Ape Radiohead, Nine Inch Nails
• Video Interview: Ian Rogers, Topspin Media
• SoundCloud Threatens MySpace as Music Destination for Twitter Era
• Wired Editor-in-Chief Chris Anderson on the Future of Free
• Nine Inch Nails and Radiohead Dominate Amazon MP3 Chart
• It Just Got Cheaper To Sell a Song on iTunes
• MySpace Music: What Went Wrong, and What's Being Done About It

Photo: Andrea Veraart

Jacksonville, Florida-based Cliqset is launching the second beta version of its online identity platform today with some nifty new features, and is also announcing that it has raised $1.5 million in financing from a single angel investor.

Cliqset is not exactly an easy concept to explain, but here goes. Essentially, the platform aims to stitch together the social web by allowing users and developers build, organize and share social information across a wide variety of services. As an end user, Cliqset can help you merge and share the social information (your status updates, location, photos, etc) currently scattered around the web with the people, applications and devices you already use and trust. Developers on the other hand get access to an extensive set of read/write social APIs they can use as an alternative to building and managing support for their own.

The second beta, launching today, comes with a new Location Services API that allows developers to build apps for web and mobile by using location info from users. With the API, developers can fetch and use address information using the latitude and longitude coordinates provided by mobile devices. The gathered location information can be tied to user activities but also be used to store more generic location information that's relevant to the applications they build on top of the Cliqset platform.

A related new element is the integration of Cliqset location services with third-party services like BrightKite, FireEagle and Twitter, basically simplifying how a user can keep their social and location information in sync across the Web.

Also new is a fresh push/pull architecture for social information that travels to and from Cliqset, Cliqset-enabled applications and the supported third-party services. A mix of push/pull functionality is now possible with over 30 third-party services, including Twitter, Facebook, Myspace, identica, laconica, Linkedin, FireEagle, and FriendFeed. It's like the latter on steroids, actually.

Cliqset has recently closed its second seed round, $1.5 million coming from angel investor Derek Mercer, founder and former chairman and CEO of Vurv Technology, a provider of talent management software that was acquired in 2008 by Taleo for about $128.8 million. This comes in addition to an earlier early-stage capital injection of $500,000 by the man, bringing the total invested in the startup to $2 million.

Cliqset - Merge, Organize and Share Social Information from cliqset on Vimeo.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.