Infrastructure Web Design

Client projects and tech blog posts about Infrastructure

It's not necessary to develop your own Web 2.0 software infrastructure to create an independent Web 2.0-powered community online. It's far simpler to set a standard for your community to use on exisiting networks and then to use Yahoo Pipes to pull it together.

I decided on about a dozen categories to use with my DIY blog aggregator (QuakerQuaker). I only want to pull in posts that are being generated for my site by community members so we use a community identifier, a unique prefix that isn't likely to be used by others.

This post will show you how to pull in tagged feeds from three sources: the Del.icio.us social bookmarking system, the Flickr photo sharing site and Google Blog Search.

Step 1: Pick a community designator

I've been using the community name followed by a dot. The prefix goes in front of category description to make a set of unique tags for the aggregator. When someone wants to add something for the site they tag it with this "community.category" tag. In my example, when someone wants to list a new Quaker blog they use "quaker.blog", "quaker" being the community name, "blog" being the category name for the "New Blogs" page.

Step 2: Collect the community prefix and category name in Pipes

You begin by going into Pipes and pulling over two text inputs: one for the community prefix, the other for the specific category.

Step 3: Construct these into tags

Now use the "String Concatenation" module to turn this into the "community.category" model. The community input goes into the top slot, a dot is the second slot and the category input goes into the last slot.

Now, when you have a tag in Flickr with a dot in it, Flickr automatically removes it in the resultant RSS feed. So with Flickr you want your tag to be "communitycategory" without a dot. Simple enough: just pull another "String Concatenation" module onto your Pipes work space. It should look the same except that it won't have the middle slot with the dot.

Step 4: Turn these tags into RSS URLs

Pull three "URLBuilder" modules into Pipes, one for each of the services we're going to query. For the Base, use the non-tag specific part of the URL that each service uses for its RSS feeds. Here they are:

Del.icio.ushttp://del.icio.us/rss/tag
Flickrhttp://api.flickr.com/services/feeds
Google Blog Searchhttp://blogsearch.google.com

Under path elements, put the correct tag: for Del.icio.us and Google it should be the community.category tag, for Flickr the dot-less communitycategory tag.

Step 5: Fetch and Dedupe

Fetch is the Pipes module that pulls in URLs and outputs RSS feeds. It can also combine them. Send each URLBuilder output into the same Fetch routine.

Since it's possible that you'll might have duplicate posts, use the "Unique" module to deduplicate entries by URL. Through a little trial and error I've determined that in cases of duplicates, feeds lower in the Fetch list trump those higher. In the actual Pipe powering my aggregator I pull a second Del.icio.us feed: my own. I have that as the last entry in the Fetch list so that I can personally override every other input.

Step 6: Sort by Date

With experimentation it seems like Pipes orders the output entries by descending date, which is probably what you want. But I want to show how Pipes can work with "dc" data, the "Dublin Core" model that allows you to extend standard RSS feeds (see yesterday's post for more on this).

Google Blog Search and Del.icio.us feeds use the "dc:date" field to record the time when the post was made. Flickr uses "dc:date.Taken" to pass on the photograph's metadata about when it was taken. Pipes' "Rename" module lets you copy both fields into one you create (I've simply used "date"), which you can then run through its "Sort" module. Again, it's a moot point since Pipes seems to do this automatically. But it's good to know how to manipulate and rename "dc" data if only because many PHP parsers have trouble laying it out on a webpage.

Update: it's all moot: according to a ZDNet blog, "Pipes now automatically appends a pubDate tag to any RSS feed that has any of the other allowable date tags." This is nice: no need to hack the date every time you want to make a Pipe!

Step 7: Output

The final step for any Pipe is the "Pipe Output" module.

In action

You can see this published Pipe here, and copy and play with it yourself. The result lets you build an RSS feed based on the two inputs.

Categories: Practical 2.0 , RSS Syndication
Tags: Category Description, Delicious, Flickr, Google, Photo Sharing, Social Bookmarking, Yahoo | Edit
Every website should try to serve a clear set of purposes. Even a personal blog has a target audience, one's friends or family perhaps. While a good site looks simple, it is often very complicated "under the hood."

Google went from being a grad school project to the world's most important search engine by ditching the design clutter of its competitors for a very clean homepage with maximum white space. This effect focused one's attention on the search function. More PhD's are said to work at Google than at any other company in the world, yet the complicated engineering and the tremendous computer infrastructure that brings that logo and search box to your computer is invisible to the average user.

Even websites without PhD designers need to marry a simple outward appearance with a more complicated set of calculations around intended audiences. The average visitor looks at one or two pages on a site and then hits the back button. Often they'll be following a search link and looking at a page buried deep in your site. They'll be there seeking out specific information and you only have about twenty seconds to pitch your site and keep them there. You need to give them a very concise description of yourself or product and you need to entice them with related material.

Any site that consists of more than three pages presents visitors with more information than they can handle. Good design works to funnel visitors to the specific content they are looking for. It's relatively easy to get a first-time visitor but successful websites keep them on your site and give them reasons to return. The key to this is defining your audience and presenting your material with them in mind.

Once you've identified your constituency and built your design, the next step is release. You don't want to pander to a potential audience, but instead converse with them. It's fine to mix different elements of your life together and to write creatively off-topic once in awhile. There are a thousand generic websites crammed full of useless bu zzphrases and unused featured. What you want is one that will have a voice, that builds a niche that no one else might ever have identified. When it comes time to produce content, forget all the slick marketing calculations you've done and let your quirkiness shine.

Categories: Web Design
Tags: Design, Google, Grad School, Personal Blog, Phd, Search Box, Search Engine, Search Function | Edit
This essay was originally written in 1995.

IT'S HARD TO IGNORE the sorry shape of the social change community. The signs of a collapsed movement are everywhere. Organizations are closing, cutting back, laying off staff, and dropping the frequency of their magazines.

On top of this, the basic resources we've depended on are getting scarcer. Paper prices and postage prices are going up. Direct mail solicitations are for many economically-unfeasible now. With every abandoned mailing list, with every discontinued peace fair, we're losing the infrastructure that used to nourish the whole movement.

Here in Philadelphia, the last few years have seen food coops close, peace organizations lay off staff, and the bookstores discontinue their political titles. I've been meeting people only a half-generation younger than I who aren't aware of the basic organizing principles that the movement has built up over the years and who don't know the meanings of Greenham Common or the Clamshell Alliance

Like many of you, I'm not giving up. We can't just abandon our work because it's becoming more difficult. We need to struggle to find creative ways of getting our message out there and communicating with others. What we need is a new media.

The Promise of the Web

The Web's revolution is it's incredibly minimal costs. Fifteen dollars a month gets you a homepage. As an editor at New Society Publishers (1991-1996), I've always had to worry whether we'd lose money on a particular editorial project, and it sometimes seemed a rule of thumb that what excited me wouldn't sell. With the Web, we don't have to worry if an idea isn't popular because we're not putting the same level of resources into each publication.

Never before has publishing been so cheap. Just about anyone can do it. You don't need a particularly fast or fancy computer to put Web pages online. And you don't have to worry about distribution: if someone sets their Web browser to your address, they'll get you "product" instantly.

All the forces pushing movement publishing over the edge of financial insolvency disappear when we go online. Switching to the Web is a matter of keeping our words in print. The Web is the latest invention to open up the distribution of words by birthing new medias. The printing press begat modern book publishing just as the photocopier begat zine culture. The Web can likewise spawn a media where words can flourish with less capital than ever before.

Advertising Each Other

The problem with the Web is not accessibility, but rather being heard above the noise. People generally find your website in two ways. The first is that they see your web address in your newsletter, get on their computers and look you up; this of course only gets you your own people. The second way is through links.

Links take you from one website to another. Webpage designers try to get linked from sites of similar interest to theirs, hoping the readers of the other site will follow the link to their webpage. This bouncing from site to site is called surfing, and it's the main way around the web.

Linking is a very primitive art nowadays. The Nonviolence Web has internal links that actively invite readers to explore the whole NV-Web. Everytime someone comes into the NV-Web through a member group, they will be inticed to stay and discover the other groups. By putting social change groups together in one place, we can have a much-more dynamic cross-referencing. Think of it as the equivalent of trading mailing lists in that we can all share those web surfers who find any one of us.

In the web world as in the real one, cooperation helps us all. If you're an activist group doing work on nonviolent social change then contact us and we'll put your words online. For free. If you have your own website already, then let's talk about how we can crosslink you with other groups working on nonviolent social change.

Come explore the Nonviolence Web and let us get you connected. Come join our revolution.

In peace,

Martin Kelley

Categories: MartinKelley.com , Niche Marketing , Web Design
Tags: Bookstores, Direct Mail, Greenham Common, Infrastructure, Mail Solicitations, Peace Organizations, Political Titles, Postage Prices | Edit

Search

As Seen In

EBook

Shortcut cover
Web 2.0 Mash-Ups & Niche Aggregators (O'Reilly Media, 2008, $9.95): Order here.

Social Networks

Other Sites

Archives