Step 1: Pick a community designator

I've been using the community name followed by a dot. The prefix goes in front of category description to make a set of unique tags for the aggregator. When someone wants to add something for the site they tag it with this "community.category" tag. In my example, when someone wants to list a new Quaker blog they use "quaker.blog", "quaker" being the community name, "blog" being the category name for the "New Blogs" page.

Step 2: Collect the community prefix and category name in Pipes

You begin by going into Pipes and pulling over two text inputs: one for the community prefix, the other for the specific category.

Step 3: Construct these into tags

Now use the "String Concatenation" module to turn this into the "community.category" model. The community input goes into the top slot, a dot is the second slot and the category input goes into the last slot.

Now, when you have a tag in Flickr with a dot in it, Flickr automatically removes it in the resultant RSS feed. So with Flickr you want your tag to be "communitycategory" without a dot. Simple enough: just pull another "String Concatenation" module onto your Pipes work space. It should look the same except that it won't have the middle slot with the dot.

Step 4: Turn these tags into RSS URLs

Pull three "URLBuilder" modules into Pipes, one for each of the services we're going to query. For the Base, use the non-tag specific part of the URL that each service uses for its RSS feeds. Here they are:

Under path elements, put the correct tag: for Del.icio.us and Google it should be the community.category tag, for Flickr the dot-less communitycategory tag.

Step 5: Fetch and Dedupe

Fetch is the Pipes module that pulls in URLs and outputs RSS feeds. It can also combine them. Send each URLBuilder output into the same Fetch routine.

Since it's possible that you'll might have duplicate posts, use the "Unique" module to deduplicate entries by URL. Through a little trial and error I've determined that in cases of duplicates, feeds lower in the Fetch list trump those higher. In the actual Pipe powering my aggregator I pull a second Del.icio.us feed: my own. I have that as the last entry in the Fetch list so that I can personally override every other input.

Step 6: Sort by Date

With experimentation it seems like Pipes orders the output entries by descending date, which is probably what you want. But I want to show how Pipes can work with "dc" data, the "Dublin Core" model that allows you to extend standard RSS feeds (see yesterday's post for more on this).

Google Blog Search and Del.icio.us feeds use the "dc:date" field to record the time when the post was made. Flickr uses "dc:date.Taken" to pass on the photograph's metadata about when it was taken. Pipes' "Rename" module lets you copy both fields into one you create (I've simply used "date"), which you can then run through its "Sort" module. Again, it's a moot point since Pipes seems to do this automatically. But it's good to know how to manipulate and rename "dc" data if only because many PHP parsers have trouble laying it out on a webpage.

Update: it's all moot: according to a ZDNet blog, "Pipes now automatically appends a pubDate tag to any RSS feed that has any of the other allowable date tags." This is nice: no need to hack the date every time you want to make a Pipe!

Step 7: Output

The final step for any Pipe is the "Pipe Output" module.

In action

You can see this published Pipe here, and copy and play with it yourself. The result lets you build an RSS feed based on the two inputs.

Just about every native Web 2.0 applications comes built-in with multiple RSS feeds. But in the real world, websites are built using an almost-infinite number of content management systems and web development software programs. Sometimes a single website will use different programs for putting its contents online and sometimes a single organization spreads its functions over multiple domains.

Step 1: Make it Del.icio.us

To begin, sign up with Del.icio.us, the popular "social bookmarking" web service (similar services can be easily adapted to work). Then add a "post to Del.icio.us" button to your browser's toolbar following the instructions here. Now whenever you put new content up on your site, go that new page, click on your "post to Del.icio.us" button and fill out a good title and description. Choose a tag to use. A tag is simply a category and you can make it whatever you want but "mysites" or your business name will be the easiest to remember. Hit save and you've started an RSS feed.

How? Well, Del.icio.us turns each tag into a RSS feed. You can see it in all its machine code glory at del.icio.us/rss/username/mysites (replacing "username" with your username and "mysites" with whatever tag you chose).

Now you could just advertise that Del.icio.us RSS feed to your audience but there are a few problems doing this. One is that Del.icio.us accounts are usually personal. If your webmaster leaves, then your published RSS feed will need to change. Not a good scenario, especially since you won't even be able to tell who's still using that old feed. Before you advertise your feed you should "future proof" it by running it through Feedburner.

Cloak that Feed

Go to Feedburner.com. Right there on the homepage they invite you to type in a URL. Enter your Del.icio.us feed's address and sign up for a Feedburner account. In the field next to feed address give it some sensible name relating to your company or site, let's say "mycompany" for our example. You'll now have a new RSS feed at feeds.feedburner.com/mycompany. Now you're in business: this is the feed you advertise to the world. If you ever need to change the source RSS feed you can do that from within Feedburner and no one need know.

The default title of your Feedburner feed will still show it's Del.icio.us roots (and the webmaster's username). To clear that out, go into Feedburner's "Optimize" tab and turn on the "Title/Description Burner," filling it out with a title and description that better matches your feed's purpose. For an example of all this in action, the Del.icio.us feed that powers my tech link blog and its Feedburner "cloak" can be found here:

• http://del.icio.us/rss/martin_kelley/tech
• http://feeds.feedburner.com/techlinksblog

Get that Feed out there

Under Feedburner's "Publicize" tag there are lots of neat features to republish your feed yourself. First off is the "Chicklet chooser" which will give you that ubiquitous RSS feed icon to let visitors know you've entered the 21st Century. Their "Buzz Boost" feature lets you create a snippet of code for your homepage that will list the latest additions. "Email subscriptions" lets your audience sign up for automatic emails whenever you add something to your site.

Final Thoughts

RSS feeds are great ways of communicating exciting news to your audiences. If you're lucky, important bloggers in your audience will subscribe to your feed and spread your news to their networks. Creating a feed through a bookmarking service allows you to add any page on any site regardless of its underlying structure.

About eighteen months ago I started a "links blog" of interesting Quaker links, incorporated as a sidebar on my popular "QuakerRanter" personal blog. I eventually gave the links their own URL (QuakerQuaker.org) and invited others to join the linking. I always stumble when trying to tell people what QuakerQuaker is all about. The best definition is that its a "collaboratively edited blog aggregator" but that's a horribly tech description.

The rise of blogs is creating the necessity for these sort of theme-based aggregators. This morning I stumbled on Original Signal, a new site that organzes the best Web 2.0 blogs. A site called PopURLs does the same for "the latest web buzz." A site called SolutionWatch has written about these in Tracking the web with Single Page Aggregators. We're all on to something here. I suspect that sometime this fall some clever person will coin a new term for these sites.

An important part of the redesign was an automatic keyword generator. Posts were run through a script that automatically pulled out keywords from the text. My 2003 article, Going all the way with Movable Type generated the following tags, which appear as links after the post:

• choate
• nonviolence
• curly quotes
• personal blog
• php
• org
• moveabletype
• movable type
• hmtl
• dashes
• textile
• garbage
• blogs

Following the links takes you to similarly-tagged articles. At least that's the conceit. When you follow a tag's link you're simply doing a site search for that keyword. A little htaccess rewrite magic is making the result look like it's a static category page.

"Fine and well" you're thinking, "big deal." Well, here's what's cool. There are 225 entries on the QuakerRanter blog. Google's just gone through and indexed the site and is now claiming it contains 1300 pages. Each tag is being indexed as its own page. Every time I mention any interesting term, it becomes a page that Google indexes and delivers to its searchers.

Which brings us to today's cool piece from the access logs. In December of 2004 a rather innocent post on Quaker Ranter became the center of a mini-whirlwind on the political blogs when it mentioned that I had gotten a call from a CBS News publicist interested in Nonviolence.org. All political blogs get publicity calls from news and opinion think tanks trying to suggest (or plant) stories but no one's supposed to talk about it. I only mentioned it because it was so unusual. One of the blogs denouncing the liberal conspiracy my post revealed was the somewhat slimy Little Green Footballs. After a few weeks the denunciations died down.

But this morning, someone looked up littlegreenfootballs in Google and came to my site. Because of my automatic keyword generator, tags, and static-loooking links, I'm now the number two entry, on two three-year old posts, now relocated to a days old quakerranter.org. Cool.

This mixing and matching of content and rich manipulation of data is sometimes lumped together in the cool bu zzphrase folksonomy. Note that none of what I've done is a tricking of Google. Every tag is really going to a page with that content. These are "natural" and "organic" search results in the lingo of SEO. I'm just presenting my information in multiple formats that appeal that the widest array of audiences.

For what it's worth, I don't think I deserve #2 status for "littlegreenfootballs" and I don't think Google will keep it there for long. It's a bit odd that they have elevated that particular term so high and no others tags seem so stratospheric.

Positive Results:

As of February 2007, Google indexes 3,540 pages on QuakerRanter.org, a blog of only 239 posts. In December 2006 30% of my Google visits were to one of the "tags" page. Reconfiguring the blog in this kind of tag-intensive way has more than doubled search engines visits, again in a very natural and organic way. Adding tags has simply made what I've written more accessible to search engines. Very cool.

Negative Ramifications:

Shortly after installing this new system, my servers started periodically crashing (about once/week). The problem would be multiple MT-Search processes overloading the memory.

My guess is that a search engine spider came along and started indexing all of the tags. Each link initiated a search query in Movable Type. The built-in search for Movable Type is just not able to handle this volume of traffic.

I installed Fast Search to solve the problem (tip of the hat to Al-Muhajabah). It took awhile: Fast Search required a MySQL upgrade at my host. After that I needed to install these plugin fixes. Then it was fine-tuning the htaccess files. It was been more work than I initially expected and the tag results now forward to a funny URL that Google doesn't love as much.

Even websites without PhD designers need to marry a simple outward appearance with a more complicated set of calculations around intended audiences. The average visitor looks at one or two pages on a site and then hits the back button. Often they'll be following a search link and looking at a page buried deep in your site. They'll be there seeking out specific information and you only have about twenty seconds to pitch your site and keep them there. You need to give them a very concise description of yourself or product and you need to entice them with related material.

Any site that consists of more than three pages presents visitors with more information than they can handle. Good design works to funnel visitors to the specific content they are looking for. It's relatively easy to get a first-time visitor but successful websites keep them on your site and give them reasons to return. The key to this is defining your audience and presenting your material with them in mind.

Once you've identified your constituency and built your design, the next step is release. You don't want to pander to a potential audience, but instead converse with them. It's fine to mix different elements of your life together and to write creatively off-topic once in awhile. There are a thousand generic websites crammed full of useless bu zzphrases and unused featured. What you want is one that will have a voice, that builds a niche that no one else might ever have identified. When it comes time to produce content, forget all the slick marketing calculations you've done and let your quirkiness shine.

Inspired by Doing Your Whole Site with MT on Brad Choate's site, I started experimenting today with putting the whole Nonviolence.org site into Movable Type. At first I thought it was just a trial experiment but I'm hooked. I especially love how much cleaner the entry for the links page now looks and I might actually be inspired to keep it up to date more now. (I've also integrated Choate's MT-Textile which makes a big difference in keeping entries clean of HMTL garbage, and the semi-related SmartyPants which makes the site more typographically elegant with easy M-dashes and curly quotes).

So here's what I'm doing: there are three Movable Type blogs interacting with one another (not including this personal blog):

• One is the more-or-less standard one that is powering the main homepage blog of Nonviolence.org.
• The second I call "NV:Static" which holds my static pages, much as Brad outlines. I put my desired URL path into the Title field (i.e., "info/index") and then put the page's real title into the Keywords field (i.e., "About Nonviolence.org") and have that give the data for the title field and the first headline of the page. It might seem backwards to use Title for URL and then use Keywords for Title, but this means that when I'm in MT looking to edit a particular file, it will be the URL paths that are listed.
• The third blog is my "NV:Design Elements." This contains the block of graphics on the top and left of every page. I know I'll have to redesign this all soon and I can do it from wherever. This blog outputs to HTML. All the other pages on the site are PHP and its a simple include to pull the top and left bars into each PHP page.

Oh yes, I'm also thinking of incorporating guest blogs in the near future and all of these elements should make that much easier.

Here's another site to check out, about how someone integrated Movable Type into their church website using some interesting techniques.