Search Web Design

Step 1: Pick a community designator

I've been using the community name followed by a dot. The prefix goes in front of category description to make a set of unique tags for the aggregator. When someone wants to add something for the site they tag it with this "community.category" tag. In my example, when someone wants to list a new Quaker blog they use "quaker.blog", "quaker" being the community name, "blog" being the category name for the "New Blogs" page.

Step 2: Collect the community prefix and category name in Pipes

You begin by going into Pipes and pulling over two text inputs: one for the community prefix, the other for the specific category.

Step 3: Construct these into tags

Now use the "String Concatenation" module to turn this into the "community.category" model. The community input goes into the top slot, a dot is the second slot and the category input goes into the last slot.

Now, when you have a tag in Flickr with a dot in it, Flickr automatically removes it in the resultant RSS feed. So with Flickr you want your tag to be "communitycategory" without a dot. Simple enough: just pull another "String Concatenation" module onto your Pipes work space. It should look the same except that it won't have the middle slot with the dot.

Step 4: Turn these tags into RSS URLs

Pull three "URLBuilder" modules into Pipes, one for each of the services we're going to query. For the Base, use the non-tag specific part of the URL that each service uses for its RSS feeds. Here they are:

Under path elements, put the correct tag: for Del.icio.us and Google it should be the community.category tag, for Flickr the dot-less communitycategory tag.

Step 5: Fetch and Dedupe

Fetch is the Pipes module that pulls in URLs and outputs RSS feeds. It can also combine them. Send each URLBuilder output into the same Fetch routine.

Since it's possible that you'll might have duplicate posts, use the "Unique" module to deduplicate entries by URL. Through a little trial and error I've determined that in cases of duplicates, feeds lower in the Fetch list trump those higher. In the actual Pipe powering my aggregator I pull a second Del.icio.us feed: my own. I have that as the last entry in the Fetch list so that I can personally override every other input.

Step 6: Sort by Date

With experimentation it seems like Pipes orders the output entries by descending date, which is probably what you want. But I want to show how Pipes can work with "dc" data, the "Dublin Core" model that allows you to extend standard RSS feeds (see yesterday's post for more on this).

Google Blog Search and Del.icio.us feeds use the "dc:date" field to record the time when the post was made. Flickr uses "dc:date.Taken" to pass on the photograph's metadata about when it was taken. Pipes' "Rename" module lets you copy both fields into one you create (I've simply used "date"), which you can then run through its "Sort" module. Again, it's a moot point since Pipes seems to do this automatically. But it's good to know how to manipulate and rename "dc" data if only because many PHP parsers have trouble laying it out on a webpage.

Update: it's all moot: according to a ZDNet blog, "Pipes now automatically appends a pubDate tag to any RSS feed that has any of the other allowable date tags." This is nice: no need to hack the date every time you want to make a Pipe!

Step 7: Output

The final step for any Pipe is the "Pipe Output" module.

In action

You can see this published Pipe here, and copy and play with it yourself. The result lets you build an RSS feed based on the two inputs.

Step 1: Input tags

The Del.icio.us social bookmarking system runs much of my aggregator: users see a post they like and bookmark it in Del.icio.us with a special tag.

The first step in Pipes is to collect the input (right). Pull the "Text Input" module (above) onto your Pipes work space. This lets you collect user input. Give it both a name and a prompt. In most instances it's fine that these be the same as the prompt won't be visible in the end. It's good to put something down in Debug for later on in the Pipes process.

Step 2: Construct the RSS call

We take our two input tags and use them to construct an URL by using the "URLBuilder". The base URL is Del.icio.us's RSS feed (http://del.icio.us/rss/). The URL builder adds the user input then the tag input to give us a valid URL (http://del.icio.us/rss/user/tag/).

Step 4: Grab the feed

Yahoo Pipes' "Fetch" module takes that URL input and turns it into an RSS feed. Shown to the right is Fetch with the final "Pipe Output".

See it in action

You can see how this fits together by going to my Del.icio.us Sample page on Pipes. You can make a copy and play with it yourself. Add "&user=username&tag=tagname&_render=rss" to that URL and you've made it a feed.

Okay so I've turned a Del.icio.us RSS feed into... a Yahoo Pipes RSS feed with identical input and output. Well, we're only getting started. Our input tags can be reused for other searches and spliced together inside of a more elaborate Pipe. That's where the fun starts and I'll get there soon.

The other advantage of sending things through Pipes is that we can easily rename fields. Del.icio.us, Flickr and other services often extend RSS standards by including metadata in "dc" fields, an abbreviation for the Dublin Core standards extension. A recent entry from my Del.icio.us feed includes this:

     <dc:creator>martin_kelley</dc:creator>
     <dc:date>2007-03-15T05:18:53Z</dc:date>
     <dc:subject>tech tech.design</dc:subject>

Standard PHP parsers like MagpieRSS and SimplePie often have trouble pulling dc data. With Pipes you can rename the fields you like; in theory that should make them more accessible to the parsers. You can also combine fields and use Pipes' Regex module to operate on them with regular expressions.

Don't assume that your website is easy to navigate. One of the neatest things about the web is that we have instant feedback on use. With just a little tracking we can see what pages people are looking at, how they're finding our site and what they're doing once they're here.

Javascript Trackers:

My most advanced sites are currently using four different tracking methods. Most utilize javascript "bugs," tiny snippets of code that send individual results to an advanced software tracking system. I put the code inside a Moveable Type "Modules Template" which is automatically imported to all pages. Installing a new system is as easy as cutting-and-pasting the javascript into the Template and rebuilding the site.

AXS Visitors Tracking System
This software installs on your server but don't let that scare you: this is one of the easiest installations I've ever seen. AXS gives you great charts of usage: you can narrow it specific pages on your site, or even particular search engines or search phrases.

There's also a option to view the lastest traffic by visitor. I love watching this! You can see how individuals are using the site and where they're navigating. I've been able to identify different types of visitors this way and understand the complexity of the audience.

It doesn't seem like AXS is not being developed anymore. The latest stable version came out over two years go, which is a shame.

HitTail
This service watches search-engine links and makes recommendations for new keywords. I wrote about this service yesterday in Blogging for the Long Tail.

Reeferss.com
This is a simple simple bit of software. Like every other tracking system it keeps track of referrers: search engines and websites that bring traffic to your site. But unlike the others that's all it does. Why care then? It provides a real-time RSS feed of these visitors. I bring the feed into my "Netvibes" page (a customized start page, see below) and scan the results multiple times a day.

Google Analytics
The internet's gatekeeper bought the Urchin analytics company in April 2005 and relaunched the product as Google Analytics shortly thereafter. This is becoming an essential tracker. It's free and it's powerful, though I haven't been as impressed by it as others have. See its Wiki page for more.

Internet Trackers:

It's easy to find out what people are saying about you online.

Technorati
This service tracks blogs but you don't need to have a blog to use it, for Technorati will tell you where blogs are linking. Give it your URLs (or those of your competitors!) and you'll know whenever a blogger puts in a link to you. You can also give it keywords and find out when a blog uses them.

Google Blog Search
Google can also let you follow blog references or keyword mentions on the blogs. Google will also track beyond blogs of course. Type "site:www.yourdomain.com" into the main Google search page and you'll see who's linking to your site (or to the competition). There are lots of other services that track blogs and mentions--Sphere, Bloglines, etc. They all have different strengths so try them and see what you think.

Feedburner
The best RSS massager has always focused on ways to track your RSS feed. They've recently introduced page tracking software too. It looks great but I just installed it this week. I still have to see if it's as good as Feedburner's other offerings.

Keeping on top of this flow of data:

It's easy to get overwhelmed by all of this information. Most of the tracking services provide RSS feeds (See The Wonders of RSS Feeds for an intro). I use Netvibes, a customized start page, to pull these all together into a single page that I can scan every morning. Here's a screenshot of part of my Netvibes tracking page--the full page currently shows fourteen tracking feeds on one screen:

So why is tracking important to SEO?

With tracking you find out what people are looking for on the internet. This helps you create pages and services that people will want to find. You might be surprised to see what they're already finding on your site. Some examples:

• Analyzing one site, I noticed that few pages I thought were obscure were bringing in high Google traffic. I looked at these pages again and realized they did a good job of describing the company's mission. I consequently redesigned the site homepage to feature them and I made sure that those pages contained direct links to its most important services.

• When I started work for another client I looked at their site and suspected that they're most important articles were not being seen--visitors had to click through about four times to get to them. Six months of tracking confirmed my hunch and gave me the hard data to convince the executive director that we made some small modifications to the design. Having this strong content linked right off the homepage helped bring in Google traffic.

It's startling to realize that my three year toddler is almost the same age as Myspace and older than Facebook. In just a few short years they've come to dominate much of the online world, especially with under-25 users. The kind of independent blogs that dominate a sites like Livejournal and Blogspot don't have the web of cross-connections--what I called the "folksonomic density"--of the new social networking sites. It seems appropriate that Myspace was founded by spammers: who knows more about sucking people in?

The question: will the net have room for independent niche sites? Myspace is changing its architecture to disable key linking features of third-party embedded plug-ins like the from the popular video site Youtube. The big search sites also want a piece of this market--new features on Yahoo local and the geotagged maps on Yahoo's Flickr are impressive). It all reminds me some of the debates about local food co-ops versus enlightened supermarkets: is it a good thing that organic produce and soymilk can be purchased at the local Acme, even if that cuts into the independent co-op's business? Don't we want everyone to have access to everything? In the end, philosophy won't settle this argument.

An important part of the redesign was an automatic keyword generator. Posts were run through a script that automatically pulled out keywords from the text. My 2003 article, Going all the way with Movable Type generated the following tags, which appear as links after the post:

• choate
• nonviolence
• curly quotes
• personal blog
• php
• org
• moveabletype
• movable type
• hmtl
• dashes
• textile
• garbage
• blogs

Following the links takes you to similarly-tagged articles. At least that's the conceit. When you follow a tag's link you're simply doing a site search for that keyword. A little htaccess rewrite magic is making the result look like it's a static category page.

"Fine and well" you're thinking, "big deal." Well, here's what's cool. There are 225 entries on the QuakerRanter blog. Google's just gone through and indexed the site and is now claiming it contains 1300 pages. Each tag is being indexed as its own page. Every time I mention any interesting term, it becomes a page that Google indexes and delivers to its searchers.

Which brings us to today's cool piece from the access logs. In December of 2004 a rather innocent post on Quaker Ranter became the center of a mini-whirlwind on the political blogs when it mentioned that I had gotten a call from a CBS News publicist interested in Nonviolence.org. All political blogs get publicity calls from news and opinion think tanks trying to suggest (or plant) stories but no one's supposed to talk about it. I only mentioned it because it was so unusual. One of the blogs denouncing the liberal conspiracy my post revealed was the somewhat slimy Little Green Footballs. After a few weeks the denunciations died down.

But this morning, someone looked up littlegreenfootballs in Google and came to my site. Because of my automatic keyword generator, tags, and static-loooking links, I'm now the number two entry, on two three-year old posts, now relocated to a days old quakerranter.org. Cool.

This mixing and matching of content and rich manipulation of data is sometimes lumped together in the cool bu zzphrase folksonomy. Note that none of what I've done is a tricking of Google. Every tag is really going to a page with that content. These are "natural" and "organic" search results in the lingo of SEO. I'm just presenting my information in multiple formats that appeal that the widest array of audiences.

For what it's worth, I don't think I deserve #2 status for "littlegreenfootballs" and I don't think Google will keep it there for long. It's a bit odd that they have elevated that particular term so high and no others tags seem so stratospheric.

Positive Results:

As of February 2007, Google indexes 3,540 pages on QuakerRanter.org, a blog of only 239 posts. In December 2006 30% of my Google visits were to one of the "tags" page. Reconfiguring the blog in this kind of tag-intensive way has more than doubled search engines visits, again in a very natural and organic way. Adding tags has simply made what I've written more accessible to search engines. Very cool.

Negative Ramifications:

Shortly after installing this new system, my servers started periodically crashing (about once/week). The problem would be multiple MT-Search processes overloading the memory.

My guess is that a search engine spider came along and started indexing all of the tags. Each link initiated a search query in Movable Type. The built-in search for Movable Type is just not able to handle this volume of traffic.

I installed Fast Search to solve the problem (tip of the hat to Al-Muhajabah). It took awhile: Fast Search required a MySQL upgrade at my host. After that I needed to install these plugin fixes. Then it was fine-tuning the htaccess files. It was been more work than I initially expected and the tag results now forward to a funny URL that Google doesn't love as much.

Even websites without PhD designers need to marry a simple outward appearance with a more complicated set of calculations around intended audiences. The average visitor looks at one or two pages on a site and then hits the back button. Often they'll be following a search link and looking at a page buried deep in your site. They'll be there seeking out specific information and you only have about twenty seconds to pitch your site and keep them there. You need to give them a very concise description of yourself or product and you need to entice them with related material.

Any site that consists of more than three pages presents visitors with more information than they can handle. Good design works to funnel visitors to the specific content they are looking for. It's relatively easy to get a first-time visitor but successful websites keep them on your site and give them reasons to return. The key to this is defining your audience and presenting your material with them in mind.

Once you've identified your constituency and built your design, the next step is release. You don't want to pander to a potential audience, but instead converse with them. It's fine to mix different elements of your life together and to write creatively off-topic once in awhile. There are a thousand generic websites crammed full of useless bu zzphrases and unused featured. What you want is one that will have a voice, that builds a niche that no one else might ever have identified. When it comes time to produce content, forget all the slick marketing calculations you've done and let your quirkiness shine.