- - - - - - - - - - - - -

Thursday, March 13, 2008

nz blogocracy | FAQ: "the ranking system is a mess"

Re: Kiwiblogblog query.

(I must respond here because I simply will not register to make comments on someone's blog I'm afraid - we don't force people to register on this site in order to be able to participate, and personally I think the practice should be discouraged.)

Q. For no apparent reason, the blogs are scored on:
the number of unique visitors they get each day from people with the Alexa Toolbar installed, or unique visitors according to the site’s meter if available.
+ the number of posts a week (not a day?, not a month? Not a fortnight?)
+ the incoming links (number of blogs linking to your blog in the last 6 months) by Technorati
+incoming links as scored by Truth Laid Bear (many blogs, like us, are not signed up with them, so score = 0)

I can’t for the life of me figure out why an incoming link every 6 months should be worth 2 posts a week or two unique visitors a day. I don’t know what an appropriate conversion rate would be, indeed it’s impossible to determine an objective one, but this rate is just arbitrary.


A. The reason is to rate blogs on how good they are. The ratios are necessarily arbitrary as it is an artificial formula based on our nevertheless considered opinion of what constitutes a good blog. The formula combines and quantifies different measurements of blog performance to create a total score that can be used to rank the set. "Objective" is an unhelpful word to employ in understanding this formula.

We are trying to assess a blog's popularity and quality. In order to do this in a way that does not involve personal judgements being made of each blog's editorial content or other aspect only assessable by professional opinion we confined our evaluation to:
  • public support (ie. traffic)
  • peer support (ie. links incoming from other blogs)
  • editorial frequency (ie. posts per week)
    so we may discover their value amongst one another in terms of the nz blogosphere.

    We want this to be done with:
  • transparency (ie. the data must be publicly available)
  • verifiability (ie. every bit of data will be linked to where available)
  • simplicity (eg. an easily understood equation; comments are too difficult to count, but posts can often be counted simply so posts are used and comments are not)

    "An appropriate conversion rate" or weighting is problematic. Some acknowledgment of editorial output seems reasonable, but peer support through incoming links seem and even better prospect, but by far the most important measure must ultimately be traffic.

    Newspapers are rated via readership surveys or sales figures, TV uses electronic sets for rating, and radio use diaries to measure ratings, and so the nz blogosphere being on the internet should be measured in ways appropriate to, and available on, the internet.

    Q. Furthermore, what is a unique visitor? An IP address, yet many organistions share one IP address between many internet users, while many individuals go through several IP addresses a day (at work, at home, Bluetooth, and more if they have dial-up). Simply counting up the number of IP addresses a day does not tell you how many people are visiting a site.

    The major problem I think, however, is relying on Alexa’s ranking of sites for the bulk of the points in each blog’s ‘score’.

    The Alexa system works by recording the sites visited and page-views from everybody who has an Alexa toolbar running and tabulating them to make a rough guide to who’s going where on the internet. But there’s something weird when you look at the stats Alexa produces. The numbers jag about hugely between days


    A. Alexa stats reliability have been dealt with empirically. The Alexa score used is relatively stable as it is not a snapshot but an average. Over a short period of polling day-to-day variance is normal, and remembering weekends have significantly less traffic than weekdays. Alexa is also a very good predictor at the higher end as I have discovered. No alternative method has yet been found.

    Q. Nzblogosphere should give away the Alexa toolbar as a measure of blogs’ performance. A far more accurate and fair solution would be to ask those blogs who wish to be ranked to sign up to a program like google analytics and make the results available.

    A. And the comprehensive nature of the list and what constitutes the nz blogosphere would be destroyed. We include a blog's stats where available. Some bloggers have sent me screen grabs of their stats or the stats they have published themselves and I have used that data. Many bloggers are reluctant to produce the goods, inter alia Public Address and The Standard, and the fact they haven't might *maybe, possibly* indicate that they are being over-counted. The one thing we would never want to do is discount any blogs existence - which means we must include as many as possible. The original base for the list was my own searching and the exploration of the blog rolls of the top 50.

  • 18 Comments:

    At 14/3/08 2:55 am, Blogger Rangi said...

    whinge whinge whinge

    don't worry about it Tim

     
    At 14/3/08 2:59 pm, Anonymous Anonymous said...

    Tim I'm not sure where you get the idea The Standard is "reluctant to produce the goods".

    We've always said we're happy to provide stats, but at the moment we're just having trouble getting any that are up to date for a variety of technical reasons. I'm happy to get you some for next time if Lynn can dig them out for us.

    For what it's worth, I do think what you're doing here is worthwhile, but having seen our actual traffic stats and compared them to Alexa I have to say it has major accuracy issues.

     
    At 14/3/08 3:18 pm, Anonymous Anonymous said...

    Tim. Honestly, I think the rankings are great. I'm just offering a critiquue that I think will lead to a better methodology.

    Alexa is a shitty measure, especially on the scale we are tlaking with NZ blogs - I have comapred it to our pageviews that wordpress supplies and there is no correlation in movements. A much better measure would be hits. Would that advantage or disadvantage Kiwiblogblog? I have no idea. That's not why I suggested the changes.

     
    At 15/3/08 11:15 am, Blogger Idiot/Savant said...

    If you want open, honest, and accurate stats, then do your bit: get Sitemeter or some similar package, and set it up so everyone can read it.

     
    At 15/3/08 12:35 pm, Blogger Tim Selwyn said...

    I/S:
    Damn right. It's very simple really. We use extremetracking, but most people use sitemeter or statcounter. Whaleoil actually uses more than one. It's the best way to assess traffic.

    Tane:
    We've always said we're happy to provide stats, but at the moment we're just having trouble getting any that are up to date for a variety of technical reasons.
    - Still having trouble, eh? You say these things and yet you don't deliver. You haven't sent me a screen grab of your stats, you haven't got any meter installed after more than a month. It is a very simple process. It is reasonable in these circumstances to be skeptical IMHO.

     
    At 15/3/08 2:01 pm, Anonymous Anonymous said...

    Tim, there's no reason to snap dude. I don't actually control the technical end of the site - that's done by Lynn Prentice. He's a busy guy and hasn't been able to do it yet. He'll get it to me when he can and I'll pass it on to you as soon as possible, but I'm not going to harass the guy as he does this on a volunteer basis.

    In the meantime I'll stand by my statement that Alexa's traffic rankings are grossly inaccurate. That doesn't mean I'm dissing your efforts - I think your ranking system is a great initiative.

     
    At 15/3/08 3:52 pm, Blogger Tim Selwyn said...

    Tane:
    You haven't seen snap yet dude. It's hardly harassment to request some stats, like "what was our unique visitor total for February." However I do share your concerns over Alexa and I do sympathise if The Standard's actual traffic is misrepresented by Alexa - BUT I stand by what I say about it being a good predictor for the more heavily trafficked sites.

    The ranking system should be able to respond to new information and systems etc. as well as critiques of methodology. These things are being considered, your concerns are being evaluated.

     
    At 16/3/08 10:09 am, Blogger Russell Brown said...

    inter alia Public Address and The Standard, and the fact they haven't might *maybe, possibly* indicate that they are being over-counted.

    Get a grip Tim! It also "maybe possibly" might indicate that I hadn't grasped you were asking. It's not the first thing I think of when I wake up in the morning.

    FWIW, both Google Analytics and Nielsen NetRatings have Public Address at around 25,000 unique visitors per month (Feb 13-March 14, for example, is 26,422 on Analytics).

    Our internal stats package (awstats) routinely counts much higher -- nearer 40,000 uniques. I think most sites doing both sorts of count experience this difference.

    I suspect that we do suffer from having a large daytime readership at workplaces with firewalls and locked-down browsers -- it's hard for anyone to count users accurately when that's the case -- but the same would apply to Kiwiblog and The Standard.

     
    At 16/3/08 10:15 am, Blogger Russell Brown said...

    I must respond here because I simply will not register to make comments on someone's blog I'm afraid

    Of course in most cases you don't need to, because you've *already registered* with one or other US corporation. Google has your ass.

    we don't force people to register on this site in order to be able to participate, and personally I think the practice should be discouraged

    It works for us. Quite well, actually.

     
    At 16/3/08 2:21 pm, Anonymous Anonymous said...

    Tim, thanks for systematically responding to Wat's comments. As he said in the post and since, they were meant to be constructive and we agree and support your goal.

    I/S, we've tried sitemeter but the free package from wordpress doesn't easily accommodate the widget. Neither Wat, nor I do much of the technical stuff on the blog, that's others' expertise.

    Tim, we have thought about removing the registration requirement but it doesn't appear to hurt our traffic.

     
    At 16/3/08 8:00 pm, Anonymous Anonymous said...

    Hits determine a blog's popularity. And comments. If you are getting comments on nearly every post, you have a popular blog. People like Whaleoil are just pasting MSM writing - I find it very hard to believe he is at the level you say without buying traffic. May get our US corporate to trace it.

    Linkage supports cliques, not worthiness.

     
    At 16/3/08 8:42 pm, Anonymous Anonymous said...

    Shit, Tim - you could get some good spuds out of that spot mate. All the worms you just turned over in that one little square.Amazing.

    *whiney leftie uni voice*

    Boo hoo,cry cry, we're really doing very well, just far to busy to prove it,boo hoo cry cry.

    * end whiney leftie uni voice*

    Get over yourselves guys and girls.

    Really, really, not that important.

    KB is #1 - thats all that really matters, the rest of you are 'also rans', 'Huckabee's' if you will.

     
    At 16/3/08 10:52 pm, Blogger Tim Selwyn said...

    All right, I've got some work to do. These are sensitive issues. For some of you.

    Russell:
    Thanks for your stats. I appreciate you made the effort. My concern was when you commented a few months ago:
    FWIW, PA gets about 5000 visits a day when everyone's at work, and we have somewhere between 23,000 and 28,000 unique users every month.
    I couldn't square your 5000 with 28000 for the month. It didn't make sense. There is
    Page views, Visits, Unique visitors, and Google Analytics have "absolute unique visitors" also. So I must have misunderstood which one(s) you meant.
    If we divide 26000 into 30 we get 866 - as in 866 per day. Which is far too low, but is whereabouts the Alexa score would put it (coincidentally you might say?) There are variances between metering packages that I have personally experienced, but none so great as to be +/- 30%. Are they measuring different things?
    And as for comments registration: It means not only do I not comment, I also rarely view them - so I can't offer a proper opinion on the desirability of the format.

    I am not entirely comfortable with this blog's comments sessions - I've said that before. I've had to issue the odd warning and I've done a fair bit of deleting on occasions (and should have done more) - it's a dog-whistle away from a munter-fest at the best of times - so it can always be improved.

    mardypants:
    This is all good, healthy discussion that will most likely result in a better system. I understand the project is not being critiqued in terms of the general concept.

    You say it hasn't hurt traffic to have registration. Is that all that matters?

    I don't care if it's only me that thinks registration is bullshit, either. I don't care whether it increases traffic or if the comments were from Nobel bloody laureates who through their discussion on your blog found a 1c per person cure for fucking cancer, I'm not signing at the door. Unless of course my blogger ID becomes instantly shared or it was just one button click away.

    Ruth:
    Is the blogosphere a collection of over-lapping cliques? Is the measurement of that factor completely without merit?

    Comments are difficult to count - I must reiterate that. But measuring the rough volume of them or how many do/don't have any comments might be easier. I see your point about this and I'll have a think about incorporating comments as part of the mix.

     
    At 17/3/08 10:03 am, Blogger Russell Brown said...

    I couldn't square your 5000 with 28000 for the month. It didn't make sense. There is
    Page views, Visits, Unique visitors, and Google Analytics have "absolute unique visitors" also. So I must have misunderstood which one(s) you meant.


    There's visits (or user sessions), which are about 5000 on a weekday and the total unique visitors (or "absolute unique visitors") in the course of a month, which is usually more than 25,000.

    The key commercial measure for us in taking advertising is page impressions (or views): we promise 60,000 impressions a week, but usually over-deliver.

    So yeah, I like what you're doing (and I'm happy to send you numbers in future), but I still don't think Alexa is in any way accurate. I've never had it (I couldn't install it if I tried, actually -- wrong browser) and I don't know anyone who uses it. Although it does seem to be useful for fiddling site rankings:

    http://seoblackhat.com/2005/07/23/the-alexa-toolbar-why-you-want-this-peice-of-crap/

     
    At 18/3/08 1:33 pm, Blogger Tim Selwyn said...

    RB:
    If we accept "user sessions" as a measure, we are still at odds with "page impressions" (or page views). The problem is your "key commercial measure" is not credible, because:

    1 Page view: When I go to the PA home page.
    2 Page views: When I click to read your blog.
    3 Page views: When I click to go back to the home page and it just re-loads Hard News again.
    4 Page views: When I click on your home page again and it finally goes there.
    5 page views: When I click to go to another PA blog.
    6 Page views: When I click to go back to the PA home page and it just re-loads that same blog again.
    7 Page views: When I click to the PA home page and it finallt goes there.
    And at this point I get fed up with the double-loading and bugger off and remind myself not to go there again unless it's necessary.

    It makes your "commercial" figure look good, but it's not accurate. It's just re-loading the same page - which will count as another page view won't it? And not putting your full blog on the RSS feed is another smart move from a commercial perspective.

    So my initial skepticism of your figures was based on the double-loading. But now I know that page views and the 5000 per day figure are different measures - that makes sense.

     
    At 18/3/08 2:27 pm, Blogger Tim Selwyn said...

    OK, I've just tried PA on a different computer and there's still a problem going back to the main page - but I don't know if it's re-loading it again. There's something odd going on. If I gave the impression (or view) that it was deliberate, I'll make it clear that was not my take.

     
    At 19/3/08 1:26 am, Anonymous Anonymous said...

    Tim, we agree that the point is to get the best possible measure of the vibrancy and dynamics of the blogosphere.

    My comments about registration was in response to yours which I understood to suggest that registration might chill commentary. I've conflated this with 'hits' but I could have been clearer. My point is that registration doesn't appear to have limited discussion or reduced the contribution, however you might measure it, we're attempting to make.

    Our goal remains to critique NZ public policy with a particularly focus on the online shenanigans of kiwiblog and the centre-right.

    Incidentally, we'll eventually get around to changing our name to reflect the expanding focus.

     
    At 18/5/08 12:44 am, Anonymous Anonymous said...

    E-mail me with what you're after at lprent at primary dot geek dot nz.

    There is a missing section of logs between the end of Jan and mid march. We shifted servers twice. I was more concerned about getting the site running well.

    I store all of the logs on my home server as part of the backup feed. So it'd be easy enough to feed them into awstats or the like. The only reason I look through them is to isolate where the latest spam attacks are coming from and I do that by sampling days in MySQL.

    I set up google analytics a month or so ago when the posters wanted to get an idea of what was going on. That is probably the most accurate as it virtually eliminates the spambots and webcrawlers as they don't run through the javascript links. I don't read that, but I could probably get access if I need to.

    There is also a 'analog' summary at the server as well that reads the apach logs that I can pull some screenshots off for you.

    It hasn't really been that high a priority for me - we don't have anything commercial on the site.

     

    Post a Comment

    << Home