Andkon has caused a bit of a stir with a new critique of the www.mozilla.org front page. However, I’m going to leave his comments on the content to one side and concentrate on his mathematics.
His assertion is that “only 50% of IE users who come to www.mozilla.org download Firefox”. He works this out by taking the number of IE-user downloads of Firefox per day (estimated at 50% of total by Asa) and dividing it by pageviews per day of www.mozilla.org and www.mozilla.org/products/firefox/, which he classes as “end user pages”.
The flawed assumptions in this logic include, but are not limited to:
- all views of those two pages are from people wanting to download Firefox – which is obviously not true for www.mozilla.org, and probably not true for /products/firefox/;
- a particular downloader only views a single one of those pages, rather than both;
- a particular downloader only views a particular one of those pages once – they don’t, for example, read about Firefox at work and then download it at home;
- all attempted downloads after viewing a page succeed, and aren’t cut off by e.g. network trouble, a little sister picking up the phone extension, or some other reason (the download stats count completed downloads).
But even if we ignore all the above and take his figures as correct, other businesses would kill for a 50% conversion rate of interested people to customers. In the circumstances, I’d say we are doing rather well.
He may well be right that the front page could use improvement – but that depends on your underlying aim for it, which is the real point of controversy. Some believe that www.mozilla.org should be focussed on all that mozilla.org does, not just Firefox, and we should use e.g. getfirefox.com to plug Firefox exclusively. Others think www.mozilla.org should focus on whatever the flavour of the month is. Without agreement on this point, suggested changes to the front page will founder on the deeper disagreement.
He is right on at least one thing, though. The phrase “declaration of independence from a monopolized and stagnant web” has no place in our newspaper advertising. It’s meaningless political posturing, and a big turn-off. (I suspect and hope it’s not going to be there, and it’s just something Blake thought up for his blog post.)
First of all, I didn’t think up that phrase.
Second of all, running an ad campaign of this magnitude requires an ability to understand what kinds of appeals will entice what kinds of audiences. In this case, the declaration of independence analogy is something used for the open source/Linux/techie crowd. As the campaign has already raised over $170,000, I’d be curious to know on what basis you claim the idea is a “big turn-off”. Until I hear convincing evidence, I’m not particularly inclined to feel insulted.
Good – that’s what I’d hoped :-) I think you’ve misunderstood me; I said that the phrase had no place in the advertising, not in the campaign to raise money for the advertising. The success of the campaign, as you say, shows that you’ve promoted it very well, and I congratulate you on that.
My point is that (as some have suggested or feared) the phrase should not appear in the advert itself. And it seems from what you say that this isn’t going to happen. So that’s great :-)
I agree. But I trust the SFX team (or whoever is in charge of this matter) to make good marketing decisions. I’m really looking forward to see the ad. Or ads? :-)
Gerv: sounds good.
I think you missed the biggest flaw in his statistics. I don’t see anything in his post about how he gathered these statistics saying that he paid any attention to what browser people were using. Firefox comes with the start page set to one of his end user pages. SO, if you download firefox and like the start page and never change it, he is counting you as one of the potentioal converts who never downloaded because you were turned off by the bad homepage every time you launch the browser.
Surely, the metric of “IE Conversion Rates” is very useful. As you point out, Andkon’s methodology is flawed, so what is a better methodology?
You are absolutely right that a 50% conversion rate is incredible. A 51% conversion rate is even more incredible (and approximately an extra 8000 downloads per day).
The place to focus attention is on http://www.mozilla.org/products/firefox/ — this page is the equivalent of your salesperson. Usage stats indicated that a large percentage of visitors to mozilla.org are only interested in this page. So, do we have a good salesperson or the best, darn salesperson possible?
“dividing it by pageviews per day of http://www.mozilla.org and http://www.mozilla.org/products/firefox/,”
WRONG. dividing by visits per day.
And yes, I pointed these critiques out in my own essay as well.
50% is not great by any means: half of all IE users then leave saying “Well, Internet Explorer’s better.”
PS Do change the error from pageviews to visits. It presents a better case for me, though as I do argue in my essay, it’s not conclusive. But is there a better gauge for conversion rates?
well, the truth is, the new design of mozilla.org front page is not as good as ppl expected, especially the row of products, and not easy to navigate.
No end-user would like to read all those blogs, and just count the links in the frontpage, how many of them is actually meaningless?
A 50% success rate is AMAZING! Only a complete idiot with a complete lack of understanding website statistics (or a troll) would think otherwise. It is utter bullshit to ASS-U-ME that the other 50% do not download the product because they don’t like it, because there are many reasons why people visit websites. Andkons statistics are horrible, but even if he was right it still wouldn’t matter: a 50% conversion rate will make a very noticable impact in the IE market share.
I completely agree with Andkon suggestion for the Mozilla and FF pages, but not with the calculations.
“If we split the estimated visits per day to enduser pages as evenly as the downloads,…”
I see no reason for this assumption. I guess that there are many FF users who visits regulary these pages, for updates, news, entry pages to other site content or just because it is their home page. It is highly reasonable to suppose that most of the Mozilla site visitors are Mozilla users. If we suppose that IE visitors are only 25%, than we will have 100% success. I guess the truth is somewhere between.
‘50% is not great by any means: half of all IE users then leave saying “Well, Internet Explorer’s better.”‘
I don’t agree. It is common to be exposed for advertisement more than one time before downloading. It may be also, IE users leave saying “Well, nice product, but I have no time no (I just entered here to see what is that Firefox from the Get Firefox button). I will download it tomorrow.”
I have to strongly disagree with the ‘50% is not great by any means’ assertion. Over the last couple of months, the tech journalism arena has run with the story that IE has lost market share for the first time ever. They consider it a major story that alternative browsers like Firefox have eaten a *full percentage point* off of IE’s dominance, down to a measly 98% percent of the market.
I guarantee it will be front page news on tech papers if six months after the initial release of Firefox, stats show IE usage down to anywhere under 90%. That could well be a tipping point that causes sites to code to Firefox compatibility and allow people and organizations to seriously consider switching browsers without losing functionality.
It will also give Ballmer something else to spin about in the news… :-)
50% is fantastic! 50% is amazing! yay, 50%
Now, can we move on to how to bump it up to 51%?
Dumb question: could you use cookies to check how many times a browser visits the download page before downloading? Obviously, they’d need to have cookies enabled…
“Well, nice product, but I have no time no (I just entered here to see what is that Firefox from the Get Firefox button). I will download it tomorrow.”
You’re making even more assumptions than I am.
The method is flawed, and I suspect the 50% conversion rate is much higher than the truth.
As for the site designs, I think changing them in the middle of a massive marketing campaign can only be a mistake.
Marty, I agree, that’s why I recommend a split run.
I don’t see how you’d do a “split run” or what it would tell you. If you do it live with the website, then the data you collect will be unreliable, and you’ll confuse and annoy a whole bunch of people into the bargain. And if you do it in controlled conditions, then I don’t see how you’d test it – monitor thousands of IE users in the hopes that they’ll happen across the site by chance? or direct them to the site and tell them to imagine that they’ve just come across it by chance? neither seems satisfactory.
With cookies, the split run would show who downloads and who doesn’t, based on the designs. Simple to do and the results would be precise, unlike my essay’s stats.
Um, I’m not sure what you mean by “the data you collect will be unreliable”. Could you please clarify what you mean?
As far as confusing and annoying people, I don’t see why this is the case. Yes, a new design will be different to us, but it will not be different to most users. It will be the default for them. But, to test your hypothesis out, we should start by doing split runs on wording and content before testing out new visual designs.
I agree that “controlled conditions” make no sense. You’d be better off with a 5 user usability test.
Doing a split run is easy: redirect 1, 5 or 10% of vistors to the new page you are testing. What you’d be measuring is the IE Conversion Rate. If it goes up, then your new page is better than the original.
I should add that even if the calculation methodology for IECR (IE Conversion Rate) is flawed, the split run can still work. As long as the metric is correlated to the actual phenomenon (i.e. higher IECR indicates a greater actual conversion rate), then it’s OK.
Finding a way to get an exact match between the metric and reality would be ideal, because then Andkon’s analysis and my 8000 more downloads a day would be based on more than a rough approximation.
Andkon: could you explain where you got your visit numbers from, then? Because the only per-page visit numbers in the Webalizer stats are exit visits. How do you know a download counts as an exit, given that people are downloading from within the same domain?
Actually, no – because he’s suggesting that some people might do that, and you are assuming that no people do it. The latter is a greater assumption.
Yeah, but if the 50% figure is actually 100%, then it can’t be bumped up no matter what we do.
Also, there’s a cost-benefit analysis you need to do. Is it worth the time and trouble to set up the tracking system (from people who are very busy doing other things), the potential PR backlash from tracking people in this way, the disruption and confusion potentially caused by having two versions of particular pages? You’d need to assess how important the “other things” were relatively.
Alex: any data collected like that is unreliable. You’re relying on something from the browser – script, cookies, user-agent, referer headers. All of that stuff can and does get messed with (changed or blocked), by users, proxies, bots.
A key thing in this case is that the downloads happen from a bunch of mirror servers, and the information from them is more limited. To do the “split run” thing and get useful data, you’d not only have to change the initial page, you’d also have to restructure the systems used to deliver the rest of the site and the downloads. Currently there is no way of tracking a “session” or “visit” – you can tell that something has downloaded a particular page, and you can tell that something has downloaded a file, but there’s no way of tying things together so that you can figure out what a particular person has done.
I use both Firefox (Win and Linux) and Konqueror on Linux. I know of at least four or five instances where I visited the firefox product page while guiding a friend/relative how to download and install firefox over the phone. Also I sometimes check the firefox page just to see if there’s anything new or when I read about a firefox security issue. I am sure I am not the only one viewing the page without any intention of downloading.
Of course, if you manage to bump up conversion rate by one or two percentage points, that would be great, but 50% looks pretty good already.
“What are the enduser pages? These are the pages where we can expect to find those wishing to download Firefox, including IE users who probably have not visited Mozilla.org before (first-time users). The only two pages which apply are mozilla.org and mozilla.org/products/firefox/. The percent of visits started at enduser pages is the sum of the percents of both the mozilla.org and mozilla.org/products/firefox/, as linked to in the webalizer stats. The average number of visitors per day is also taken from the webalizer stats. The estimated average number visits started at enduser pages per day is just a product of the two previous cells.”
I never counted pageviews, just *starting visits* made at specific pages.
> there’s a cost-benefit analysis you need to do
Fantastic! Now you’re talking about something we can compare. Consider the NYT ad:
Cost:$50-75K (I’ll pick the most favourable conditions, so let’s say $50K)
Benefit:
People who see ad:
Circulation of NYT: 1.13 million (Source: http://www.nytadvertising.com/was/circulation/pages/contentCirculation/0,1013,,00.html?l1Id=5&l2Id=52)
Readers per Copy: 1.7 (Source: http://www.mmrsurveys.com/ppctotal01.htm)
Total People who see ad: 2.01 million (Source: http://www.mmrsurveys.com/ppctotal01.htm) (Note: 1.13*1.7 is less than 2 million though)
People who download Firefox as a result of ad:
Percentage who see ad and visit: 5% (high assumption)
New visitors: 100,500
Percentage who download: 97% (very close to Gerv’s 100% argument)
New downloads: 97,485
There’s is also a huge word-of-mouth benefit and additional publicity from stories on Slashdot and mainstream media about the Firefox ad. This could be pretty substantial, so let’s multiply the new visitors by 10:
New visitors from indirect benefits: 1.005 million
Percentage who download: 97%
New downloads: 974,850
Total new downloads (direct and indirect) from ad: 1.07 million
Cost/benefit ratio: 1.07 million/$50K = 21 downloads per dollar (4.66 cents per new download)
This is incredible! Of course, my assumptions are wildly optimistic, but let’s now consider a pessimistic scenario for improving the download page:
Optimizing Download Page
Cost:
Split-run architecture:
Assume $150/h for an independent consultant (not using open source community)
Assume 30 days work with 8 hours days or 240 hours
Total labour cost: $36,000
Equipment cost: assume $5,000
Usability testing:
3 rounds with 5 users a round conducted by Jakob Nielsen (highly regarded authority in Usability circles)
Each round lasts one day (8 hours) and requires 8 hours setup. Participant time: 40 hours. Facilitator time: 16 hours
Pay for participants: $25/h, $1000 total
Pay for facilitator: $250/h, $4000 total
Total cost for 1 usability test: $5,000
Cost for all 3: $15,000
Grand total (split-run and usability): $56,000
Benefits:
Daily downloads: 150,000 (Source: http://www.spreadfirefox.com/?q=node/view/4935)
Assume 75% of downloads are from IE users (other 25% are upgrades, other Gecko browsers, etc)
Daily downloads from IE users: 112,500
IECR: 97%
Daily page visits from IE users: 115,979
Assume a 1% increase in IE conversion rate per round from 97% baseline for each usability round (98% after round 1, 99% after round 2, 100% after round 3)
Using these extremely pessimistic assumptions, each 1% increase translates into 1160 more downloads per day.
If we space the usability tests out by 30 days each, then here is our progress after 90 days from completion of the first round of improvements:
Days…New downloads during time frame
30…..30*1160 = 34800
60…..30*2320 = 69600
90…..30*3480 = 104400
Total after 90 days: 208,800
After 365 days: 208,800 + 275*3480 = 1,165,800
Cost/benefit after 365 days: 1,165,800 new downloads/$56K = 21 downloads per dollar (4.8 cents a download)
Thus, even with wildly favourable assumptions for the NYT ad and pessimistic assumptions for download page optimization, the cost/benefit ratios are roughly equal (4.6 vs 4.8 cents per download).
Moreover, I have assumed that the download page is “nearly perfect” (97% IECR). If the IECR is lower, then the cost/benefit ratio is greater, since you can continue to optimize until you hit 100%.
Other concerns:
-user confusion over 2 download pages
-PR backlash over tracking
-too busy doing other things
User confusion:
I doubt that many users would get confused. Take a look at the difference between amazon.com and amazon.ca . Oh my god, they moved the search box! I’m so confused! How do I search for books? I’ll go to barnesandnoble.com instead… Yikes, the colour is different. I can’t figure out how to search for a book I like. Obviously the task of “searching for a book” is far more complicated than “downloading software”. To download, you click on the “Free Download” link. You could change it to “Download Now” and do a split run to see if that slight change in wording improves your IECR by 0.01% (a couple hundred users per day), but no, it might be TOO CONFUSING!
PR Backlash:
Does Amazon face a backlash over their use of tracking? Try this: 1) delete any amazon cookies you have; 2) go to amazon.com (don’t click anywhere); 3) check what cookies you have from amazon.com. See those ones called “session-id” and “session-id-time”? Who knows what they are gleaning from user agent strings and referer info, but anyone who has a lot of traffic to their website should be doing more sophisticated tracking than looking at webalizer.
Too busy:
Is a CEO “too busy” to increase revenue and profit? The success of Firefox is not measured by revenue or profit. It’s measured by market share and downloads. Take a look at what else you are doing. Can you show that it can attract new downloads at a cost of less than 5 cents each? If you value your time at all of $1.20 an hour, that 5 minutes you spend to convince or help a friend to download Firefox costs you 10 cents!
As the number of visitors to the download page increases, the cost/benefit ratio of page optimization increases too. In fact, right now, download page optimization as measured by IECR is the single greatest opportunity available to increased the spread of Firefox.
To sum up:
-NYT ad was an awesome idea and I’m sure we’ll see incredible results
-optimization offers an equally compelling opportunity and is cumulative with any advertising campaign
Ok, fellas…here’s a reality. We’re talking about 50% of IE users who visit the site, right? For the typical IE user, 50% is pretty impressive. The typical IE user knows nothing about the web or about browsers. He just knows that there are things about it that annoy him. Crap, I’m a consultant/tech, and the majority of people I do end-user support for still call IE “The Internet.” To them, there is no distinction. And if IE tells them to D/L something with a nifty popup window, then it must be OK (hence, the amount of money I make from removing viruses and spyware–relatively simple jobs they’re afraid to do–and they have *NO* idea how it got there).
My point: most people will be afraid to switch at first sight. You’d probably have a much higher conversion rate if you used an ActiveX control to pop up the old “Do you want to install and run Mozilla Firefox from The Mozilla Organization…” window. Most average IE users will click “Yes.” Of course, this would be unethical.
Trust me, I do EUS to make a living…and I can tell you that a huge percentage of IE users will *NOT* download it just because it sounds good and just because you tell them to. I don’t care how well the site is designed (although a good design does go a long way…this one’s better, but do keep tweaking it.) There’s a huge number of them who don’t even know what a web browser is (even though they use one almost daily).
On the other hand, I personally have a nearly 100% conversion rate with my clients. I tell them, “Hey, end the spyware, the viruses, and the popup windows. US-CERT recommends it, and I use it every day. It’s better, faster, and if it breaks it won’t bring down your whole computer.” The typical reply is, “Really? What’ll it cost?” I tell them, “The time to install it…less than 5 minutes.” And they respond, “Do it, then.”