HTML5 Video – Codec Downloader

It looks like the implementation of HTML5 <video> in Firefox will use the native system media framework (DirectX, QuickTime, GStreamer) rather than come with any built-in codecs.

This decision has certain implications for cross-OS web video interoperability. There are very few or no formats that Windows, Mac and free Linux all can and do support.

So a Firefox codec downloader might be a good addition to core Firefox. Like the plugin finder, when it encountered media using a codec which it could not read, it would ask a codec finder service if a codec was available, but then instead of installing a browser-specific plugin it would install the codec at OS level.

So if a Windows user came across an Theora file, Firefox would (ideally) say something like “I can’t play this – get codec?”. If the user said yes, Firefox would go away, download DirectX Theora and Vorbis codecs, install them at OS level, and then start playing the video via DirectX as normal. As a side-effect, players like Windows Media Player would also gain the ability to play Theora, because it uses DirectX too.

Similarly, if a Linux user came across an MPEG4 file, a similar thing could happen (if the necessary licensing agreements were in place). There are trickier issues with binary compatibility here, but perhaps they could be overcome if the data in the codec finder service were well-maintained.

This would be a great aid to smoothing the user experience for all sorts of video on the web, and helping content authors publish without worrying too much that some people couldn’t play their format. Chris Double (the <video> guy) told me he liked the idea, but didn’t have time to code it. Do any of you? Or do you know anyone who cares about web video interoperability who does?

27 thoughts on “HTML5 Video – Codec Downloader

  1. Sounds like a good idea, especially with the ‘halo effect’ of enabling more file formats on a user’s machine.

    Does HTML5 recommend/require any specific formats for use with <video/>? Obviously Theora would have the lightest licensing restrictions but as you say it would require extra software on Windows on OS X, and MPEG 4 locks out some free Linux users.

  2. Wouldn’t a better solution be to hold off implementing video altogether until a good codec is decided upon? This has the potential to lock out some OS’s permanently from web video and Firefox shouldn’t do that.

  3. @Nick – I believe this decision has been made because a good codec can’t be decided on; if you ask the browser makers what codec should be used Apple will say MP4/Quicktime and Microsoft will say WMV, and never the twain shall meet. I think as long as getting the correct codec is made as easy as getting Flash, the solution above is the best one.

  4. I’d still prefer to have at least one free speech codec to come with Firefox, with autodownloads filling the gaps. This would ensure interoperability between Firefox users on all platforms. Autodownloads aren’t all that reliable, especially if the user does not have the rights to download and install codecs. Plus obviously autodownloads are an obvious push for formats most people don’t want to see in the web, like Windows Media, which works for Windows but e.g. not on Linux (either users would have to download an unlicensed stack of decoders, pushing users into the darker side of legal-or-not, or they have to buy proprietary decoders with Microsoft’s blessing – those exist for Gstreamer).

    One way or another, you’re pushing all the nastyness of non-free formats onto the users (“click here if you’re sure you have no idea about patent licensing”) with an automated codec delivery system and potentially scaring them with disclaimers to keep yourself out of trouble.

  5. As a computer doctor, this is how many of my patients get infected, via “codec” downloads through Windows Media Player on Porn sites.

    Asking if someone wants to install something in order to view a video is really only asking:
    “Cancel or Allow something you don’t understand to happen, but will get this video to play”.

    This problem needs to be solved in a way that doesn’t ask people a security question they do not know how to answer. Asking to install a Codec Yes/No doesn’t give you any information about how trustworthy that codec is, only that you can either choose to continue, or cancel.

    A central repository of vetted codecs needs to be kept, they could even be extensions/plugins stored on AMO and installed much like as the Plugin Finder Service.

  6. Neil T.: HTML 5 currently makes no recommendations because there is no known codec which meets all its criteria.

    Maik: The question of shipping particular codecs with Firefox is a complex one, and I don’t want to get drawn into it here. Suffice it to say that at the moment, it’s not looking like that will happen.

    I agree that pushing the problem on to the user is not good. With any luck, people will choose Theora rather than patented forms, so the autodownloading will be on the Windows side, and there hopefully won’t be legal issues.

  7. Gervas, the plugin finder never ever worked on Linux. So I’m not so excited to have an equally useless codec finder. It’s not about binary compatibility, it is about package system integration, which is hard and needs to be maintained.

    I know it’s hard to choose a common codec, but that’s why people support Firefox, to push for standards on the Web. So please… push a little more.

  8. Gervas, the plugin finder never ever worked on Linux.

    Works fine here on Fx3 at least. Only time it hasn’t found something is for Shockwave but that always happen on Windows too for me.

  9. Flavio: I would hope the service would be written in a way which allowed Linux vendors to plug in their own services which supplied their own packages. This is not an alternative to pushing for standards. But I promise you, the lack of agreement so far has not been for want of pushing.

    Kroc: I would anticipate that the codec finder service would be run by us, and point only to known good and safe codecs. The fact that people click Yes on any dialog which says “Click Yes to view the kittens” is out of scope for this discussion. Unless you think we should install the codec without asking? :-)

  10. Do not implement this by using the system framework. At least not on windows. I have done this for a computer game a couple of years ago. This (mis)feature stands out in that it has caused by a wide margin the most support incidents of all products I have ever worked on. And I can assure you that it wasn’t a bug or error in the game. You will see that a lot of your users install codecs or worse codec packs without thinking twice. Most of these codecs are outdated, buggy and faultily installed or registered. And this are the good ones not riddled with malware and viruses.

    There is a reason that media players like VLC package all important codecs and use them directly. There is a reason that games use private codecs (mostly bink or divx) without using DirectX/DirectShow.

    If possible settle on a couple of important codecs. Make them part of your application and use them directly. So you know what you have to support and don’t get blamed for bugs that aren’t caused by your software.

  11. You really should not do something firefox centric for codec in linux, just allow distribution to integrate their existing codec finder (codec buddy).

  12. With GStreamer there’s already infrastructure for detecting missing codecs and prompting the user to install them. They integrate with whatever package system is on the computer and will even offer to take you to Fluendo’s webstore for legally licensed MPEG and WMV codecs if you don’t want to use the dodgy-ffmpeg ones.

    Aren’t there already codec-finder services on windows & mac. I remember my Mac opening a web page and prompting me to download quicktime codecs.

    Ideally Firefox would use GStreamer across all platforms. That’s where we’re moving to with Songbird. Our most recent hire, Mike Smith has a ton of GStreamer experience and is bringing up the final pieces we need to do a really kick-arse implementation across platforms.

    Ian

  13. S. T.: You need to take that up with Chris Double, not me.

    Ian: will the Windows codec finder service offer you a Theora codec if you need one?

    I think it’s unlikely that we would choose to bundle GStreamer with Firefox for Windows and Mac. But again, talk to Chris Double.

  14. I don’t feel like reading the specs, so here’s a couple lazy questions:
    1) Does the <video> markup list the codec required right in the tag, or does the video need to be downloaded and analyzed by the browser to determine whether or not the appropriate codec is on-hand? (I’d hate to see my browser downloading large files I will never view)

    2) Does the spec provide for listing multiple formats in a single <video> tag? I mean, can the content distributor provide a single video tag which points to multiple different encoded versions, and the browser can just display the first one that it has a codec for?

  15. 1) The markup can list the codec required for each stream
    2) the spec allows for multiple formats in the tag

    Gerv, it’s not “use the system framework *rather than* built-in codecs”, it’s *in addition to*. There probably isn’t an open source video codec we can ship — and that’s definitely a discussion for another time — but we can probably ship Ogg Vorbis for audio and I’m hoping we will.

  16. “There are very few or no formats that Windows, Mac and free Linux all can and do support.”

    I know one, and it’s already the most widely-used on the Web…. ;-)

    jd/adobe

  17. John: I think you missed the “free” in “free Linux”. Although if you count gnash, yes, I guess that Flash is supported in some way across all three. Although my understanding is that the video codecs used in Flash have similar patent restrictions, and are therefore unsuitable for speccing as part of HTML5. I’m not sure how gnash deals with that problem. Maybe it ignores it.

    If Macromedia were able to grant royalty-free patent licences to the video codecs which Flash uses, then I’m sure the HTML5 working group would be very interested. :-)

  18. “… my understanding is that the video codecs used in Flash have similar patent restrictions….”

    True, that. Good codecs do cost. Adobe handles that licensing, and also gets it deployed onto the world’s various types of consumers systems for you too. You can rely on Flash Player capabilities, just like you can rely on PostScript when you hit “Print”.

    It’s like how Adobe (and Macromedia) paid the licensing costs for .JPG production, invisibly, although I suspect this didn’t cost as much overall as creating a standard video capability atop the world’s varied computers.

    jd/adobe

  19. Usally FlashBlock protects me from annoying Flash adverts but two seem to have slipped through on this page.

  20. John: So good codecs cost, but good printing systems don’t? After all, my computer runs a Free version of PostScript just fine. And, for that matter, it decodes JPEG images without paying any licensing fees.

    Also, please don’t conflate the actual monetary costs you incurred in achieving near-ubiquity for Flash with the idea that there is some monetary cost necessarily associated with using a video codec. That’s a fallacy. There is nothing intrinsically special about video which means that there has to be a licence fee associated with software used to view or manipulate it.

  21. Yes, that’s true, Adobe published the PostScript specification, and there were various third-party implementations, but no patents like we saw later with video codecs. Different scene.

    (To my recollection, it was JPEG viewing which was free-to-implement, but JPEG creation which required licensing. Macromedia, Adobe and other firms ate those fees, so .JPEG creation was seemingly free-of-cost to content creators.)

    I agree with you that video codecs don’t *necessarily* need patents or licensing… the good ones currently do, though. That’s why the question of achieving predictable runtime capability is such a key issue.

    On a related note, did you catch reports yesterday of putative codec downloads actually reconfiguring routers? That’s a scary one:
    http://blog.washingtonpost.com/securityfix/2008/06/malware_silently_alters_wirele_1.html

    jd/adobe

  22. > 1) The markup can list the codec required for each stream
    > 2) the spec allows for multiple formats in the tag

    I was a little dubious about the codecs (fallbacks for *mime types* have been around for ages, which doesn’t help when you might have one of a million whizz-bang codecs in the same container format), but was pleased to see that, indeed, they’ve extended the media type with a ‘codec’ option for this purpose: http://www.whatwg.org/specs/web-apps/current-work/#source

    Neat!

    At Wikipedia & Wikimedia Commons, we’ve standardized on Ogg Theora+Vorbis for videos, and have a rather scary little system for inline playback which attempts to detect possible playback support via:

    * Plugin supporting application/ogg via <object> or <embed>
    * Java applet
    * <video> element (from back when the HTML 5 drafts required Theora… ahh those were the days! We’ll have to update it to specify the codec I guess.)

    The question for cases like ours tends to be, how can we attempt several alternate methods transparently without triggering “download this plugin!” or “download this codec!” dialogs.

    If we can reliably use a <video> element (maybe even when JavaScript is disabled) that would fallback transparently to the plugin and Java applet tests, that would be super… and if they don’t have _any_ support, then prompting them for a Theora download would be great.

    But given a choice between prompting the end-user to download something (Theora plugin) and *not* (Java applet if they have Java already), I’d rather avoid the prompt.

  23. Isn’t MPEG out of patent protection *yet*? Man, twenty years is *forever* in internet time. That’s a format that was in widespread use for video back in the days of NCSA Mosaic, when Yahoo was on akebono for crying out loud. That was… [does the subtraction]. Okay, so it was less than fifteen years ago. Sheesh, twenty years is a veritable eternity on the internet. I can’t even imagine trying to use a web browser from five years ago on today’s web, let alone fifteen!

    > So if a Windows user came across an Theora file, Firefox would
    > (ideally) say something like “I can’t play this – get codec?”.
    > If the user said yes, Firefox would go away, download DirectX
    > Theora and Vorbis codecs, install them at OS level, and then
    > start playing the video via DirectX as normal.

    I don’t think it’s sane to assume that the end user necessarily has permissions to install software at the OS level, and I don’t think it’s good to encourage systems to be set up that way. (I know, there are other applications that encourage or even require this, but that’s a Very Bad Things and Needs To Stop, and security-minded applications like Firefox absolutely should not be contributing to the problem.)

    If it can’t be installed in the user’s profile like other add-ons, it needs to either ship in the installer with the application, or be installed separately by the sysadmin.

  24. As outlined in a recent blog post I think they should go with gstreamer cross platform. Its sad that Mozilla can not directly acknowledge the patent situation.

    This essentially screws the free media software community by not giving any specific patents that could be work around to be non-infringing… instead Mozilla simply says “we can’t give you free media and we can’t tell you why”. If one of the most well funded and well resourced open source projects on the web can’t ship free media than what chance do smaller players have to realize mozilla stated goals?