The author failed to question software patents directly, but merely considered whether the patent-holders have announced that they will seek royalties. We learned enough of a lesson from JPEG, GIF, and PNG last time; we should not just demand that our image formats be royalty-free, but that they be patent-free.
It’s a hard problem because there are so many existing patents and you even have to worry about submarine patents that are filed but not published yet.
AOMedia has a multi-fold approach for AV1/AVIF:
Use relatively safe building blocks
old stuff (e.g. the DCT)
things AOMedia members have patents on (and have put under a royalty-free license)
tech already widely used in e.g. VP9 (since it’s been out for years, there’s some hope that the trolls would have already come out of the woodwork)
simple extensions of tested ideas (e.g. adding more DCT shapes or prediction directions)
things novel enough (e.g. CDEF, ADST, non-binary arithmetic encoding) that likely nobody else patented them
Put everything through legal review
Defensive patent license: anyone suing over AV1 can be sued by any patent holder for infringement–helpful for deterring product companies from attacking the codec, though pure IP licensing companies are still a threat
Have a legal defense fund
I don’t know about JXL’s defense approach in as much detail. In JXL’s favor, it’s a simpler codec; the AV1 spec is several times longer than JXL’s.
You can’t completely zero out patent risk; an IP holding company that did not join AOMedia is already trying to squeeze some money by licensing allegedly-relevant patents, though I don’t see that they’ve worked the nerve up to sue anybody. This is still better than the HEVC situation where they just made no effort to avoid patents and there are now multiple groups claiming to have patented essential parts of the codec.
It’s still a bad situation, less because of imperfect efforts to make free codecs than because the patent system is just incredibly broken as applied to software and the current technological environment in general.
It’s so fucking horrible that all kinds of innovation in the area is completely stifled by patents. Imagine if the best compression engineers could just get together and design the best video compression algorithm possible without worrying as long as they keep clear of copyright issues. It’s ridiculous that companies can patent the math, then not do jack shit with the math they patented, except for occasionally fuck over projects which came up with the same math independently.
The author worked on an image format. In a very real sense, that would be demanding something about their image formats, not our.
Software, protocols and formats all share the trait that their terms are decided by those who work on them or pay for that work, not by people who wait until the result is shipped and then want to use it for free and without restrictions. Open source is a gift and a fine one. People who receive gifts and them demand more do not behave well.
You don’t have a responsibility to “give away” your work freely. But you do have a responsibility to not actively work to make the world worse. If you’re making a patent-encumbered image format, and you’re fighting to make your format ubiquitous, you’re fighting for a decidedly worse world than our current world where most popular image formats are relatively unencumbered.
FUIF is royalty free and it has a reference implementation that is free and open source software. Rationale: we don’t want no patent mess, please.
The terms of use of software, protocols, or formats are given by the users themselves. It is up to our society to understand that these things are not scarce goods and that economics do not determine how they behave.
Some gifts are raw and need to be cooked; cryptographic protocols and compressed formats usually get hardened by peer review. It is hardly a demand to publish a critique to a published prospective format. But the very nature of the critique requires us to already accept that prospective formats are not claimed as private intellectual property. Indeed, some software vendors use licenses to try to forbid users from benchmarking, debugging, improving, extending, or otherwise altering their formats or protocols.
I don’t know why WebP2 even gets a mention here. AFAIK it’s only a single developer exploring an idea. It’s not on track to become a major format.
If Google wanted a practical “WebP2”, they best thing they could do is upgrade WebP’s VP8 to VP9. That would be a significant gain with nearly zero hassle, since everyone who ships WebP already ships both decoders
But AVIF is conceptually very close to WebP, but with a newer codec. AV1 is based on VP10 + contributions, with reference encoder literally built from libvpx codebase, so AVIF is basically a “WebP3” already.
Moot now but it’s kinda surprising they didn’t update WebP right after they released VP9. Wouldn’t be as good as today’s stuff, but it would’ve been solid for ~2013, and they had all the pieces.
VP9’s decoder / encoder was pretty slow at then with most Android devices. WebP has similar perf characteristics to jpeg (more or less at that point in time).
I can’t really predict how it will play out, but I suspect JXL could carve out a durable niche on the Web thanks to the progressive loading and transcoding features (see something very soon after loading a page + migration path for existing .jpgs) and a nice progressive lossless mode (greatly improved FLIF).
AVIF seems on track to get broad hardware support; I could imagine your phone camera saving in it because of that, for example. It is also better at maintaining presentable results when a file is very heavily compressed; it tends to just lose low-contrast detail instead of adding attention-getting DCT artifacts or blurring sharp edges. (Maybe that makes heavily compressed AVIFs of very high-res (4K+) images another niche? Not sure.) AVIF’s use of spatial prediction, which JXL does not have, tends to make progressive decoding harder but handling sharp edges easier.
HEIC uses the same set of compression tools (HEVC/H.265’s keyframe compression) but with a different container format–both the compression pluses/minuses and the IP situation should be pretty similar.
The author failed to question software patents directly, but merely considered whether the patent-holders have announced that they will seek royalties. We learned enough of a lesson from JPEG, GIF, and PNG last time; we should not just demand that our image formats be royalty-free, but that they be patent-free.
It’s a hard problem because there are so many existing patents and you even have to worry about submarine patents that are filed but not published yet.
AOMedia has a multi-fold approach for AV1/AVIF:
I don’t know about JXL’s defense approach in as much detail. In JXL’s favor, it’s a simpler codec; the AV1 spec is several times longer than JXL’s.
You can’t completely zero out patent risk; an IP holding company that did not join AOMedia is already trying to squeeze some money by licensing allegedly-relevant patents, though I don’t see that they’ve worked the nerve up to sue anybody. This is still better than the HEVC situation where they just made no effort to avoid patents and there are now multiple groups claiming to have patented essential parts of the codec.
It’s still a bad situation, less because of imperfect efforts to make free codecs than because the patent system is just incredibly broken as applied to software and the current technological environment in general.
It’s so fucking horrible that all kinds of innovation in the area is completely stifled by patents. Imagine if the best compression engineers could just get together and design the best video compression algorithm possible without worrying as long as they keep clear of copyright issues. It’s ridiculous that companies can patent the math, then not do jack shit with the math they patented, except for occasionally fuck over projects which came up with the same math independently.
The author worked on an image format. In a very real sense, that would be demanding something about their image formats, not our.
Software, protocols and formats all share the trait that their terms are decided by those who work on them or pay for that work, not by people who wait until the result is shipped and then want to use it for free and without restrictions. Open source is a gift and a fine one. People who receive gifts and them demand more do not behave well.
You don’t have a responsibility to “give away” your work freely. But you do have a responsibility to not actively work to make the world worse. If you’re making a patent-encumbered image format, and you’re fighting to make your format ubiquitous, you’re fighting for a decidedly worse world than our current world where most popular image formats are relatively unencumbered.
Quoting the author themselves in their prospectus for FUIF, the predecessor to JPEG XL:
The terms of use of software, protocols, or formats are given by the users themselves. It is up to our society to understand that these things are not scarce goods and that economics do not determine how they behave.
Some gifts are raw and need to be cooked; cryptographic protocols and compressed formats usually get hardened by peer review. It is hardly a demand to publish a critique to a published prospective format. But the very nature of the critique requires us to already accept that prospective formats are not claimed as private intellectual property. Indeed, some software vendors use licenses to try to forbid users from benchmarking, debugging, improving, extending, or otherwise altering their formats or protocols.
Is it just me, or does WebP/WebP2 feel like another of those format/protocol death marches employed by Google to frustrate other browser vendors?
I don’t know why WebP2 even gets a mention here. AFAIK it’s only a single developer exploring an idea. It’s not on track to become a major format.
If Google wanted a practical “WebP2”, they best thing they could do is upgrade WebP’s VP8 to VP9. That would be a significant gain with nearly zero hassle, since everyone who ships WebP already ships both decoders
But AVIF is conceptually very close to WebP, but with a newer codec. AV1 is based on VP10 + contributions, with reference encoder literally built from libvpx codebase, so AVIF is basically a “WebP3” already.
Moot now but it’s kinda surprising they didn’t update WebP right after they released VP9. Wouldn’t be as good as today’s stuff, but it would’ve been solid for ~2013, and they had all the pieces.
VP9’s decoder / encoder was pretty slow at then with most Android devices. WebP has similar perf characteristics to jpeg (more or less at that point in time).
More plausible reason than I’ve heard before at least!
[Comment removed by author]
I can’t really predict how it will play out, but I suspect JXL could carve out a durable niche on the Web thanks to the progressive loading and transcoding features (see something very soon after loading a page + migration path for existing .jpgs) and a nice progressive lossless mode (greatly improved FLIF).
AVIF seems on track to get broad hardware support; I could imagine your phone camera saving in it because of that, for example. It is also better at maintaining presentable results when a file is very heavily compressed; it tends to just lose low-contrast detail instead of adding attention-getting DCT artifacts or blurring sharp edges. (Maybe that makes heavily compressed AVIFs of very high-res (4K+) images another niche? Not sure.) AVIF’s use of spatial prediction, which JXL does not have, tends to make progressive decoding harder but handling sharp edges easier.
Hope there’s good adoption for both (FWIW: JXL format Chrome bug, JXL transcoding Content-Encoding bug, post on AVIF adoption), patent trolls fail to effectively interfere, and practical progress continues in encode/decode speed and hardware support.
Why is Better Portable Graphics (BPG) by Fabrice Bellard not on the list of formats that are being compared?
HEIC uses the same set of compression tools (HEVC/H.265’s keyframe compression) but with a different container format–both the compression pluses/minuses and the IP situation should be pretty similar.
That makes sense, thanks.
Few years ago Microsoft suggested JPEG XR. It uses integer compression algorithm hence decompressing and compressing doesn’t have Quantization errors.
I wonder why it didn’t get wide support (Microsoft granted a free use of patents).