March 28, 2023
  1. Video preprocessing, which encompasses any transformation utilized to the high-quality supply video previous to encoding. Video downscaling is probably the most pertinent instance herein, which tailors our encoding to display resolutions of various units and optimizes image high quality underneath various community circumstances. With video downscaling, a number of resolutions of a supply video are produced. For instance, a 4K supply video will probably be downscaled to 1080p, 720p, 540p and so forth. That is sometimes completed by a standard resampling filter, like Lanczos.
  2. Video encoding utilizing a standard video codec, like AV1. Encoding drastically reduces the quantity of video knowledge that must be streamed to your gadget, by leveraging spatial and temporal redundancies that exist in a video.
  • A discovered method for downscaling can enhance video high quality and be tailor-made to Netflix content material.
  • It may be built-in as a drop-in resolution, i.e., we don’t want every other adjustments on the Netflix encoding facet or the shopper gadget facet. Tens of millions of units that assist Netflix streaming mechanically profit from this resolution.
  • A definite, NN-based, video processing block can evolve independently, be used past video downscaling and be mixed with completely different codecs.
Structure of the deep downscaler mannequin, consisting of a preprocessing block adopted by a resizing block.
Left: Lanczos downscaling; proper: deep downscaler. Each movies are encoded with VP9 on the similar bitrate and have been upscaled to FHD decision (1920×1080). You might must zoom in to see the visible distinction.
  • The neural community structure was designed to be computationally environment friendly and in addition keep away from any detrimental visible high quality affect. For instance, we discovered that just some neural community layers have been enough for our wants. To scale back the enter channels even additional, we solely apply NN-based scaling on luma and scale chroma with a regular Lanczos filter.
  • We applied the deep downscaler as an FFmpeg-based filter that runs along with different video transformations, like pixel format conversions. Our filter can run on each CPU and GPU. On a CPU, we leveraged oneDnn to additional cut back latency.
A top-down view of integrating the deep downscaler into Cosmos.