Repository navigation
Blit optimisations - #1270
Open
kierank wants to merge 30 commits into
Open
Blit optimisations#1270kierank wants to merge 30 commits into
kierank wants to merge 30 commits into
Conversation
When the input picture is not writable (always the case with dup_uref_before_blit), the pipe did a full-frame ubuf_pic_copy() and then re-wrote the subpicture rectangles a second time with uref_pic_blit(), with the blend reading back the pixels it had just copied. Add _bg variants of the ubuf_pic_blit primitives taking a separate background ubuf, so the blend reads the underlying pixels from the original frame and writes every pixel of the destination rectangle exactly once (dest = blend(bg, src)). With bg == NULL the kernels behave exactly as before and the old names remain as wrappers. In _upipe_blit_prepare(), allocate the output picture, copy only the four bands around the bottom-most subpicture rectangle and compose the rectangle in one fused pass; higher z-index subpictures still blit in place into the now-writable picture. Any failure falls back to the previous full-copy-then-blit path.
Take run count on the command line and repeatedly call ubuf_pic_blit for yuv422p and yuv422p10le pictures. Currently only tests threshold copy using alpha plane.
Reworked on top of the background blit: the _bg functions carry the implementation, so those are what moves to ubuf_pic.c. The in-place entry points (ubuf_pic_blit, ubuf_pic_blit_alpha and ubuf_pic_blit_alpha10) stay in the header as the one-line wrappers they already were.
Squashed from the SIMD half of the original series: the 10-bit and 8-bit intrinsic kernels, their replacement by external asm, and the two commits that removed the intrinsic versions again. Only the asm survived that series, so only the asm is carried over here. The kernels compute dst = alpha > threshold ? src : bg. The background comes from its own pointer so that a freshly allocated destination can be composed in a single pass; callers blending in place pass bg == dst, which makes writing the background samples a no-op. For the same reason the scalar tails now store every sample instead of skipping the ones below the threshold. blit8_threshold_hsub2_avx2 was missing the XOR of the threshold with 0x80 that the other byte kernels do: the packed signed compare then tested alpha - 128 against the raw threshold, so with a threshold below 128 the source pixel was only selected for an alpha above threshold + 128. The unit test only exercised alpha 0 and 255, which land on the right side of the comparison either way.
…ound The existing checks sample a 3x3 grid of a 64x64 picture, which leaves most of a line unverified and never exercises the background form of the blit at all. Blit a whole single-plane picture instead and compare every sample against the scalar reference, once composed out of a separate background and once blended in place. The widths run either side of the point where the SIMD kernels take over and are mostly not whole vectors, so the vector loops and their scalar tails are both covered, for hsub 1 and 2 at 8 and 10 bits. The source, alpha and background planes carry a pattern in which neighbouring samples differ, which is what catches a kernel that loses the order of the samples in a vector or mis-compacts a subsampled alpha plane.
The sub dropped its stored ubuf whenever a flow definition was set on it. That was harmless while every subpicture of a flow had the same geometry, because the output helper deduplicates identical flow definitions and none reached the sub. With the size and padding attributes changing per subpicture a flow definition arrives with every one of them, so the picture being blitted onto each frame was freed as the next one was negotiated and there was nothing left to blit. Keep the geometry the ubuf was accepted under alongside it, and blit with that. The geometry read from the flow definition is now only what the next input is validated against, so the two no longer have to describe the same subpicture. A change of the main input's flow format does still invalidate the subpictures, since they were scaled and placed for the previous format, so drop them where that is handled instead.
The alpha levels are used rather than keyed on when the threshold is the maximum, and that path had no SIMD, so a burnt-in subtitle spent its time in a per sample multiply, add and divide by 0x3ff, the last of which the compiler turns into a 64 bit magic multiply. The numerator is a dot product of (bg, src) with (0x3ff - a, a), which is one pmaddwd on interleaved pairs; the inputs are all below 0x400 so the 32 bit accumulators cannot overflow and the results pack back to words without saturating. The divide has an exact truncating shift form over the range the numerator can take, so the kernels stay bit exact with the C.
The whole-plane test in ubuf_pic_blit_test already checks them against the reference, but through ubuf_pic_blit, so it cannot bench them or say which instruction set it just exercised. Call the kernels directly instead, over a width that is not a multiple of any vector so the scalar tail runs, with a destination prefilled with a value the blend cannot produce and a guard past the end of the line.
Leaves AVX2 as the only vector path; anything older falls back to C.
blit8_blend_hsub1/2 had no test. The same shape as the 10 bit one: an odd length so that the scalar tail runs, a guard past the end, and the extremes of alpha and of the samples.
With the weights (a, 0xff - a) unsigned and the samples flipped to signed, one pmaddubsw gives N - 0x7f80 for each pair of samples, and adding 0x7f81 and a pmulhuw by 0x101 divides by 0xff exactly (checked over every (a, src, bg)). 14 vector ops per 32 samples instead of 23.
Only the blend kernels had one. The threshold kernels are checked at a multiple of every vector width, with a short tail, with a tail long enough to need both alpha vectors of a subsampled kernel, and at the smallest width the dispatcher passes on, with guard samples after the line.
The mask registers remove most of the work the avx2 kernels do around the compare: vpcmpub and vpcmpuw compare unsigned, so the +0x80 bias disappears, and a merge-masked load selects the source samples in place of pand/pandn/por. The subsampled kernels no longer need the two pshufb, the punpcklqdq and the vpermq that undid avx2's lane split. 10-bit compresses the alpha with a single vpermt2w taking the second vector straight from memory, 8-bit with vpmovwb and vinserti64x4, both of which cross lanes on their own. The trailing samples are predicated rather than handled by a scalar loop. Masked loads suppress faults on masked-out samples, so the tail never reads past the end of the row and the kernels accept any hsize; the callers still gate on the avx2 minimum for now.
When the subpicture is blitted in place (bg == dst), the selected source samples are stored under the compare mask and the background is never loaded; a vector with no selected sample is not written at all. These kernels are dispatched when bg == dst. Measured on Zen 4 (EPYC 8324P) and Raptor Cove at a 1920x256 strip with text alpha, they are 29-37% faster than avx2 at every depth and subsampling. Also measured and not kept: the in place version without the skip, an avx2 variant selecting with one saturating add or subtract into vpblendvb (up to 16% slower on Raptor Cove at 517 samples, parity at frame size), and an avx512icl vpermt2b alpha gather for 8 bit hsub2 (3-5% at 517 samples, nothing at frame size). The fused avx512 hsub1 kernels are 7-10% slower than avx2 at frame size on Zen 4. Two candidates, checkasm-only, separate the two suspects: _blendm selects with a register blend after plain loads, _ymm is the current kernel at 256 bits. The threshold checks now cover both sides of 128 and the extremes, and a frame sized benchmark (1920 x 256, text-like alpha, aligned planes) runs every kernel at a threshold below and above the middle.
On Zen 4 with planes that overflow the cache (1920x256, text alpha) the zmm fused hsub1 kernels lose 8% to avx2, with a masked load or with a register blend after plain loads alike; at 256 bits they are level with avx2 (8 bit +0.7%, 10 bit -1.2%) and still 1.3-1.7x faster from L1. The hsub2 and in place kernels stay at 512 bits. The candidates go.
…vx512 blend in place In place (bg == dst), a vector with nothing to change need not be loaded or stored, which leaves the cache lines of transparent areas clean; that skip was most of the avx512 threshold kernels' gain. - avx2 threshold in place: a vector with no sample over the threshold is skipped. Dispatched when bg == dst. - avx2 blend in place: a vector whose alpha is all 0 is skipped, one whose alpha is all at the maximum takes the source without the blend (the same samples). Dispatched when bg == dst. DVB and TTML subtitles blend. - avx512 blend in place, the same at 512 bits, checkasm-only until benchmarked against the avx2 one. checkasm checks the blend in place at random and at subtitle-like alpha (transparent, opaque and glyph edge blocks), and benchmarks the blend fused and in place at frame size with that alpha. BLEND8 takes the background base as an argument.
The loop runs once before its test, which the avx2 kernels rely on the dispatcher's minimum for; at 512 bits that minimum is half a vector, and the first vector wrote past the row. Such a row goes to the scalar tail. Found by checkasm at the minimum widths.
Measured at frame size (1920x256, subtitle and text alpha) against the plain kernels: - Zen 4: the avx512 blend in place is 0.57-0.85 of the avx2 blend, and 0.75-0.89 of the avx2 blend in place, at every depth and subsampling. Dispatched for bg == dst on avx512bw. - Raptor Cove: the avx2 in place kernels win at hsub1 (threshold 1.47x/1.26x, blend 1.12x/1.18x) and lose at hsub2 (0.91-0.995x). Dispatched at hsub1 only; the hsub2 ones go.
The 10 bit hsub1 kernel at 256 bits built its tail mask with kmovw from r7w, which nasm 3 accepts but nasm 2.14 rejects: kmovw takes a 32 bit register.
__builtin_cpu_supports ran for every row. Read the avx512 features once in a constructor instead. gcc before 8 has no avx512 feature names for it, so the avx512 kernels are not used with those compilers.
As in FFmpeg's x86inc, avx512icl implies avx512.
Per half, vpdpwssd into an accumulator of 1 gives M = N + 1, and (M * 1025) >> 20 equals (M + (M >> 10)) >> 10, the BLEND10 result (N + (N >> 10) + 1) >> 10, for every N from 0 to 0x3ff * 0x3ff (checked exhaustively; M * 1025 stays below 2^31): 12 ops a vector instead of 16. hsub2 gathers alpha with one vpermt2w instead of pshufb x2, punpcklqdq and vpermq. ymm: on Zen 4 a zmm op takes two of the four 256 bit pipes. Dispatched for hsub1 when dst != bg (in place keeps the avx512 kernel), and for hsub2 when the in place avx512 kernel is not taken. checkasm 158/158 (ubuf_pic_blit 52/52 on seeds 1, 2, 3, 77, 9999). EPYC 8324P (Zen 4c), checkasm --bench min of 5, cycles: blit10_blend_hsub1 190.8 -> 184.2 (-3.5%) blit10_blend_hsub1_frame 179341 -> 168332 (-6.1%) blit10_blend_hsub2 276.7 -> 205.2 (-25.8%) blit10_blend_hsub2_frame 253926 -> 188260 (-25.9%)
One vpermt2b (AVX512-VBMI) takes the even alpha bytes of two registers instead of pshufb x2, punpcklqdq and vpermq: 17 ops a vector instead of 20. The zmm blit8_threshold_hsub2 and blit8_blend_inplace_hsub2 keep their vpmovwb gathers: they are built for plain avx512, without VBMI, and an ICL variant would save about one op. checkasm 160/160 (ubuf_pic_blit 54/54 on seeds 1, 2, 3, 77, 9999). EPYC 8324P (Zen 4c), checkasm --bench min of 5, cycles: blit8_blend_hsub2 126.7 -> 108.6 (-14%) blit8_blend_hsub2_frame 109544 -> 94141 (-14%)
The blit dispatch was compiled in for any x86-64 gcc build, but the kernels are only assembled when nasm is available, so a build without it failed to link. Test HAVE_X86ASM as well. The kernels use x86-64 registers, so assemble them, and link them into checkasm, on x86-64 only. ubuf_pic_plane_set_color's SSE2 path got <emmintrin.h> through the blit header, so include it directly.
The x86 kernels, their dispatch header and the checkasm test had no SPDX-License-Identifier (the checkasm test carried a GPL notice). Use the MIT licence of the rest of the blit code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.