Skip to content

Blit optimisations - #1270

Open
kierank wants to merge 30 commits into
Upipe:masterfrom
kierank:blit-1.5
Open

kierank wants to merge 30 commits into
Upipe:masterfrom
kierank:blit-1.5

Conversation

@kierank

@kierank kierank commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

No description provided.

kierank and others added 30 commits November 5, 2026 12:12
When the input picture is not writable (always the case with
dup_uref_before_blit), the pipe did a full-frame ubuf_pic_copy() and
then re-wrote the subpicture rectangles a second time with
uref_pic_blit(), with the blend reading back the pixels it had just
copied.

Add _bg variants of the ubuf_pic_blit primitives taking a separate
background ubuf, so the blend reads the underlying pixels from the
original frame and writes every pixel of the destination rectangle
exactly once (dest = blend(bg, src)). With bg == NULL the kernels
behave exactly as before and the old names remain as wrappers.

In _upipe_blit_prepare(), allocate the output picture, copy only the
four bands around the bottom-most subpicture rectangle and compose the
rectangle in one fused pass; higher z-index subpictures still blit in
place into the now-writable picture. Any failure falls back to the
previous full-copy-then-blit path.
Take run count on the command line and repeatedly call ubuf_pic_blit for
yuv422p and yuv422p10le pictures.  Currently only tests threshold copy
using alpha plane.
Reworked on top of the background blit: the _bg functions carry the
implementation, so those are what moves to ubuf_pic.c.  The in-place
entry points (ubuf_pic_blit, ubuf_pic_blit_alpha and
ubuf_pic_blit_alpha10) stay in the header as the one-line wrappers they
already were.
Squashed from the SIMD half of the original series: the 10-bit and 8-bit intrinsic
kernels, their replacement by external asm, and the two commits that
removed the intrinsic versions again.  Only the asm survived that
series, so only the asm is carried over here.

The kernels compute dst = alpha > threshold ? src : bg.  The background
comes from its own pointer so that a freshly allocated destination can
be composed in a single pass; callers blending in place pass bg == dst,
which makes writing the background samples a no-op.  For the same reason
the scalar tails now store every sample instead of skipping the ones
below the threshold.

blit8_threshold_hsub2_avx2 was missing the XOR of the threshold with
0x80 that the other byte kernels do: the packed signed compare then
tested alpha - 128 against the raw threshold, so with a threshold below
128 the source pixel was only selected for an alpha above
threshold + 128.  The unit test only exercised alpha 0 and 255, which
land on the right side of the comparison either way.
…ound

The existing checks sample a 3x3 grid of a 64x64 picture, which leaves
most of a line unverified and never exercises the background form of the
blit at all.  Blit a whole single-plane picture instead and compare every
sample against the scalar reference, once composed out of a separate
background and once blended in place.

The widths run either side of the point where the SIMD kernels take over
and are mostly not whole vectors, so the vector loops and their scalar
tails are both covered, for hsub 1 and 2 at 8 and 10 bits.  The source,
alpha and background planes carry a pattern in which neighbouring
samples differ, which is what catches a kernel that loses the order of
the samples in a vector or mis-compacts a subsampled alpha plane.
The sub dropped its stored ubuf whenever a flow definition was set on it.  That
was harmless while every subpicture of a flow had the same geometry, because the
output helper deduplicates identical flow definitions and none reached the sub.
With the size and padding attributes changing per subpicture a flow definition
arrives with every one of them, so the picture being blitted onto each frame was
freed as the next one was negotiated and there was nothing left to blit.

Keep the geometry the ubuf was accepted under alongside it, and blit with that.
The geometry read from the flow definition is now only what the next input is
validated against, so the two no longer have to describe the same subpicture.

A change of the main input's flow format does still invalidate the subpictures,
since they were scaled and placed for the previous format, so drop them where
that is handled instead.
The alpha levels are used rather than keyed on when the threshold is the
maximum, and that path had no SIMD, so a burnt-in subtitle spent its time in a
per sample multiply, add and divide by 0x3ff, the last of which the compiler
turns into a 64 bit magic multiply.

The numerator is a dot product of (bg, src) with (0x3ff - a, a), which is one
pmaddwd on interleaved pairs; the inputs are all below 0x400 so the 32 bit
accumulators cannot overflow and the results pack back to words without
saturating.  The divide has an exact truncating shift form over the range the
numerator can take, so the kernels stay bit exact with the C.
The whole-plane test in ubuf_pic_blit_test already checks them against the
reference, but through ubuf_pic_blit, so it cannot bench them or say which
instruction set it just exercised.  Call the kernels directly instead, over a
width that is not a multiple of any vector so the scalar tail runs, with a
destination prefilled with a value the blend cannot produce and a guard past
the end of the line.
Leaves AVX2 as the only vector path; anything older falls back to C.
blit8_blend_hsub1/2 had no test.  The same shape as the 10 bit one: an
odd length so that the scalar tail runs, a guard past the end, and the
extremes of alpha and of the samples.
With the weights (a, 0xff - a) unsigned and the samples flipped to
signed, one pmaddubsw gives N - 0x7f80 for each pair of samples, and
adding 0x7f81 and a pmulhuw by 0x101 divides by 0xff exactly (checked over
every (a, src, bg)).  14 vector ops per 32 samples instead of 23.
Only the blend kernels had one.  The threshold kernels are checked at a
multiple of every vector width, with a short tail, with a tail long
enough to need both alpha vectors of a subsampled kernel, and at the
smallest width the dispatcher passes on, with guard samples after the
line.
The mask registers remove most of the work the avx2 kernels do around the
compare: vpcmpub and vpcmpuw compare unsigned, so the +0x80 bias disappears,
and a merge-masked load selects the source samples in place of pand/pandn/por.

The subsampled kernels no longer need the two pshufb, the punpcklqdq and the
vpermq that undid avx2's lane split.  10-bit compresses the alpha with a single
vpermt2w taking the second vector straight from memory, 8-bit with vpmovwb and
vinserti64x4, both of which cross lanes on their own.

The trailing samples are predicated rather than handled by a scalar loop.
Masked loads suppress faults on masked-out samples, so the tail never reads
past the end of the row and the kernels accept any hsize; the callers still
gate on the avx2 minimum for now.
When the subpicture is blitted in place (bg == dst), the selected source
samples are stored under the compare mask and the background is never
loaded; a vector with no selected sample is not written at all.  These
kernels are dispatched when bg == dst.

Measured on Zen 4 (EPYC 8324P) and Raptor Cove at a 1920x256 strip with
text alpha, they are 29-37% faster than avx2 at every depth and
subsampling.  Also measured and not kept: the in place version without
the skip, an avx2 variant selecting with one saturating add or subtract
into vpblendvb (up to 16% slower on Raptor Cove at 517 samples, parity
at frame size), and an avx512icl vpermt2b alpha gather for 8 bit hsub2
(3-5% at 517 samples, nothing at frame size).

The fused avx512 hsub1 kernels are 7-10% slower than avx2 at frame size
on Zen 4.  Two candidates, checkasm-only, separate the two suspects:
_blendm selects with a register blend after plain loads, _ymm is the
current kernel at 256 bits.

The threshold checks now cover both sides of 128 and the extremes, and a
frame sized benchmark (1920 x 256, text-like alpha, aligned planes) runs
every kernel at a threshold below and above the middle.
On Zen 4 with planes that overflow the cache (1920x256, text alpha) the
zmm fused hsub1 kernels lose 8% to avx2, with a masked load or with a
register blend after plain loads alike; at 256 bits they are level with
avx2 (8 bit +0.7%, 10 bit -1.2%) and still 1.3-1.7x faster from L1.  The
hsub2 and in place kernels stay at 512 bits.  The candidates go.
…vx512 blend in place

In place (bg == dst), a vector with nothing to change need not be
loaded or stored, which leaves the cache lines of transparent areas
clean; that skip was most of the avx512 threshold kernels' gain.

- avx2 threshold in place: a vector with no sample over the threshold
  is skipped.  Dispatched when bg == dst.
- avx2 blend in place: a vector whose alpha is all 0 is skipped, one
  whose alpha is all at the maximum takes the source without the blend
  (the same samples).  Dispatched when bg == dst.  DVB and TTML
  subtitles blend.
- avx512 blend in place, the same at 512 bits, checkasm-only until
  benchmarked against the avx2 one.

checkasm checks the blend in place at random and at subtitle-like
alpha (transparent, opaque and glyph edge blocks), and benchmarks the
blend fused and in place at frame size with that alpha.  BLEND8 takes
the background base as an argument.
The loop runs once before its test, which the avx2 kernels rely on the
dispatcher's minimum for; at 512 bits that minimum is half a vector, and
the first vector wrote past the row.  Such a row goes to the scalar
tail.  Found by checkasm at the minimum widths.
Measured at frame size (1920x256, subtitle and text alpha) against the
plain kernels:

- Zen 4: the avx512 blend in place is 0.57-0.85 of the avx2 blend, and
  0.75-0.89 of the avx2 blend in place, at every depth and subsampling.
  Dispatched for bg == dst on avx512bw.
- Raptor Cove: the avx2 in place kernels win at hsub1 (threshold
  1.47x/1.26x, blend 1.12x/1.18x) and lose at hsub2 (0.91-0.995x).
  Dispatched at hsub1 only; the hsub2 ones go.
The 10 bit hsub1 kernel at 256 bits built its tail mask with kmovw from
r7w, which nasm 3 accepts but nasm 2.14 rejects: kmovw takes a 32 bit
register.
__builtin_cpu_supports ran for every row.  Read the avx512 features once
in a constructor instead.  gcc before 8 has no avx512 feature names for
it, so the avx512 kernels are not used with those compilers.
As in FFmpeg's x86inc, avx512icl implies avx512.
Per half, vpdpwssd into an accumulator of 1 gives M = N + 1, and
(M * 1025) >> 20 equals (M + (M >> 10)) >> 10, the BLEND10 result
(N + (N >> 10) + 1) >> 10, for every N from 0 to 0x3ff * 0x3ff (checked
exhaustively; M * 1025 stays below 2^31): 12 ops a vector instead of 16.
hsub2 gathers alpha with one vpermt2w instead of pshufb x2, punpcklqdq and
vpermq. ymm: on Zen 4 a zmm op takes two of the four 256 bit pipes.
Dispatched for hsub1 when dst != bg (in place keeps the avx512 kernel), and
for hsub2 when the in place avx512 kernel is not taken.

checkasm 158/158 (ubuf_pic_blit 52/52 on seeds 1, 2, 3, 77, 9999).
EPYC 8324P (Zen 4c), checkasm --bench min of 5, cycles:
  blit10_blend_hsub1        190.8 -> 184.2   (-3.5%)
  blit10_blend_hsub1_frame  179341 -> 168332 (-6.1%)
  blit10_blend_hsub2        276.7 -> 205.2   (-25.8%)
  blit10_blend_hsub2_frame  253926 -> 188260 (-25.9%)
One vpermt2b (AVX512-VBMI) takes the even alpha bytes of two registers
instead of pshufb x2, punpcklqdq and vpermq: 17 ops a vector instead of 20.
The zmm blit8_threshold_hsub2 and blit8_blend_inplace_hsub2 keep their
vpmovwb gathers: they are built for plain avx512, without VBMI, and an ICL
variant would save about one op.

checkasm 160/160 (ubuf_pic_blit 54/54 on seeds 1, 2, 3, 77, 9999).
EPYC 8324P (Zen 4c), checkasm --bench min of 5, cycles:
  blit8_blend_hsub2        126.7 -> 108.6   (-14%)
  blit8_blend_hsub2_frame  109544 -> 94141  (-14%)
The blit dispatch was compiled in for any x86-64 gcc build, but the
kernels are only assembled when nasm is available, so a build without
it failed to link.  Test HAVE_X86ASM as well.  The kernels use x86-64
registers, so assemble them, and link them into checkasm, on x86-64
only.

ubuf_pic_plane_set_color's SSE2 path got <emmintrin.h> through the blit
header, so include it directly.
The x86 kernels, their dispatch header and the checkasm test had no
SPDX-License-Identifier (the checkasm test carried a GPL notice).  Use
the MIT licence of the rest of the blit code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants