Image codecs

Meep carries its own image codecs rather than relying on the browser for everything. The centrepiece is src/format/image/avif/**: a from-scratch AV1 intra encoder and decoder written in plain JavaScript. There is no WebAssembly, no worker requirement and nothing asynchronous about it, so the same code runs in a browser, in a worker and in node. Beside it, src/format/image/png/ and src/format/image/jpeg/ hold the PNG and JPEG decoders, and src/engine/asset/loaders/image/codec/ puts all three behind the Codec interface the asset loaders speak.

An .avif asset loads through ImageRGBADataLoader like a PNG does - guessAssetType maps the extension to "image" - and the codec is also reachable by direct import for anything the asset path does not cover, HDR decoding above all.

The AVIF codec

import {
    decode_avif,
    encode_avif
} from "@woosh/meep-engine/src/format/image/avif/index.js";

import { avif_to_sampler2d }
    from "@woosh/meep-engine/src/engine/graphics/texture/sampler/avif_to_sampler2d.js";
import { sampler2d_to_avif }
    from "@woosh/meep-engine/src/engine/graphics/texture/sampler/sampler2d_to_avif.js";

Every symbol below is a named export. src/format/image/avif/index.js is the codec’s door - the decoder and encoder, their reusable contexts, the constants, the colour conversion and the container-level parse and write. It imports nothing from the engine. The two Sampler2D adapters live beside the type they serve, in src/engine/graphics/texture/sampler/, and the worker pool that encodes off the main thread is under src/engine/asset/loaders/image/avif/.

Decoding and encoding

All four of the top functions are synchronous. await on the result of decode_avif costs nothing but does nothing either; a caller that must not block the thread it is on should run the decode in a worker it owns, which is exactly what the encode pool below does for encoding.

import { decode_avif, encode_avif } from "@woosh/meep-engine/src/format/image/avif/index.js";

const { data, width, height } = decode_avif(bytes);      // interleaved RGBA
const file = encode_avif({ data, width, height }, { quality: 80 });   // Uint8Array

encode_avif defaults, read from the source rather than inferred: quality 75, bit_depth 8, subsampling SUBSAMPLING_420, color_primaries BT.709, transfer_characteristics sRGB. effort and decode_effort are left to whatever the encode context already carries, which for a fresh one is EFFORT_BALANCED and DECODE_FAST.

Full option set for encode_avif:

decode_avif takes everything convert_to_rgba takes - bit_depth, output_float16, output_colour_space, linear_reference_nits, chroma_upsampling - plus apply_transformations, which honours the orientation the file declares. That one is off by default, so a caller that already handles orientation does not apply it twice.

Reusing the scratch space

AvifDecoder and Av1EncodeContext are the working memory a decode or an encode needs: plane stores, frame contexts, the colour transform. Both are exported, and both are accepted as the last argument of the functions above. Passing the same one back in is how a batch of images avoids reallocating everything per image.

import { AvifDecoder, decode_avif } from "@woosh/meep-engine/src/format/image/avif/index.js";

const decoder = new AvifDecoder();

for (const bytes of files) {
    const image = decode_avif(bytes, {}, decoder);
    // ...
}

AvifDecoder gives colour and alpha a frame context each on purpose: sharing one would mean decoding the alpha plane overwrote the picture, and every caller would have to copy the picture out first - the exact allocation the reuse is there to avoid.

The constants

Chroma layout, passed as subsampling:

Encode effort - how long the encoder spends deciding. The figures are the codec’s own, measured against libavif at speed 6 over eight pictures:

Decode effort - which in-loop filters the written file asks the reader to run. This is a separate axis from effort and an unrelated one: encode effort spends your time, decode effort spends the reader’s.

DECODE_FAST is the default because a game texture is decoded on the machine that draws with it, once per load, and a file five per cent larger costs a few kilobytes of download that a cache makes free after the first time. Reverse it when bytes are the scarce thing rather than CPU.

Two more sets, both from convert_to_rgba: CHROMA_UPSAMPLING_{AUTOMATIC, BEST, FASTEST} (BEST is bilinear, FASTEST is nearest neighbour) and COLOUR_SPACE_{SIGNAL, LINEAR} - SIGNAL hands back the samples as coded, LINEAR undoes the transfer function.

The HDR path

avif_to_sampler2d is the bridge from a file to something the renderer can upload, and it is where the high dynamic range decision lives:

import { avif_to_sampler2d }
    from "@woosh/meep-engine/src/engine/graphics/texture/sampler/avif_to_sampler2d.js";

const sdr = avif_to_sampler2d(bytes);                 // 8-bit RGBA Sampler2D
const hdr = avif_to_sampler2d(bytes, { hdr: true });  // half-float linear Sampler2D

With hdr: true the decode runs at 12 bits into half floats with output_colour_space: COLOUR_SPACE_LINEAR, so the sampler holds linear light, normalised the way every other sampler in the engine is: 1.0 is SDR reference white, 203 nits per ITU-R BT.2408. Either way the result is a 4-channel Sampler2D at the file’s dimensions, and chroma upsampling is forced to CHROMA_UPSAMPLING_BEST.

sampler2d_to_avif is the exact inverse, and it decides from the data rather than from a flag: a Uint16Array-backed sampler is taken as those same linear half floats and coded through the PQ transfer function at 12 bits with SUBSAMPLING_422; anything else is coded as ordinary sRGB at SUBSAMPLING_420. Its quality is 0 .. 1 (unlike encode_avif’s 0 .. 100), defaulting to 0.75, and its options bag accepts only subsampling and alpha.

Note the asymmetry in the default reference white: the codec’s own LINEAR_REFERENCE_NITS is 100 nits, but this pair overrides it with the renderer’s 203 in both directions. That override is what makes the round trip exact.

Shade uses this pair itself: scene serialization stores image sources as AVIF, decodes HDR ones through avif_to_sampler2d and prefers the browser’s hardware decoder for SDR ones, falling back to this codec where createImageBitmap does not exist.

What is inside

Worth knowing when a stack trace lands in the middle of it:

The decoder reads what libavif writes sample for sample, and the encoder writes what libavif reads, sample for sample. Both claims are checked by the codec’s own tooling rather than asserted.

Encoding off the main thread

Encoding is much more expensive than decoding, so there is a pool for it.

import { get_threaded_image_encoder }
    from "@woosh/meep-engine/src/engine/asset/loaders/image/avif/threaded_image_encoder.js";

const encoder = get_threaded_image_encoder();                        // process singleton

const bytes = await encoder.encode_bitmap(bitmap, 0.8);              // Promise<ArrayBuffer>
const more  = await encoder.encode_sampler2d(sampler, 0.8, 4);       // Promise<ArrayBuffer>

encoder.shutdown();

ThreadedImageEncoder is also exported from the same module if you want your own pool: new ThreadedImageEncoder({ worker_count, idle_timeout_ms }). worker_count defaults to navigator.hardwareConcurrency - 1 with a floor of 1, and idle_timeout_ms to 5000.

Four behaviours to know:

• Workers spawn lazily. Nothing starts until the first encode is submitted.
• They self-terminate after 5 s idle, and start again on the next request. shutdown() ends them immediately.
• The inputs are consumed. An ImageBitmap is closed and a sampler’s buffer is transferred to the worker. Do not touch either afterwards.
• Absence of workers is not an error. Where no Worker can be had - node, a worker that cannot nest one, a browser that refuses a module worker - the pool decides that once, remembers it, and runs the identical encode inline on the calling thread. Nothing throws for want of a Worker, and nothing tells you it happened. The one thing that genuinely can fail is encode_bitmap without OffscreenCanvas, because reading a bitmap’s samples back needs a canvas to draw it on.

speed is a 0 (slowest, best) to 10 (fastest) dial, mapped onto the three EFFORT_* points by effort_of_speed (src/engine/asset/loaders/image/avif/encode_image_source.js): <= 2 is EFFORT_THOROUGH, <= 7 is EFFORT_BALANCED, above that is EFFORT_FAST, and omitting it is EFFORT_BALANCED.

Note that the pool hands back ArrayBuffer, while encode_avif and sampler2d_to_avif hand back Uint8Array.

The asset-path codecs

src/engine/asset/loaders/image/codec/ holds the codecs the asset pipeline goes through. The interface sits one level up, in src/engine/asset/codec/, and is small:

import { Codec } from "@woosh/meep-engine/src/engine/asset/codec/Codec.js";

Codec is abstract with two methods - async decode(data: Uint8Array) and async test(data): Promise<boolean>, the latter defaulting to true. Every image codec returns the same record, DecodedImage (codec/DecodedImage.js): { data, width, height, itemSize, bitDepth }, where data is the interleaved samples as an ArrayBuffer or a typed view of one, itemSize is channels per texel and bitDepth is bits per sample - 8 and below is one byte per sample, 16 is Uint16 in platform byte order.

ImageRGBADataLoader - the loader registered for GameAssetType.Image ("image") by TerrainSystem, DecalSystem and ParticleEmitterSystem - composes exactly those, in this order:

this.decoder = new CodecWithFallback(
    new ThreadedImageDecoder({ worker_path }),
    new NativeImageDecoder(),
    new JpegCodec(),
    new AvifCodec()
);

That composition is the whole practical story of image loading:

PNG decodes off the main thread and keeps its bit depth. PNGReader normalises 16-bit samples to platform endianness, so ImageRGBADataLoader views the returned buffer as a Uint16Array directly - which is what makes 16-bit heightmaps work. A bit depth of 1, 2, 4 or 8 becomes a Uint8Array; anything else throws.

PNGReader.parse() is asynchronous: await it before reading the decoded samples. PNG inflation and zlib-compressed KTX2 levels use the platform’s DecompressionStream; direct callers of ktx2_read must also await its result.

Everything else goes to the browser at 8 bits where there is one. JPEG, WebP, GIF, AVIF and the rest reach NativeImageDecoder, which draws the image into a canvas and reads it back as 8-bit RGBA on the main thread. There is no precision above 8 bits on that path and no way to ask for one. The in-tree JPEG and AVIF decoders sit after it deliberately: the platform decoder is faster - the in-tree JPEG decoder measured fifteen to twenty times the latency of the browser’s on 1024² textures - so they answer only where the native codec throws: node, or a worker without Image.

The asset is an ImageRGBADataAsset; its create() returns a Sampler2D.

The decoder worker

The worker ships prebuilt at @woosh/meep-engine/build/bundle-worker-image-decoder.js (the package ships exactly two worker bundles, this one and bundle-worker-terrain.js). Its source is src/engine/asset/loaders/image/ImageDecoderWorker.js, which exposes self.Lib.decode(arrayBuffer, type = 'png') and handles only 'png' - any other type throws Unsupported type. JPEG is kept out of the worker on purpose: moving a decode that is an order of magnitude slower than the platform’s off the main thread saves about 1.5 ms of main-thread time per texture and costs the rest of the latency.

worker_path defaults to the package’s own bundle, resolved against the codec module with import.meta.url, so it needs no configuration under any bundler. Pass your own through new ImageRGBADataLoader({ worker_path }) when the bundle is served from somewhere else. A blob worker inherits the page’s COEP, so under Cross-Origin-Embedder-Policy: require-corp the bundle must be served with Cross-Origin-Resource-Policy, or its importScripts is blocked - and the loader then falls through to the main-thread decoder with nothing reported, so the output stays correct and only the speed changes.

Direct import

Two things are reachable only by direct import:

• jpeg_decode (src/format/image/jpeg/jpeg_decode.js, named export) (jpegData: ArrayBuffer, userOpts?) => { width, height, data: Uint8Array, exifBuffer: Uint8Array }
  • the decoder JpegCodec wraps, with the EXIF payload the codec drops.
• HDR AVIF. The asset path decodes an AVIF to its 8-bit SDR signal. The half-float linear reading is avif_to_sampler2d(buffer, { hdr: true }), which nothing registers with the AssetManager - fetch the bytes as GameAssetType.ArrayBuffer and call it yourself.

Related

• Asset pipeline - the AssetManager, loaders and the transformer chain
• Meshes & materials - ShadeImage and ShadeTexture, which is where a decoded Sampler2D usually ends up
• Sky & environment - environment maps, the other place HDR pixels enter