RFC: Lossless fixed-width integer bit-packing codec for ADC-style data

[sakshi2433]

Hi! This is an exploratory design discussion based on a working prototype and benchmarks.
I’d really appreciate feedback on API shape and integration direction.

Motivation

Many scientific datasets store integer signals where only a subset of bits are meaningful
(e.g. 10–12 bit ADC data stored in uint16). Zarr/numcodecs currently rely on byte-level
compression for such data, which does not explicitly remove unused bits.

Observations

In an ADC-style benchmark (uint16, effective 12 bits, 10M samples), default Zarr compression
reduced storage from ~19 MB to ~7 MB, but further gains plateaued. Existing bit-level tools
in numcodecs (e.g. PackBits) do not support lossless integer bit-width packing.

Proposal

Introduce a lossless integer bit-packing codec/filter that:

• Packs fixed-width integer values using exactly N bits
• Operates per chunk
• Is fully reversible
• Can be composed with existing compression

Prototype

I implemented a pure-Python prototype to validate feasibility:

• Correct round-trip verified
• Storage reduction proportional to effective bit-width
• Zarr v2 compatible (self-describing stream)

Results (summary)

• Bit-packing alone reduces storage predictably (e.g. 12/16 → ~0.75×)
• Bit-packing + Blosc/Zstd achieves size near default compression
• Python implementation is CPU-heavy → optimized backend likely needed

Questions

• Should this live as a codec or filter in numcodecs?
• How should bit-width metadata be handled (header vs external)?
• Is limiting initial scope to uint16 reasonable?
• Any guidance on aligning this with Zarr v3’s codec pipeline?

[LDeakin]

Any guidance on aligning this with Zarr v3’s codec pipeline?

There is a codec specification for a Zarr V3 packbits codec that can do this. https://github.com/zarr-developers/zarr-extensions/tree/main/codecs/packbits

[sakshi2433]

Thanks for the pointer — that’s very helpful.

I took a look at the Zarr v3 PackBits codec specification. As I understand it, that codec is primarily intended for packing bit arrays / boolean-like data, whereas the use case I’m exploring is lossless fixed-width packing of integer values (e.g. storing uint16 ADC data using exactly 10–12 bits per value).

One of the motivations for the RFC was that existing PackBits-style approaches don’t directly address integer bit-width reduction while preserving exact values. My prototype currently handles this by packing integer values bit-wise and carrying minimal metadata for reversibility.

Would the preferred direction be to:

• extend the existing v3 PackBits codec to explicitly support fixed-width integer packing, or
• treat integer bit-packing as a separate codec within the v3 pipeline?

I’d appreciate any guidance on how you’d envision this fitting into the v3 codec model.

[LDeakin]

Maybe I am misunderstanding, but doesn't the last_bit parameter of packbits support your use case?

{
  "data_type": "uint16",
  "codecs": [
    {
      "name": "packbits",
      "configuration": {
        "last_bit": 10
      }
    }
  ],
  ...
}

[sakshi2433]

Yes — at the specification level, I agree that packbits with first_bit / last_bit does express the use case I’m exploring (e.g. using last_bit to limit effective bit-width for uint16 data).

What motivated the RFC was less about the expressiveness of the spec itself, and more about the current practical gap between the v3 design and usable tooling today. In particular:

• the packbits codec is currently specified for v3, but not readily available to users via numcodecs / Zarr v2,
• there is no reference implementation or guidance on using first_bit / last_bit for integer data (as opposed to boolean/bit arrays),
• and there is limited evaluation around how this interacts with downstream compression and real datasets (e.g. ADC-style signals).

My prototype and benchmarks were intended as a way to explore these questions empirically and help inform how this capability should be implemented, exposed, and documented in practice.

If the preferred direction is to treat this entirely as an implementation and adoption question around the existing v3 packbits codec, I’m very happy to align with that and focus on prototyping, benchmarking, and integration guidance rather than proposing a new abstraction.