How I fitted CLIP’s full ViT-B-32 text encoder, a distilled image encoder and a 100,000-image catalog into a single Akamai Functions app, and the hurdles along the way.


Where I started

Miranda, by Alex Leung, is a fashion image search demo. It embeds a catalog of clothing photos with OpenCLIP ViT-B-32, embeds your text query with the same model, and ranks the images by cosine similarity. It runs as a Streamlit app with PyTorch on a server.

My first edge version of Miranda avoided running CLIP at query time. I distilled the text encoder into a lookup table with one vector per token, so turning a query into a vector took about 1 ms. Search quality on labelled queries was as good as the original. But the lookup table ignores word order, and its top 5 results matched full CLIP’s only about half the time.

A lookup table is not the model, though. I wanted a hard goal: run the full ViT-B-32 encoder entirely inside an Akamai Function. So I set myself these targets:

  • Run the real ViT-B-32 text encoder, all 12 transformer blocks, inside the Function.
  • Add search by photo: upload a picture and get similar items. That needs an image encoder, in the same app.
  • Keep the 100,000-image catalog in the same app, with no outbound calls, no GPU and no external vector database.
  • Match the quality of the original server setup as closely as possible.

That is all inside one Akamai Functions component, which has to fit in 52,428,800 bytes (50 MiB).

This post is the journey: what I measured, what broke, what I threw away, and the techniques that got both encoders into one Function.

The rules I set myself

  1. Measure against the original. Every version is scored against full-precision OpenCLIP over the same catalog: do the top 5 results match, and are the results relevant
  2. A golden check before every deploy. The Rust code must reproduce a Python reference within a stated tolerance, under plain wasmtime, before anything goes to Functions.
  3. Run all the heavy work offline. Training, embedding and exporting run on a Linode GPU instance. Only per-query work runs on Functions.

What I’m reusing from earlier posts

The foundations are the same as in my earlier posts. I use them here without explaining them again:

  • From Porting a 23M-Parameter BERT Model to Akamai Functions:
    • a hand-written forward pass instead of a model runtime
    • weights exported to a flat binary, compiled in with include_bytes! and read in place as slices, with no parsing per request
    • a golden reference to check the Rust output against
    • the +simd128 target flag and the release profile
    • the /... wildcard route and an empty allowed_outbound_hosts
  • From From 1 Second to 260 ms:
    • the int8 SIMD GEMM (i16x8_extend + i32x4_dot_i16x8, with a 4-row × 2-output register tile)
    • doing more work per loop iteration, because of epoch interruption
    • skipping unneeded rows in the last layer
    • per-stage Server-Timing headers and a /ping baseline
    • one change per deployment, measured as a median of repeated requests
  • From From Server Bottleneck to 100% Edge: compact binary formats read in place, instead of JSON.

Everything else in this post is new. The challenge here wasn’t one model. It was fitting three large pieces (a 63M-parameter transformer, an image encoder and a 100,000-image index) into an app that can barely hold one of them. That took a different set of techniques: very low-bit weights, rotations, distillation into a different architecture, index compression, and folding transforms into the weights. A summary table near the end lists every technique, what it saved, and whether it’s new.


The budget

Here’s what has to share 50 MiB:

PieceFull precisionProblem
ViT-B-32 text encoder (63M parameters)252 MB in fp32, 63 MB in int8Larger than the whole app even at int8
ViT-B-32 image encoder (about 88M parameters)350 MB in fp32Even bigger, and slow (more on that below)
100,000 catalog vectors (512 dims)205 MB in fp32Has to be compressed about 10× per image, to about 200 bytes
Code, tokenizer, web pageabout 1 MBFine

None of it fits as is. Each piece had to be compressed, and all the compression had to be paid for with training so the quality came back.


Part 1: The real text encoder, from 7 seconds to 24 ms

Before thinking about the image encoder, I needed to know whether the text encoder alone could run fast enough on Functions.

The first straightforward port loaded int8 weights, dequantised them to f32 and ran all 77 token positions, as CLIP does. It took 7.1 seconds per query on Functions.

Three changes fixed that:

Stepwasmtime Akamai Functions
77 tokens, int8 weights dequantised to f325,340 ms7,147 ms
+ run only the real tokens347 ms469–494 ms
+ int8 × int8 SIMD GEMM15.3 ms~24 ms
+ last block computes only the <eot> row12.8 ms24.0 ms p50

Run only the real tokens

This was the biggest win, and it costs nothing in accuracy. CLIP’s text transformer uses a causal attention mask: each token only looks at the tokens before it. The output is read at the <eot> (end-of-text) token. The padding after <eot> can never affect it.

So a query like “black winter coat” needs only <sot> black winter coat <eot>, 5 positions instead of 77, with an identical result (cosine 1.00000 against the 77-token version).

This is different from the padding trick in my BERT post. That model’s attention was bidirectional, so the padding was load-bearing and had to stay. CLIP’s causal mask means the padding can be removed entirely.

On top of that, I reused the last-layer trick from the same post: in the last block, only the <eot> row is needed after attention, so the MLP for the other rows is skipped:

for (bi, b) in m.blocks.iter().enumerate() {
    // In the last block only the <eot> row (the last one) is carried forward.
    let last = bi + 1 == m.blocks.len();

    layer_norm(&x, b.ln1, &mut h);
    let (xq, sx) = linear_in(&b.qkv, &h, l, ag);
    gemmq(&xq, &sx, l, ag, &b.qkv, &mut qkv);

    let (r0, rows) = if last { (l - 1, 1) } else { (0, l) };
    attention(&qkv, l, r0, &mut att);
    // ... out_proj, MLP and residuals run on `rows` rows only
}

The int8 SIMD GEMM

The matrix multiplies use the int8 SIMD kernel and 4 × 2 register tile described in From 1 Second to 260 ms, unchanged. That took encoding from about 480 ms to 24 ms.

A smaller tokenizer and token table

Two smaller size savings came from the tokenizer and the token embedding table.

The tokenizer stores only merge pairs. CLIP’s BPE vocabulary has a useful property: the token produced by merge rank r always has id 512 + r. So the token strings aren’t needed at all. The tokenizer file holds the byte ids and a sorted list of merge pairs, searched by binary search:

//! in CLIP's vocab the token made by merge rank r has id 512 + r, so only the
//! byte symbol ids and the merge pairs are needed (0.45 MB instead of 1.8 MB).
//!
//! vit_bpe.bin (little-endian):
//!   "BPE2" u32 n_merges | u16 byte_id[128] | u16 byte_id_w[128] (with "</w>")
//!   | u64 key[n_merges] sorted, key = (left_id << 32) | right_id | u16 rank[n_merges]
//! Merges whose result ASCII text can never produce are left out.

The token table keeps only reachable tokens. The token embedding table has 49,408 rows of 512 values, which is 101 MB in fp32. Queries are ASCII text, and 2,269 of those tokens can only come from non-ASCII bytes. The export keeps the 47,139 tokens that ASCII can produce and stores a u16 remap from token id to row:

keep = sorted(i for t, i in tk.encoder.items()
              if all(tk.byte_decoder[c] < 128 for c in t.replace("</w>", "")) or i in (49406, 49407))
remap = np.full(V, 0xFFFF, np.uint16); remap[keep] = np.arange(len(keep))

The rows themselves are quantised to 3 bits after a Hadamard rotation (explained in Part 4) and decoded one row per token at query time.

At this point the encoder ran in 24 ms on Functions, but the component was 51.9 MB with no catalog in it. The int8 encoder alone filled the app.


Part 2: 4 bits, and a catalog that fits beside it

To fit the catalog, the encoder had to shrink again, to 4-bit weights.

Hurdle: 4-bit without training doesn’t work

Rounding the trained weights to 4 bits (post-training quantisation) dropped top-5 agreement with full CLIP from about 0.95 to 0.50–0.67. That would make the results worse than the lookup table I was trying to replace.

Per-group activation scales

My BERT posts quantised activations to int8 with one scale per row. That isn’t enough for CLIP. A loss breakdown showed why: in the MLP’s second linear layer, some input channels are up to 152× larger than the median channel. With one scale per row, those outliers squash every other value in the row to a few integer levels.

Switching to one scale per group of 128 values fixed most of it without any training. With int8 weights, top-5 agreement went from 0.905 (per row) to 0.942–0.949 (per group). In Rust it’s a small change to the quantiser, and the GEMM applies one scale per group:

pub fn quantize_groups(x: &[f32], rows: usize, k: usize, g: usize) -> (Vec<i16>, Vec<f32>) {
    let mut q = vec![0i16; rows * k];
    let mut s = vec![1f32; rows * k / g];
    for (gi, (xg, qg)) in x[..rows * k].chunks_exact(g).zip(q.chunks_exact_mut(g)).enumerate() {
        let mx = xg.iter().fold(0f32, |m, v| m.max(v.abs()));
        if mx > 0.0 {
            s[gi] = mx / 127.0;
        }
        for (o, &v) in qg.iter_mut().zip(xg) {
            *o = (v / s[gi]).round_ties_even().clamp(-127.0, 127.0) as i16;
        }
    }
    (q, s)
}

The 4-bit weights get the same treatment: one f16 scale per 64 inputs of each output row. The GEMM sums integer dot products within each group, then adds group_sum × weight_scale × activation_scale into an f32 accumulator.

Quantisation-aware training with distillation

The fix for 4-bit weights was quantisation-aware training (QAT) with distillation. The student model runs the exact 4-bit and int8 arithmetic of the Rust code in its forward pass, with straight-through gradients, and learns to reproduce the full-precision teacher. For example, the activation quantiser in training mirrors quantize_groups exactly:

def act_q(x, group=128):
    sh = x.shape
    g = x.reshape(*sh[:-1], sh[-1] // group, group)
    s = g.abs().amax(-1, keepdim=True).clamp_min(1e-8).detach() / 127
    return (ste_round(g / s).clamp(-127, 127) * s).reshape(sh)

The weight scales are learnable too (LSQ-style), so training also finds the best rounding range for each group.

The loss has three parts: cosine to the teacher’s vector, a KL divergence over how the teacher ranks the catalog, and a per-block hidden-state match. The ranking term matters most: search only cares about the order of results.

After 6,000 steps (1.7 hours on the GPU), top-5 agreement went from 0.687 back up to 0.884.

Hurdle: the catalog became the main loss

The catalog has to be compressed from 2 KB to about 200 bytes per image, and still be searched in a few tens of milliseconds without decompressing it.

A byte-packed catalog with lookup-table search

The compression has four steps, all done offline:

  1. Rotate to a PCA basis. After rotation, the first dimensions carry most of the variance.
  2. Give bits where the variance is. Dimensions get between 0 and 8 bits each, allocated greedily: each extra bit cuts a dimension’s error by about 4×, so the next bit always goes where it removes the most error.
  3. Use Lloyd-Max levels. Each dimension gets its own set of levels fitted to its distribution, instead of evenly spaced ones.
  4. Pack whole codes into bytes. Codes are packed so that each byte holds complete codes whose widths add up to exactly 8 bits.
def allocate(var, budget_bits, maxb=8):
    """Greedy reverse water-filling: distortion of dim j ~ var_j * 4^-bits_j."""
    bits = np.zeros(len(var), int)
    gain = np.asarray(var, np.float64).copy()
    for _ in range(budget_bits):
        j = int(np.argmax(gain * (bits < maxb)))
        bits[j] += 1
        gain[j] /= 4
    return bits

The byte packing is what makes the search fast. Each byte holds whole codes, so its contribution to the score depends only on that byte’s value. Per query, the function precomputes a 256-entry table for each byte position:

/// Per-query lookup tables: lut[b*256 + x] = score contribution of code byte b with value x.
pub fn luts(cat: &Catalog, q: &[f32]) -> Vec<f32> {
    let mut lut = vec![0f32; cat.bytes * 256];
    for (b, slots) in cat.slots.iter().enumerate() {
        let t = &mut lut[b * 256..(b + 1) * 256];
        for s in slots {
            let lv = &cat.levels[cat.off[s.dim]..];
            let mask = (1u32 << s.bits) - 1;
            let qj = q[s.dim];
            for (x, acc) in t.iter_mut().enumerate() {
                *acc += qj * lv[((x as u32 >> s.shift) & mask) as usize];
            }
        }
    }
    lut
}

Scoring an image is then 212 table reads and additions, with no decoding and no multiplications. Eight rows are scored at once, so the independent additions hide the memory latency:

// Eight rows at a time: independent accumulation chains hide lookup latency.
while i < nr {
    let c = &cat.codes[i * nb..(i + R) * nb];
    let mut s = [0f32; R];
    for b in 0..nb {
        let l = &lut[b * 256..b * 256 + 256];
        for r in 0..R {
            // SAFETY: c has R*nb bytes; l has 256 entries and each code is < 256.
            unsafe { s[r] += *l.get_unchecked(*c.get_unchecked(r * nb + b) as usize) };
        }
    }
    // ... keep the top k
}

At 184 bytes per image, end-to-end top-5 agreement fell to 0.742. Two changes won it back:

  1. A 3-bit token embedding table saved about 3 MB, which raised the catalog to 212 bytes per image.
  2. A catalog-aware fine-tune. The student ranks the compressed catalog while the teacher ranks the full-precision one, so the encoder learns to compensate for the compression. This took top-5 from 0.740 to 0.812.

That shipped as Miranda v5: 81.4% top-5 agreement with full CLIP, label precision 0.402 against the original’s 0.398, and about 60 ms per search on Functions.

The component was 52,239,588 bytes, 189 KB below the limit. There was no room left for an image encoder.


Part 3: Measure the image encoder before training it

Benchmark with random weights first

Before training anything, I wanted to know which image models could run fast enough. Speed doesn’t depend on the weight values, only on the shapes. So I built a throwaway app with random weights at the real shapes, using the same kernels, and measured it on Functions. That took an afternoon, against days for training:

ModelWeightsIn-function time (p50)
ViT-Ti/32 (192 dims, 12 layers)int850.8 ms
ViT-S/32 (384 dims, 12 layers)int8129.7 ms
ViT-B/32 (the real CLIP image encoder)int8374.0 ms
ViT-B/324-bit470.1 ms

Two findings:

  • The real image encoder was not an option. 374 ms per photo, and about 88 MB even at int8.
  • 4-bit is slower than int8 for images. Unpacking nibbles costs more than the memory bandwidth it saves, because the image encoder processes many patches per weight.

So the plan became: train a small student image model to produce the same CLIP vectors as ViT-B/32, and make room for it in the app.

The idea that makes this work: if the student’s output lives in the same CLIP space as the catalog, the catalog stays as it is. Text queries, photo queries and the catalog share one search.


Part 4: Making room with 3-bit weights and a Hadamard rotation

The image student needed about 4–6 MB. The only place to find that was the text encoder.

I tried lower bit widths with the same QAT recipe (1,000 steps each):

WeightsBefore training (top-5 / top-1)After 1,000 stepsTrend
4-bit0.687 / 0.5700.878 / 0.837flattening
3-bit0.384 / 0.2480.788 / 0.752still rising
2-bit0.040 / 0.0080.605 / 0.496stuck

2-bit was a dead end for a 63M-parameter model. 3-bit was promising, but needed help.

The Hadamard rotation

Low-bit quantisation struggles with outliers. A few large values in a weight group force a large scale, and every other weight in that group gets rounded to almost nothing. Transformer activations have the same problem: in this model, some MLP input channels were up to 152× larger than the median.

A Hadamard rotation spreads outliers evenly across all channels. If R is orthogonal, then x · W = (x R)(Rᵀ W), so rotating both the input and the weights gives the same result in exact arithmetic. But the rotated values are much easier to quantise.

In training, each linear layer’s weights are stored pre-rotated and the input is rotated in the forward pass:

def rotation(k, seed):
    """Orthogonal R = diag(sign) @ H / sqrt(k), sign from torch.Generator().manual_seed(seed)."""
    H = torch.ones(1, 1)
    while H.shape[0] < k:
        H = torch.cat([torch.cat([H, H], 1), torch.cat([H, -H], 1)], 0)
    g = torch.Generator().manual_seed(seed)
    sign = (torch.randint(0, 2, (k,), generator=g) * 2 - 1).float()
    return (sign[:, None] * H / math.sqrt(k)).to(dev)

In Rust, there’s no need to multiply by a 2048 × 2048 matrix. The Hadamard transform has a fast O(k log k) butterfly form, like an FFT:

fn fwht(v: &mut [f32]) {
    let mut h = 1;
    while h < v.len() {
        for i in (0..v.len()).step_by(2 * h) {
            for j in i..i + h {
                let (a, b) = (v[j], v[j + h]);
                v[j] = a + b;
                v[j + h] = a - b;
            }
        }
        h *= 2;
    }
}

Every linear layer input goes through sign flip → FWHT → 1/√k scaling, then int8 quantisation:

fn linear_in(lin: &LinearQ, x: &[f32], rows: usize, ag: usize) -> (Vec<i16>, Vec<f32>) {
    let k = lin.k;
    match lin.sign {
        None => quantize_groups(x, rows, k, ag),
        Some(sign) => {
            let norm = 1.0 / (k as f32).sqrt();
            let mut r = x[..rows * k].to_vec();
            for row in r.chunks_exact_mut(k) {
                for (v, &s) in row.iter_mut().zip(sign) {
                    *v *= s as f32;
                }
                fwht(row);
                row.iter_mut().for_each(|v| *v *= norm);
            }
            quantize_groups(&r, rows, k, ag)
        }
    }
}

The same rotation is used for the 3-bit token table. Each row is stored rotated, and the function undoes the rotation when it reads a row:

// Undo the rotation: row = sign * FWHT(h) / sqrt(D)
fwht(&mut h);
let norm = 1.0 / (D as f32).sqrt();
for ((o, &v), &sg) in out.iter_mut().zip(&h).zip(self.sign) {
    *o = v * norm * sg;
}

The trap: the random signs must match training exactly. They come from torch.Generator().manual_seed(1000 + 10·block + j), so the export script writes the sign vectors into the model file rather than regenerating them in Rust. One wrong sign would give a different model with no error message. The golden check would catch it.

3-bit weights in bit planes

3 bits don’t divide a byte evenly. I stored each group of 64 weights as two planes: a 16-byte plane holding the low 2 bits, and an 8-byte plane holding the high bit. That gives 24 bytes per 64 weights. The SIMD kernel unpacks them with masks, shifts and a byte swizzle:

/// 64 weights (24 bytes) -> 8 x i16x8 in input order.
unsafe fn unpack(p: *const u8) -> [v128; 8] {
    let lo = v128_load(p as *const v128);
    let hi = v128_load64_zero(p.add(16) as *const u64);
    let three = u8x16_splat(3);
    let four = u8x16_splat(4);
    let bit = u8x16(1, 2, 4, 8, 16, 32, 64, 128, 1, 2, 4, 8, 16, 32, 64, 128);
    let ls = [v128_and(lo, three), v128_and(u8x16_shr(lo, 2), three),
              v128_and(u8x16_shr(lo, 4), three), u8x16_shr(lo, 6)];
    // ... for each quarter: select the high-bit byte, test the bit, combine
    let hb = v128_and(i8x16_eq(v128_and(i8x16_swizzle(hi, sel[t]), bit), bit), four);
    let v = i8x16_sub(v128_or(ls[t], hb), four);   // 0..7 -> -4..3
    // ... widen to i16 for i32x4_dot_i16x8
}

The result

After the full run and the catalog-aware fine-tune:

Text encoderTop-5 same as full CLIPTop-1 sameLabel precision@5
Original (fp32)100%100%0.398
version with (4-bit)81.4%77.0%0.402
version with (3-bit + rotation)80.3%74.2%0.405

About one point of top-5 lost. The text encoder file is now 25.5 MB, and 4.8 MB was freed for the image encoder. Encoding got slower (15.4 ms instead of 11.1 ms in wasmtime) because of the extra unpacking and the FWHT. That was a trade I was happy to make.


Part 5: The image student, and a dead end

Dead end: a small ViT trained from scratch

My first student was a ViT-Ti/32: 6 million parameters, the same architecture family as the teacher, and Phase 0 had shown it would run in about 50 ms.

I trained it for 30 epochs on 505,000 images: 405,000 Open Images photos (CC BY 2.0) plus the catalog. I already had the teacher’s embeddings for all of them, so no extra GPU time went to the teacher.

It failed:

StudentTop-5 agreementCosine to teacherFind this item (first)
ViT-Ti/32 from scratch, 30 epochs0.1330.50959.6%

Training cosine was also only about 0.54, so this was underfitting, not overfitting. ViTs are known to need a lot of data. Half a million images is not enough to learn vision from scratch.

Lesson: a student doesn’t have to share the teacher’s architecture. It has to reproduce the teacher’s output space. Start from something that already knows how to see.

Distil into a different, pretrained architecture

I switched to ImageNet-pretrained CNNs from timm (Apache-2.0), with a linear projection to CLIP’s 512 dimensions: MobileNetV3-Large and EfficientNet-B0.

One more hurdle: at a learning rate of 1e-3, the pretrained features collapsed to zero within a few hundred steps. At 3e-4 they trained well. Pretrained backbones need a gentler start than a model trained from scratch.

The loss is the same idea as for the text encoder: cosine to the teacher, plus a KL divergence over how the teacher ranks the whole 100k catalog:

with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    t = F.normalize(teacher(x).float(), dim=-1)
with torch.autocast("cuda", dtype=torch.bfloat16):
    s = model(down(x))
s = F.normalize(s.float(), dim=-1)
l_cos = (1 - (s * t).sum(1)).mean()
l_kl = F.kl_div(F.log_softmax(s @ Tc.T / args.tau, 1), F.log_softmax(t @ Tc.T / args.tau, 1),
                log_target=True, reduction="batchmean")
loss = l_cos + args.kl * l_kl

Tc is the catalog matrix (100,000 × 512) kept on the GPU, so every training step scores the whole catalog for every image in the batch.

How to measure an image student

Exact top-5 agreement with the teacher turned out to be the wrong metric for images. Many catalog items are near-ties, and no small student will reproduce the teacher’s order among them. So I used two tests that don’t depend on the teacher:

  • Similar items: 2,000 fashion photos not in the catalog. Do the top 5 results share a garment label (dress, jacket, bag…) with the photo? (label precision@5)
  • Find this item: 500 catalog images, held out of training, re-cropped, resized, brightness-shifted and re-saved as low-quality JPEG. Does the original come back first, or in the top 5?

After 10 epochs:

StudentLabel precision@5Find this item @1@5
ViT-B/32 teacher0.48397.2%99.6%
ViT-Ti/32 from scratch0.39159.6%79.0%
MobileNetV3-Large (pretrained)0.46882.8%93.8%
EfficientNet-B0 (pretrained)0.48285.6%95.6%

On relevance, the pretrained CNNs were already close to the teacher.


Part 6: The preprocessing trap

This was the most surprising hurdle of the project.

While building the evaluation sets, I first resized images with a plain bilinear resize and a simple centre crop. Then I fed those pixels to the real ViT-B/32 image encoder and compared its results with its own results on OpenCLIP’s preprocessing.

PreprocessingReal ViT-B/32 vs itself (top-5 / top-1)Cosine
Bilinear resize, floor crop0.776 / 0.7350.982
OpenCLIP’s exact transform0.997 / 0.998~1.0

The same model lost 22 points of top-5 agreement with itself just from a slightly different resize. The retrieval metric is steep: a cosine of 0.98 sounds close, but with 100,000 near-tied catalog items it reorders the top results.

A bit-exact port of Pillow’s resize

So the Rust side had to reproduce OpenCLIP’s preprocessing exactly. That means Pillow’s antialiased bicubic resize: separable passes, a = −0.5, the filter support scaled by the downscale factor, fixed-point coefficients with 22 fractional bits, and rounding to u8 between the horizontal and vertical passes.

I ported it from Pillow’s Resample.c:

const PRECISION_BITS: u32 = 32 - 8 - 2;

fn bicubic_filter(x: f64) -> f64 {
    const A: f64 = -0.5;
    let x = x.abs();
    if x < 1.0 {
        ((A + 2.0) * x - (A + 3.0)) * x * x + 1.0
    } else if x < 2.0 {
        (((x - 5.0) * x + 8.0) * x - 4.0) * A
    } else {
        0.0
    }
}

fn coeffs(in_size: usize, out_size: usize) -> (usize, Vec<(usize, usize)>, Vec<i32>) {
    let scale = in_size as f64 / out_size as f64;
    let filterscale = scale.max(1.0);
    let support = 2.0 * filterscale;
    // ...
    for xx in 0..out_size {
        let center = (xx as f64 + 0.5) * scale;
        // C casts truncate toward zero; center - support + 0.5 can be negative
        let xmin = ((center - support + 0.5) as i64).max(0) as usize;
        // ... normalise in f64, then round to fixed point the way C does
        kk[xx * ksize + x] = if w < 0.0 { (-0.5 + f) as i32 } else { (0.5 + f) as i32 };
    }
}

The details that matter are the C-specific ones. Casts truncate toward zero, negative weights round differently from positive ones, and a pass is skipped when that dimension doesn’t change. With all of them in place, the Rust resize matches Pillow bit for bit on the same input pixels. The crop offsets also follow torchvision exactly: round((n − 224) / 2), with half rounding to even.

The part I couldn’t make exact: JPEG decoding

Rust decodes JPEGs with zune-jpeg. Pillow uses libjpeg-turbo. Both are correct, but they differ by a few levels per pixel (up to 4–13 on some images), because JPEG’s inverse DCT and chroma upsampling allow small differences between implementations.

That meant the golden check needed two thresholds:

  • Encoder alone, fed Pillow’s exact crop: cosine ≥ 0.99 against the Python model.
  • End to end, from the JPEG bytes: cosine ≥ 0.98, allowing for the decoder difference.

Lesson: for a distilled vision model, preprocessing is part of the model. Measure the teacher against itself under your preprocessing before you blame the student.


Part 7: A CNN engine in Rust, and choosing 160 px

The text kernels were for transformers. A CNN needed new ops: dense convolution for the stem, 1×1 (pointwise) convolution, depthwise convolution, squeeze-and-excite blocks, and hardswish/SiLU activations.

As in my BERT port, I didn’t ship a runtime. What’s new is how a CNN gets reduced to a few ops the Rust engine can run fast.

An op list with folded BatchNorm and padded channels

The export script walks the timm model and writes a flat op list. Three things happen on the way:

  • BatchNorm is folded into the convolution before it. At inference time BatchNorm is a fixed per-channel scale and shift, so it can be multiplied into the conv’s weights and bias. The BatchNorm layers disappear.
  • Channels are padded to multiples of 16, so every SIMD loop runs on full vectors with no remainder handling. The padded weights and biases are zero, so the padded channels stay zero.
  • Every matrix op becomes one int8 GEMM. The 1×1 convolutions, the squeeze-and-excite layers, the projection and the stem (via im2col) all use the int8 kernel from the text encoder.
def fold(conv, bn):
    """Conv weight (out, in/groups, k, k) and BN -> folded weight, bias (float64)."""
    w = conv.weight.detach().double()
    b = conv.bias.detach().double() if conv.bias is not None else torch.zeros(w.shape[0], dtype=torch.float64)
    if bn is not None and hasattr(bn, "running_var"):  # qsim-folded models carry only the activation
        g = bn.weight.detach().double() / torch.sqrt(bn.running_var.double() + bn.eps)
        w = w * g[:, None, None, None]
        b = (b - bn.running_mean.double()) * g + bn.bias.detach().double()
    return w, b

The resulting format:

"CNN1" u32 n_ops u32 out_dim u32 res | f32 mean[3] std[3] | n_ops x op
  1 DENSE [k, stride, pad, cin, cout, act, kk]: kxk conv via im2col
  2 PW    [cin, cout, act]: 1x1 conv
  3 DW    [k, stride, pad, c, act]: depthwise
  4 SE    [c, cr, act, gate]: squeeze-excite
  5 SAVE  push the activation (residual)
  6 ADD   pop and add
  7 POOL  global average
  8 FC    [cin, out]: final projection
  mat(n x k): i8 w[n*k] | f32 scale[n] | f32 bias[n]

The Rust engine is a loop over this list, with the model compiled into the binary.

A register-blocked depthwise kernel

Depthwise convolutions have only k × k weights per channel, too little work for a GEMM, so they stay in f32 with their own kernel. Thanks to the channel padding, it processes 16 channels at a time and keeps those 16 accumulators in SIMD registers across all taps. The valid tap range at the image border is computed once per output row and column, so the inner loops have no bounds checks:

for cb in (0..c).step_by(16) {
    let mut acc = [0f32; 16];
    acc.copy_from_slice(&bias[cb..cb + 16]);
    for ky in ky0..ky1 {
        let iy = oy * s + ky - p;
        for kx in kx0..kx1 {
            let ix = ox * s + kx - p;
            let xi = &x[(iy * w + ix) * c + cb..][..16];
            let wk = &wt[(ky * k + kx) * c + cb..][..16];
            mac16(&mut acc, xi, wk);
        }
    }
    out[ob + cb..ob + cb + 16].copy_from_slice(&acc);
}

MobileNetV3’s activations also help: hardswish and hard sigmoid are just clamps and multiplies, with no exp.

On Functions, with random weights at the real shapes:

Model (no conv_head)224 px192 px160 px
MobileNetV3-Large69.7 ms51.4 ms36.3 ms
EfficientNet-B0102.8 ms75.2 ms52.4 ms

The SIMD depthwise kernel alone took MobileNetV3 from 80.7 to 69.7 ms at 224 px.

Drop the head to fit

As trained, MobileNetV3-Large has a 960 → 1280 conv_head before the projection, and it was too big for the 4.8 MB budget at int8. I retrained without it (--nohead, 960 → 512 projection): about 3.5M parameters, 3.9 MB.

Lower resolution, same quality

The input is the OpenCLIP 224 crop, resized again to the student’s resolution. I trained 5-epoch pilots at three sizes:

StudentFunctions timeLabel precision@5Find this item @1@5
MobileNetV3 @ 160 px36 ms0.46185.4%95.6%
MobileNetV3 @ 192 px51 ms0.46085.0%94.8%
MobileNetV3 @ 224 px70 ms0.47081.2%94.2%
EfficientNet-B0 @ 160 px52 ms0.45886.6%96.6%
ViT-B/32 teacher374 ms0.48397.2%99.6%

Going from 224 to 160 px cost no measurable quality and made the encoder 1.9× faster. I chose MobileNetV3-Large at 160 px: the fastest model, and it fits the app.


Part 8: int8 needs its own training too

EfficientNet-B0 survived int8 rounding without retraining (cosine ≥ 0.996 against the float model). MobileNetV3 didn’t: 0.969–0.992, below the 0.99 gate. MobileNets are known to be sensitive to quantisation, especially in the depthwise and squeeze-and-excite layers.

Simulate the CNN engine’s int8 maths in training

As for the text encoder in Part 2, I simulated the Rust arithmetic in training. qsim folds every BatchNorm into its convolution and fake-quantises each matrix op the way cnn-core runs it: int8 per pixel for inputs, int8 per output channel for weights, and depthwise in f32:

def q_rows(x, dim):
    """Symmetric int8 per slice along `dim` (max |x| -> 127)."""
    s = x.abs().amax(dim, keepdim=True).clamp_min(1e-12).detach() / 127
    return ste_round(x / s).clamp(-127, 127) * s


class QConv(nn.Conv2d):
    """Conv2d with int8 weights; matrix convs also quantise their input per pixel."""

    def forward(self, x):
        w = q_weight(self.weight) if self.groups == 1 else self.weight
        if self.groups != 1:
            return F.conv2d(x, w, self.bias, self.stride, self.padding, self.dilation, self.groups)
        if self.kernel_size == (1, 1):
            return F.conv2d(q_rows(x, 1), w, self.bias, self.stride, self.padding)
        # dense kxk conv as Rust's im2col: rows are output pixels, (ky, kx, ci) per row
        # ...

Even the stem convolution is simulated as Rust runs it, via unfold (im2col), so that the quantisation groups match.

Final training: 25 epochs, then a 3-epoch QAT fine-tune at a learning rate of 3e-5:

ModelLabel precision@5Find this item @1@5
25 epochs, fp320.46285.0%94.2%
25 epochs, int8 without QAT0.46583.8%93.2%
25 epochs + QAT, int8 (shipped)0.46185.0%94.6%
ViT-B/32 teacher0.48397.2%99.6%

QAT recovered the int8 loss completely. The Rust golden check passed: encoder cosine ≥ 0.9935 on all 16 test images, and ≥ 0.987 end to end from the JPEG.

One surprise: the 25-epoch run got closer to the teacher (higher cosine) than the 5-epoch pilot, but retrieval didn’t improve. Cosine to the teacher is a proxy. What matters is the ranking, so measure that.


Part 9: One space, one catalog, and “More like this” for free

The catalog is stored in a PCA basis: the compression rotates the vectors so that the first dimensions carry the most information and get the most bits. The text encoder already had this rotation folded into its final projection, so its output is in catalog coordinates with no extra work per query.

Fold the catalog’s rotation into the last layer

I did the same for the image student. A rotation doesn’t change cosine similarity, so it can be multiplied into the last linear layer at export time. The query never needs a separate 512 × 512 rotation at run time:

w = model.proj.weight.detach().double()
if basis is not None:  # output in catalog coordinates (rotation: cosines unchanged)
    w = torch.from_numpy(basis) @ w

The basis is recomputed exactly as the catalog export computes it (SVD of the normalised catalog embeddings). The export checks it by decoding the compressed catalog in numpy (mean cosine 0.995 against the original vectors).

Now text vectors, photo vectors and catalog rows are all in the same coordinates, and one search function serves all three. That also gave me a third feature at no cost. “More like this” decodes a catalog row and uses it as the query:

fn similar(id: Option<usize>, k: usize, t_total: Instant) -> Response {
    let cat = miranda_core::Catalog::load(&CATALOG_BLOB.0);
    // ...
    let mut v = miranda_core::row(&cat, id);
    let n = v.iter().map(|x| x * x).sum::<f32>().sqrt();
    v.iter_mut().for_each(|x| *x /= n);
    let (res, search_ms, meta_ms, count) = results(&v, k, 0.0, Some(id));
    // ...
}

It takes 31 ms on Functions, almost all of it the catalog scan.


Part 10: One component

Everything compiled in

The text encoder, the tokenizer, the image encoder, the catalog and the web page are all embedded with include_bytes! and read in place. The +simd128 flag and release profile are the same as in the BERT port, and the translation post shows what happens without the flag. The only addition is a 16-byte-aligned wrapper, because three different model files are now cast in place to &[f32], &[u16] and &[u64] slices:

static CATALOG_BLOB: &Aligned<[u8]> = &Aligned(*include_bytes!("../../assets/catalog6.bin"));
static CNN_BLOB: &cnn_core::Aligned<[u8]> = &cnn_core::Aligned(*include_bytes!("../../assets/cnn6.bin"));

The final component is 51,589,983 bytes, with 838,817 bytes to spare. That’s smaller than v5, which had no image encoder.

PieceSize
Text encoder (3-bit, with 3-bit token table)25.5 MB
Catalog (100,000 × 212 B + levels)21.2 MB
Image encoder (int8)3.9 MB
Tokenizer (merge pairs only)0.47 MB
Code and web page~0.5 MB

Endpoints

GET  /search?q=…      routing → BPE tokenizer → ViT-B-32 text encoder (3-bit)
POST /image (JPEG)    JPEG decode → OpenCLIP 224 crop → 160 px → MobileNetV3 (int8)
GET  /similar?id=N    the catalog's own vector for image N
                      ↓ all three in catalog coordinates
                      LUT search over 100k × 212 B → top-k image URLs + credits

Hurdles from real photos

Testing with real uploads found one bug and one risk that the test set didn’t:

  • A grayscale JPEG crashed the resize. zune-jpeg returns one channel for grayscale images even when asked for RGB.
  • A very large photo could exceed the 128 MB per-request memory limit while decoding, before the code could check its size.

Both are now handled in decode, which reads the header before decoding any pixels:

pub fn decode(jpeg: &[u8]) -> Result<(usize, usize, Vec<u8>), String> {
    // ...
    dec.decode_headers().map_err(|e| format!("jpeg: {e:?}"))?;
    let (w, h) = dec.dimensions().ok_or("jpeg: no dimensions")?;
    if w == 0 || h == 0 {
        return Err("jpeg: empty image".into());
    }
    if w * h > MAX_PIXELS {
        return Err(format!("image too large: {w}x{h} (max {MAX_PIXELS} pixels)"));
    }
    let px = dec.decode().map_err(|e| format!("jpeg: {e:?}"))?;
    // grayscale JPEGs come out as one channel despite the RGB output setting
    match px.len() / (w * h) {
        3 => Ok((w, h, px)),
        1 => Ok((w, h, px.iter().flat_map(|&v| [v, v, v]).collect())),
        c => Err(format!("jpeg: unexpected {c} channels")),
    }
}

The handler also rejects bodies over 2 MiB (413) and anything that doesn’t start with the JPEG marker FF D8 (415).

Let the browser do the cheap part

A phone photo can be 12 megapixels, but the encoder only looks at a 224 px centre crop. So the web page shrinks the photo before uploading it. It also applies the EXIF orientation, which the server would otherwise have to handle:

// Photo -> JPEG, shortest side at most 448 px (the encoder uses a 224 centre crop), EXIF orientation applied.
async function shrink(file) {
  const bmp = await createImageBitmap(file, {imageOrientation: "from-image"});
  const s = Math.min(1, 448 / Math.min(bmp.width, bmp.height));
  const c = document.createElement("canvas");
  // ... draw at scale s
  return await new Promise(ok => c.toBlob(ok, "image/jpeg", 0.92));
}

This also means PNG, HEIC and WebP photos work on the page: the browser converts them to JPEG. Uploads are a few hundred KB and decode in 2–6 ms.


Every technique, at a glance

Size: getting three pieces into 50 MiB

TechniqueWhat it boughtWhere
QAT + distillation down to 4-bit, then 3-bit weightsText encoder from 252 MB (fp32) to 25.5 MB. At 3 bits, top-5 agreement is 0.38 without training and 0.80 after itParts 2, 4
Hadamard rotation of weights and token table (FWHT in Rust)Made 3-bit weights and a 3-bit table usable; freed 4.8 MB for the image encoderPart 4
3-bit bit-plane packing (2-bit plane + 1-bit plane per 64 weights)3 bits per weight with a fast SIMD unpackPart 4
Token table limited to ASCII-reachable tokens, with a u16 remap2,269 fewer rowsPart 1
Tokenizer stored as merge pairs only (id = 512 + rank)1.8 MB → 0.47 MBPart 1
PCA + per-dimension bit allocation + Lloyd-Max levels + byte packingCatalog from 2,048 to 212 bytes per imagePart 2
Catalog-aware fine-tune of the text encoderWon back 7 points of top-5 lost to catalog compressionPart 2
Distilling CLIP’s image encoder into a pretrained MobileNetV3Image encoder from about 88 MB (int8) to 3.9 MBPart 5
Dropping the CNN’s conv_headMade the student fit the 4.8 MB budgetPart 7
BN folding into convolutionsNo BatchNorm layers at run timePart 7
int8 QAT that simulates the CNN engineint8 image model with no quality lossPart 8
include_bytes!, flat binaries read in placeNo parse step, no copies (reused: BERT port)Part 10

Latency: keeping each request under 100 ms

TechniqueWhat it boughtWhere
Running only the real tokens (causal mask + <eot> pooling)7.1 s → ~0.5 s per text query, identical outputPart 1
<eot> row only in the last block15.3 → 12.8 ms in wasmtime (reused idea: 1 s → 260 ms)Part 1
int8 SIMD GEMM with a 4 × 2 register tile~0.5 s → 24 ms (reused: 1 s → 260 ms)Part 1
Per-group (128) activation scalesTop-5 0.905 → 0.94+ at no speed costPart 2
Lookup-table catalog search, 8 rows at a time100k images scored in ~30 ms with no decompressionPart 2
Random-weights benchmark before trainingRuled out the ViT-B/32 image encoder (374 ms) and 4-bit image weightsPart 3
Register-blocked SIMD depthwise convolution, channels padded to 16MobileNetV3 80.7 → 69.7 ms at 224 pxPart 7
160 px student input instead of 22470 → 36 ms with the same qualityPart 7
Catalog PCA rotation folded into both encoders’ last layerOne search path for text, photos and “More like this”; no extra rotation per queryPart 9
Header check before decoding, browser-side shrink to 448 pxUploads of a few hundred KB, decode in 2–6 ms, no memory blow-upsPart 10

Quality: matching the original

TechniqueWhat it boughtWhere
KL loss over the teacher’s ranking of the whole catalogThe student learns the order of results, which is what search usesParts 2, 5
Bit-exact port of Pillow’s bicubic resizeAvoided a 22-point top-5 loss from preprocessing alonePart 6
Two-threshold golden check (encoder ≥ 0.99, end to end ≥ 0.98)Caught real errors without failing on JPEG decoder differencesPart 6

The full picture

Quality against the original server setup

Text search, 252 retrieval queries, same 100k catalog:

PipelineTop-5 same as originalTop-1 sameLabel precision@5
Original (full-precision CLIP, uncompressed catalog)100%100%0.398
Lookup table (first edge version)52.3%41.3%0.416
Final Version on Functions80.3%74.2%0.405

Photo search:

Image encoderLabel precision@5Find this item @1@5Time per photo
ViT-B/32 (the catalog’s own encoder)0.48397.2%99.6%374 ms (encoder only, on Functions)
On Functions (MobileNetV3, int8, compressed catalog, over HTTP)0.46282.2%93.6%82 ms (whole request)

Text search relevance is at least as good as the original. Photo search relevance is within about 2 points of the full ViT-B/32 image encoder. “Find this exact item” is where the gap shows: the original comes back first 82% of the time, against 97% for the teacher.

Speed on Akamai Functions

Median of 12 requests, measured from India:

RequestInside the FunctionRound trip
Photo search82 ms (decode 2–6, resize 9–13, encode 34–40, search 29–31)167 ms
Text search58 ms124 ms
More like this31 ms

What didn’t work

IdeaWhy it failed
Running all 77 token positionsPadding after <eot> can’t affect a causal model’s output. It was 15× wasted work
4-bit weights without retrainingTop-5 agreement dropped to 0.50–0.67
2-bit text encoderPlateaued at 0.60 top-5, even with QAT
Shrinking the catalog instead of the encoderAt 184 B/image, top-5 fell to 0.74. Bits in the catalog were worth more than bits in the encoder
4-bit image encoderSlower than int8 on Functions: unpacking costs more than it saves
The real ViT-B/32 image encoder374 ms per photo and about 88 MB at int8
A ViT-Ti/32 student from scratchUnderfit badly (cosine 0.51) on half a million images
A learning rate of 1e-3 for pretrained CNNsThe features collapsed to zero
Bilinear resize “close enough”Cost the real ViT-B/32 22 points of top-5 agreement with itself
Training longer for a higher cosine25 epochs raised the cosine to the teacher, but not retrieval quality

Lessons

  1. Measure the platform first. A random-weights benchmark ruled out the real image encoder and 4-bit image weights before any training time was spent.
  2. Exploit the model’s structure. The causal mask and <eot> pooling cut text encoding by more than 10× with an identical result.
  3. Quantisation is free only with training. Simulate the exact integer arithmetic of your runtime in the forward pass, and distil against the full-precision model.
  4. Rotate before you quantise. A Hadamard rotation spreads outliers across all channels, and a fast Walsh-Hadamard transform makes it cheap at run time.
  5. Train for the ranking, not the vector. A KL loss over how the teacher ranks the catalog matters more than cosine to the teacher.
  6. A student doesn’t need the teacher’s architecture. It needs the teacher’s output space. A pretrained CNN beat a ViT trained from scratch by a wide margin.
  7. Preprocessing is part of the model. Port it bit-exactly, and measure the teacher against itself under your preprocessing.
  8. Fold fixed linear maps into the weights. The catalog’s PCA rotation costs nothing at run time once it’s inside the last layer.
  9. Spend the byte budget where it buys the most quality. Every byte was traded between the text encoder, the token table, the catalog and the image encoder by measuring end-to-end quality, not per-component accuracy.
  10. Keep a golden check with honest thresholds. Two thresholds, one for the encoder and one end to end, made the JPEG decoder difference visible instead of hiding it.

What’s next

  • A catalog-aware fine-tune for the image student. The text encoder gained 7 points of top-5 from this. For images, compression costs about 2.4 points of “find this item” @1.
  • Faster catalog search. At about 30 ms, scanning 100k compressed rows is now a third of a photo search.
  • A SIMD Walsh-Hadamard transform and a cheaper 3-bit unpack, to win back the few milliseconds the 3-bit text encoder added.

For now, one 51.6 MB WebAssembly component on Akamai Functions runs CLIP’s full text encoder, a photo encoder trained to match CLIP’s image encoder, and search over 100,000 images. It answers a text query in about 60 ms and a photo in about 80 ms, with no GPU and no calls to any other service.


Written By
Fareeth John

I’m an Enterprise Architect at Akamai Technologies with 15+ years across mobile engineering, edge infrastructure, security, and AI systems. Having launched 45+ apps on the App Store and Play Store (iOS, Android, Flutter, React Native), I specialize in mobile SDK internals, Frida-based security, and high-concurrency edge runtimes like Akamai EdgeWorkers, Akamai Functions and HarperDB. My current focus is optimizing small language models and porting them to constrained, CPU-only runtimes like Akamai Functions, alongside Agentic AI frameworks (LangGraph, MCP), WASM-based AI guardrails, self-hosted LLM inference, and real-time voice pipelines.

Leave a Reply

Your email address will not be published. Required fields are marked *