How I fitted CLIP’s full ViT-B-32 text encoder, a distilled image encoder and a 100,000-image catalog into a single Akamai Functions app, and the hurdles along the way.
Where I started
Miranda, by Alex Leung, is a fashion image search demo. It embeds a catalog of clothing photos with OpenCLIP ViT-B-32, embeds your text query with the same model, and ranks the images by cosine similarity. It runs as a Streamlit app with PyTorch on a server.
My first edge version of Miranda avoided running CLIP at query time. I distilled the text encoder into a lookup table with one vector per token, so turning a query into a vector took about 1 ms. Search quality on labelled queries was as good as the original. But the lookup table ignores word order, and its top 5 results matched full CLIP’s only about half the time.
A lookup table is not the model, though. I wanted a hard goal: run the full ViT-B-32 encoder entirely inside an Akamai Function. So I set myself these targets:
- Run the real ViT-B-32 text encoder, all 12 transformer blocks, inside the Function.
- Add search by photo: upload a picture and get similar items. That needs an image encoder, in the same app.
- Keep the 100,000-image catalog in the same app, with no outbound calls, no GPU and no external vector database.
- Match the quality of the original server setup as closely as possible.
That is all inside one Akamai Functions component, which has to fit in 52,428,800 bytes (50 MiB).
This post is the journey: what I measured, what broke, what I threw away, and the techniques that got both encoders into one Function.
The rules I set myself
- Measure against the original. Every version is scored against full-precision OpenCLIP over the same catalog: do the top 5 results match, and are the results relevant
- A golden check before every deploy. The Rust code must reproduce a Python reference within a stated tolerance, under plain
wasmtime, before anything goes to Functions. - Run all the heavy work offline. Training, embedding and exporting run on a Linode GPU instance. Only per-query work runs on Functions.
What I’m reusing from earlier posts
The foundations are the same as in my earlier posts. I use them here without explaining them again:
- From Porting a 23M-Parameter BERT Model to Akamai Functions:
- a hand-written forward pass instead of a model runtime
- weights exported to a flat binary, compiled in with
include_bytes!and read in place as slices, with no parsing per request - a golden reference to check the Rust output against
- the
+simd128target flag and the release profile - the
/...wildcard route and an emptyallowed_outbound_hosts
- From From 1 Second to 260 ms:
- the int8 SIMD GEMM (
i16x8_extend+i32x4_dot_i16x8, with a 4-row × 2-output register tile) - doing more work per loop iteration, because of epoch interruption
- skipping unneeded rows in the last layer
- per-stage
Server-Timingheaders and a/pingbaseline - one change per deployment, measured as a median of repeated requests
- the int8 SIMD GEMM (
- From From Server Bottleneck to 100% Edge: compact binary formats read in place, instead of JSON.
Everything else in this post is new. The challenge here wasn’t one model. It was fitting three large pieces (a 63M-parameter transformer, an image encoder and a 100,000-image index) into an app that can barely hold one of them. That took a different set of techniques: very low-bit weights, rotations, distillation into a different architecture, index compression, and folding transforms into the weights. A summary table near the end lists every technique, what it saved, and whether it’s new.
The budget
Here’s what has to share 50 MiB:
| Piece | Full precision | Problem |
|---|---|---|
| ViT-B-32 text encoder (63M parameters) | 252 MB in fp32, 63 MB in int8 | Larger than the whole app even at int8 |
| ViT-B-32 image encoder (about 88M parameters) | 350 MB in fp32 | Even bigger, and slow (more on that below) |
| 100,000 catalog vectors (512 dims) | 205 MB in fp32 | Has to be compressed about 10× per image, to about 200 bytes |
| Code, tokenizer, web page | about 1 MB | Fine |
None of it fits as is. Each piece had to be compressed, and all the compression had to be paid for with training so the quality came back.
Part 1: The real text encoder, from 7 seconds to 24 ms
Before thinking about the image encoder, I needed to know whether the text encoder alone could run fast enough on Functions.
The first straightforward port loaded int8 weights, dequantised them to f32 and ran all 77 token positions, as CLIP does. It took 7.1 seconds per query on Functions.
Three changes fixed that:
| Step | wasmtime | Akamai Functions |
|---|---|---|
| 77 tokens, int8 weights dequantised to f32 | 5,340 ms | 7,147 ms |
| + run only the real tokens | 347 ms | 469–494 ms |
| + int8 × int8 SIMD GEMM | 15.3 ms | ~24 ms |
+ last block computes only the <eot> row | 12.8 ms | 24.0 ms p50 |
Run only the real tokens
This was the biggest win, and it costs nothing in accuracy. CLIP’s text transformer uses a causal attention mask: each token only looks at the tokens before it. The output is read at the <eot> (end-of-text) token. The padding after <eot> can never affect it.
So a query like “black winter coat” needs only <sot> black winter coat <eot>, 5 positions instead of 77, with an identical result (cosine 1.00000 against the 77-token version).
This is different from the padding trick in my BERT post. That model’s attention was bidirectional, so the padding was load-bearing and had to stay. CLIP’s causal mask means the padding can be removed entirely.
On top of that, I reused the last-layer trick from the same post: in the last block, only the <eot> row is needed after attention, so the MLP for the other rows is skipped:
for (bi, b) in m.blocks.iter().enumerate() {
// In the last block only the <eot> row (the last one) is carried forward.
let last = bi + 1 == m.blocks.len();
layer_norm(&x, b.ln1, &mut h);
let (xq, sx) = linear_in(&b.qkv, &h, l, ag);
gemmq(&xq, &sx, l, ag, &b.qkv, &mut qkv);
let (r0, rows) = if last { (l - 1, 1) } else { (0, l) };
attention(&qkv, l, r0, &mut att);
// ... out_proj, MLP and residuals run on `rows` rows only
}
The int8 SIMD GEMM
The matrix multiplies use the int8 SIMD kernel and 4 × 2 register tile described in From 1 Second to 260 ms, unchanged. That took encoding from about 480 ms to 24 ms.
A smaller tokenizer and token table
Two smaller size savings came from the tokenizer and the token embedding table.
The tokenizer stores only merge pairs. CLIP’s BPE vocabulary has a useful property: the token produced by merge rank r always has id 512 + r. So the token strings aren’t needed at all. The tokenizer file holds the byte ids and a sorted list of merge pairs, searched by binary search:
//! in CLIP's vocab the token made by merge rank r has id 512 + r, so only the //! byte symbol ids and the merge pairs are needed (0.45 MB instead of 1.8 MB). //! //! vit_bpe.bin (little-endian): //! "BPE2" u32 n_merges | u16 byte_id[128] | u16 byte_id_w[128] (with "</w>") //! | u64 key[n_merges] sorted, key = (left_id << 32) | right_id | u16 rank[n_merges] //! Merges whose result ASCII text can never produce are left out.
The token table keeps only reachable tokens. The token embedding table has 49,408 rows of 512 values, which is 101 MB in fp32. Queries are ASCII text, and 2,269 of those tokens can only come from non-ASCII bytes. The export keeps the 47,139 tokens that ASCII can produce and stores a u16 remap from token id to row:
keep = sorted(i for t, i in tk.encoder.items()
if all(tk.byte_decoder[c] < 128 for c in t.replace("</w>", "")) or i in (49406, 49407))
remap = np.full(V, 0xFFFF, np.uint16); remap[keep] = np.arange(len(keep))
The rows themselves are quantised to 3 bits after a Hadamard rotation (explained in Part 4) and decoded one row per token at query time.
At this point the encoder ran in 24 ms on Functions, but the component was 51.9 MB with no catalog in it. The int8 encoder alone filled the app.
Part 2: 4 bits, and a catalog that fits beside it
To fit the catalog, the encoder had to shrink again, to 4-bit weights.
Hurdle: 4-bit without training doesn’t work
Rounding the trained weights to 4 bits (post-training quantisation) dropped top-5 agreement with full CLIP from about 0.95 to 0.50–0.67. That would make the results worse than the lookup table I was trying to replace.
Per-group activation scales
My BERT posts quantised activations to int8 with one scale per row. That isn’t enough for CLIP. A loss breakdown showed why: in the MLP’s second linear layer, some input channels are up to 152× larger than the median channel. With one scale per row, those outliers squash every other value in the row to a few integer levels.
Switching to one scale per group of 128 values fixed most of it without any training. With int8 weights, top-5 agreement went from 0.905 (per row) to 0.942–0.949 (per group). In Rust it’s a small change to the quantiser, and the GEMM applies one scale per group:
pub fn quantize_groups(x: &[f32], rows: usize, k: usize, g: usize) -> (Vec<i16>, Vec<f32>) {
let mut q = vec![0i16; rows * k];
let mut s = vec![1f32; rows * k / g];
for (gi, (xg, qg)) in x[..rows * k].chunks_exact(g).zip(q.chunks_exact_mut(g)).enumerate() {
let mx = xg.iter().fold(0f32, |m, v| m.max(v.abs()));
if mx > 0.0 {
s[gi] = mx / 127.0;
}
for (o, &v) in qg.iter_mut().zip(xg) {
*o = (v / s[gi]).round_ties_even().clamp(-127.0, 127.0) as i16;
}
}
(q, s)
}
The 4-bit weights get the same treatment: one f16 scale per 64 inputs of each output row. The GEMM sums integer dot products within each group, then adds group_sum × weight_scale × activation_scale into an f32 accumulator.
Quantisation-aware training with distillation
The fix for 4-bit weights was quantisation-aware training (QAT) with distillation. The student model runs the exact 4-bit and int8 arithmetic of the Rust code in its forward pass, with straight-through gradients, and learns to reproduce the full-precision teacher. For example, the activation quantiser in training mirrors quantize_groups exactly:
def act_q(x, group=128):
sh = x.shape
g = x.reshape(*sh[:-1], sh[-1] // group, group)
s = g.abs().amax(-1, keepdim=True).clamp_min(1e-8).detach() / 127
return (ste_round(g / s).clamp(-127, 127) * s).reshape(sh)
The weight scales are learnable too (LSQ-style), so training also finds the best rounding range for each group.
The loss has three parts: cosine to the teacher’s vector, a KL divergence over how the teacher ranks the catalog, and a per-block hidden-state match. The ranking term matters most: search only cares about the order of results.
After 6,000 steps (1.7 hours on the GPU), top-5 agreement went from 0.687 back up to 0.884.
Hurdle: the catalog became the main loss
The catalog has to be compressed from 2 KB to about 200 bytes per image, and still be searched in a few tens of milliseconds without decompressing it.
A byte-packed catalog with lookup-table search
The compression has four steps, all done offline:
- Rotate to a PCA basis. After rotation, the first dimensions carry most of the variance.
- Give bits where the variance is. Dimensions get between 0 and 8 bits each, allocated greedily: each extra bit cuts a dimension’s error by about 4×, so the next bit always goes where it removes the most error.
- Use Lloyd-Max levels. Each dimension gets its own set of levels fitted to its distribution, instead of evenly spaced ones.
- Pack whole codes into bytes. Codes are packed so that each byte holds complete codes whose widths add up to exactly 8 bits.
def allocate(var, budget_bits, maxb=8):
"""Greedy reverse water-filling: distortion of dim j ~ var_j * 4^-bits_j."""
bits = np.zeros(len(var), int)
gain = np.asarray(var, np.float64).copy()
for _ in range(budget_bits):
j = int(np.argmax(gain * (bits < maxb)))
bits[j] += 1
gain[j] /= 4
return bits
The byte packing is what makes the search fast. Each byte holds whole codes, so its contribution to the score depends only on that byte’s value. Per query, the function precomputes a 256-entry table for each byte position:
/// Per-query lookup tables: lut[b*256 + x] = score contribution of code byte b with value x.
pub fn luts(cat: &Catalog, q: &[f32]) -> Vec<f32> {
let mut lut = vec![0f32; cat.bytes * 256];
for (b, slots) in cat.slots.iter().enumerate() {
let t = &mut lut[b * 256..(b + 1) * 256];
for s in slots {
let lv = &cat.levels[cat.off[s.dim]..];
let mask = (1u32 << s.bits) - 1;
let qj = q[s.dim];
for (x, acc) in t.iter_mut().enumerate() {
*acc += qj * lv[((x as u32 >> s.shift) & mask) as usize];
}
}
}
lut
}
Scoring an image is then 212 table reads and additions, with no decoding and no multiplications. Eight rows are scored at once, so the independent additions hide the memory latency:
// Eight rows at a time: independent accumulation chains hide lookup latency.
while i < nr {
let c = &cat.codes[i * nb..(i + R) * nb];
let mut s = [0f32; R];
for b in 0..nb {
let l = &lut[b * 256..b * 256 + 256];
for r in 0..R {
// SAFETY: c has R*nb bytes; l has 256 entries and each code is < 256.
unsafe { s[r] += *l.get_unchecked(*c.get_unchecked(r * nb + b) as usize) };
}
}
// ... keep the top k
}
At 184 bytes per image, end-to-end top-5 agreement fell to 0.742. Two changes won it back:
- A 3-bit token embedding table saved about 3 MB, which raised the catalog to 212 bytes per image.
- A catalog-aware fine-tune. The student ranks the compressed catalog while the teacher ranks the full-precision one, so the encoder learns to compensate for the compression. This took top-5 from 0.740 to 0.812.
That shipped as Miranda v5: 81.4% top-5 agreement with full CLIP, label precision 0.402 against the original’s 0.398, and about 60 ms per search on Functions.
The component was 52,239,588 bytes, 189 KB below the limit. There was no room left for an image encoder.
Part 3: Measure the image encoder before training it
Benchmark with random weights first
Before training anything, I wanted to know which image models could run fast enough. Speed doesn’t depend on the weight values, only on the shapes. So I built a throwaway app with random weights at the real shapes, using the same kernels, and measured it on Functions. That took an afternoon, against days for training:
| Model | Weights | In-function time (p50) |
|---|---|---|
| ViT-Ti/32 (192 dims, 12 layers) | int8 | 50.8 ms |
| ViT-S/32 (384 dims, 12 layers) | int8 | 129.7 ms |
| ViT-B/32 (the real CLIP image encoder) | int8 | 374.0 ms |
| ViT-B/32 | 4-bit | 470.1 ms |
Two findings:
- The real image encoder was not an option. 374 ms per photo, and about 88 MB even at int8.
- 4-bit is slower than int8 for images. Unpacking nibbles costs more than the memory bandwidth it saves, because the image encoder processes many patches per weight.
So the plan became: train a small student image model to produce the same CLIP vectors as ViT-B/32, and make room for it in the app.
The idea that makes this work: if the student’s output lives in the same CLIP space as the catalog, the catalog stays as it is. Text queries, photo queries and the catalog share one search.
Part 4: Making room with 3-bit weights and a Hadamard rotation
The image student needed about 4–6 MB. The only place to find that was the text encoder.
I tried lower bit widths with the same QAT recipe (1,000 steps each):
| Weights | Before training (top-5 / top-1) | After 1,000 steps | Trend |
|---|---|---|---|
| 4-bit | 0.687 / 0.570 | 0.878 / 0.837 | flattening |
| 3-bit | 0.384 / 0.248 | 0.788 / 0.752 | still rising |
| 2-bit | 0.040 / 0.008 | 0.605 / 0.496 | stuck |
2-bit was a dead end for a 63M-parameter model. 3-bit was promising, but needed help.
The Hadamard rotation
Low-bit quantisation struggles with outliers. A few large values in a weight group force a large scale, and every other weight in that group gets rounded to almost nothing. Transformer activations have the same problem: in this model, some MLP input channels were up to 152× larger than the median.
A Hadamard rotation spreads outliers evenly across all channels. If R is orthogonal, then x · W = (x R)(Rᵀ W), so rotating both the input and the weights gives the same result in exact arithmetic. But the rotated values are much easier to quantise.
In training, each linear layer’s weights are stored pre-rotated and the input is rotated in the forward pass:
def rotation(k, seed):
"""Orthogonal R = diag(sign) @ H / sqrt(k), sign from torch.Generator().manual_seed(seed)."""
H = torch.ones(1, 1)
while H.shape[0] < k:
H = torch.cat([torch.cat([H, H], 1), torch.cat([H, -H], 1)], 0)
g = torch.Generator().manual_seed(seed)
sign = (torch.randint(0, 2, (k,), generator=g) * 2 - 1).float()
return (sign[:, None] * H / math.sqrt(k)).to(dev)
In Rust, there’s no need to multiply by a 2048 × 2048 matrix. The Hadamard transform has a fast O(k log k) butterfly form, like an FFT:
fn fwht(v: &mut [f32]) {
let mut h = 1;
while h < v.len() {
for i in (0..v.len()).step_by(2 * h) {
for j in i..i + h {
let (a, b) = (v[j], v[j + h]);
v[j] = a + b;
v[j + h] = a - b;
}
}
h *= 2;
}
}
Every linear layer input goes through sign flip → FWHT → 1/√k scaling, then int8 quantisation:
fn linear_in(lin: &LinearQ, x: &[f32], rows: usize, ag: usize) -> (Vec<i16>, Vec<f32>) {
let k = lin.k;
match lin.sign {
None => quantize_groups(x, rows, k, ag),
Some(sign) => {
let norm = 1.0 / (k as f32).sqrt();
let mut r = x[..rows * k].to_vec();
for row in r.chunks_exact_mut(k) {
for (v, &s) in row.iter_mut().zip(sign) {
*v *= s as f32;
}
fwht(row);
row.iter_mut().for_each(|v| *v *= norm);
}
quantize_groups(&r, rows, k, ag)
}
}
}
The same rotation is used for the 3-bit token table. Each row is stored rotated, and the function undoes the rotation when it reads a row:
// Undo the rotation: row = sign * FWHT(h) / sqrt(D)
fwht(&mut h);
let norm = 1.0 / (D as f32).sqrt();
for ((o, &v), &sg) in out.iter_mut().zip(&h).zip(self.sign) {
*o = v * norm * sg;
}
The trap: the random signs must match training exactly. They come from torch.Generator().manual_seed(1000 + 10·block + j), so the export script writes the sign vectors into the model file rather than regenerating them in Rust. One wrong sign would give a different model with no error message. The golden check would catch it.
3-bit weights in bit planes
3 bits don’t divide a byte evenly. I stored each group of 64 weights as two planes: a 16-byte plane holding the low 2 bits, and an 8-byte plane holding the high bit. That gives 24 bytes per 64 weights. The SIMD kernel unpacks them with masks, shifts and a byte swizzle:
/// 64 weights (24 bytes) -> 8 x i16x8 in input order.
unsafe fn unpack(p: *const u8) -> [v128; 8] {
let lo = v128_load(p as *const v128);
let hi = v128_load64_zero(p.add(16) as *const u64);
let three = u8x16_splat(3);
let four = u8x16_splat(4);
let bit = u8x16(1, 2, 4, 8, 16, 32, 64, 128, 1, 2, 4, 8, 16, 32, 64, 128);
let ls = [v128_and(lo, three), v128_and(u8x16_shr(lo, 2), three),
v128_and(u8x16_shr(lo, 4), three), u8x16_shr(lo, 6)];
// ... for each quarter: select the high-bit byte, test the bit, combine
let hb = v128_and(i8x16_eq(v128_and(i8x16_swizzle(hi, sel[t]), bit), bit), four);
let v = i8x16_sub(v128_or(ls[t], hb), four); // 0..7 -> -4..3
// ... widen to i16 for i32x4_dot_i16x8
}
The result
After the full run and the catalog-aware fine-tune:
| Text encoder | Top-5 same as full CLIP | Top-1 same | Label precision@5 |
|---|---|---|---|
| Original (fp32) | 100% | 100% | 0.398 |
| version with (4-bit) | 81.4% | 77.0% | 0.402 |
| version with (3-bit + rotation) | 80.3% | 74.2% | 0.405 |
About one point of top-5 lost. The text encoder file is now 25.5 MB, and 4.8 MB was freed for the image encoder. Encoding got slower (15.4 ms instead of 11.1 ms in wasmtime) because of the extra unpacking and the FWHT. That was a trade I was happy to make.
Part 5: The image student, and a dead end
Dead end: a small ViT trained from scratch
My first student was a ViT-Ti/32: 6 million parameters, the same architecture family as the teacher, and Phase 0 had shown it would run in about 50 ms.
I trained it for 30 epochs on 505,000 images: 405,000 Open Images photos (CC BY 2.0) plus the catalog. I already had the teacher’s embeddings for all of them, so no extra GPU time went to the teacher.
It failed:
| Student | Top-5 agreement | Cosine to teacher | Find this item (first) |
|---|---|---|---|
| ViT-Ti/32 from scratch, 30 epochs | 0.133 | 0.509 | 59.6% |
Training cosine was also only about 0.54, so this was underfitting, not overfitting. ViTs are known to need a lot of data. Half a million images is not enough to learn vision from scratch.
Lesson: a student doesn’t have to share the teacher’s architecture. It has to reproduce the teacher’s output space. Start from something that already knows how to see.
Distil into a different, pretrained architecture
I switched to ImageNet-pretrained CNNs from timm (Apache-2.0), with a linear projection to CLIP’s 512 dimensions: MobileNetV3-Large and EfficientNet-B0.
One more hurdle: at a learning rate of 1e-3, the pretrained features collapsed to zero within a few hundred steps. At 3e-4 they trained well. Pretrained backbones need a gentler start than a model trained from scratch.
The loss is the same idea as for the text encoder: cosine to the teacher, plus a KL divergence over how the teacher ranks the whole 100k catalog:
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
t = F.normalize(teacher(x).float(), dim=-1)
with torch.autocast("cuda", dtype=torch.bfloat16):
s = model(down(x))
s = F.normalize(s.float(), dim=-1)
l_cos = (1 - (s * t).sum(1)).mean()
l_kl = F.kl_div(F.log_softmax(s @ Tc.T / args.tau, 1), F.log_softmax(t @ Tc.T / args.tau, 1),
log_target=True, reduction="batchmean")
loss = l_cos + args.kl * l_kl
Tc is the catalog matrix (100,000 × 512) kept on the GPU, so every training step scores the whole catalog for every image in the batch.
How to measure an image student
Exact top-5 agreement with the teacher turned out to be the wrong metric for images. Many catalog items are near-ties, and no small student will reproduce the teacher’s order among them. So I used two tests that don’t depend on the teacher:
- Similar items: 2,000 fashion photos not in the catalog. Do the top 5 results share a garment label (dress, jacket, bag…) with the photo? (label precision@5)
- Find this item: 500 catalog images, held out of training, re-cropped, resized, brightness-shifted and re-saved as low-quality JPEG. Does the original come back first, or in the top 5?
After 10 epochs:
| Student | Label precision@5 | Find this item @1 | @5 |
|---|---|---|---|
| ViT-B/32 teacher | 0.483 | 97.2% | 99.6% |
| ViT-Ti/32 from scratch | 0.391 | 59.6% | 79.0% |
| MobileNetV3-Large (pretrained) | 0.468 | 82.8% | 93.8% |
| EfficientNet-B0 (pretrained) | 0.482 | 85.6% | 95.6% |
On relevance, the pretrained CNNs were already close to the teacher.
Part 6: The preprocessing trap
This was the most surprising hurdle of the project.
While building the evaluation sets, I first resized images with a plain bilinear resize and a simple centre crop. Then I fed those pixels to the real ViT-B/32 image encoder and compared its results with its own results on OpenCLIP’s preprocessing.
| Preprocessing | Real ViT-B/32 vs itself (top-5 / top-1) | Cosine |
|---|---|---|
| Bilinear resize, floor crop | 0.776 / 0.735 | 0.982 |
| OpenCLIP’s exact transform | 0.997 / 0.998 | ~1.0 |
The same model lost 22 points of top-5 agreement with itself just from a slightly different resize. The retrieval metric is steep: a cosine of 0.98 sounds close, but with 100,000 near-tied catalog items it reorders the top results.
A bit-exact port of Pillow’s resize
So the Rust side had to reproduce OpenCLIP’s preprocessing exactly. That means Pillow’s antialiased bicubic resize: separable passes, a = −0.5, the filter support scaled by the downscale factor, fixed-point coefficients with 22 fractional bits, and rounding to u8 between the horizontal and vertical passes.
I ported it from Pillow’s Resample.c:
const PRECISION_BITS: u32 = 32 - 8 - 2;
fn bicubic_filter(x: f64) -> f64 {
const A: f64 = -0.5;
let x = x.abs();
if x < 1.0 {
((A + 2.0) * x - (A + 3.0)) * x * x + 1.0
} else if x < 2.0 {
(((x - 5.0) * x + 8.0) * x - 4.0) * A
} else {
0.0
}
}
fn coeffs(in_size: usize, out_size: usize) -> (usize, Vec<(usize, usize)>, Vec<i32>) {
let scale = in_size as f64 / out_size as f64;
let filterscale = scale.max(1.0);
let support = 2.0 * filterscale;
// ...
for xx in 0..out_size {
let center = (xx as f64 + 0.5) * scale;
// C casts truncate toward zero; center - support + 0.5 can be negative
let xmin = ((center - support + 0.5) as i64).max(0) as usize;
// ... normalise in f64, then round to fixed point the way C does
kk[xx * ksize + x] = if w < 0.0 { (-0.5 + f) as i32 } else { (0.5 + f) as i32 };
}
}
The details that matter are the C-specific ones. Casts truncate toward zero, negative weights round differently from positive ones, and a pass is skipped when that dimension doesn’t change. With all of them in place, the Rust resize matches Pillow bit for bit on the same input pixels. The crop offsets also follow torchvision exactly: round((n − 224) / 2), with half rounding to even.
The part I couldn’t make exact: JPEG decoding
Rust decodes JPEGs with zune-jpeg. Pillow uses libjpeg-turbo. Both are correct, but they differ by a few levels per pixel (up to 4–13 on some images), because JPEG’s inverse DCT and chroma upsampling allow small differences between implementations.
That meant the golden check needed two thresholds:
- Encoder alone, fed Pillow’s exact crop: cosine ≥ 0.99 against the Python model.
- End to end, from the JPEG bytes: cosine ≥ 0.98, allowing for the decoder difference.
Lesson: for a distilled vision model, preprocessing is part of the model. Measure the teacher against itself under your preprocessing before you blame the student.
Part 7: A CNN engine in Rust, and choosing 160 px
The text kernels were for transformers. A CNN needed new ops: dense convolution for the stem, 1×1 (pointwise) convolution, depthwise convolution, squeeze-and-excite blocks, and hardswish/SiLU activations.
As in my BERT port, I didn’t ship a runtime. What’s new is how a CNN gets reduced to a few ops the Rust engine can run fast.
An op list with folded BatchNorm and padded channels
The export script walks the timm model and writes a flat op list. Three things happen on the way:
- BatchNorm is folded into the convolution before it. At inference time BatchNorm is a fixed per-channel scale and shift, so it can be multiplied into the conv’s weights and bias. The BatchNorm layers disappear.
- Channels are padded to multiples of 16, so every SIMD loop runs on full vectors with no remainder handling. The padded weights and biases are zero, so the padded channels stay zero.
- Every matrix op becomes one int8 GEMM. The 1×1 convolutions, the squeeze-and-excite layers, the projection and the stem (via im2col) all use the int8 kernel from the text encoder.
def fold(conv, bn):
"""Conv weight (out, in/groups, k, k) and BN -> folded weight, bias (float64)."""
w = conv.weight.detach().double()
b = conv.bias.detach().double() if conv.bias is not None else torch.zeros(w.shape[0], dtype=torch.float64)
if bn is not None and hasattr(bn, "running_var"): # qsim-folded models carry only the activation
g = bn.weight.detach().double() / torch.sqrt(bn.running_var.double() + bn.eps)
w = w * g[:, None, None, None]
b = (b - bn.running_mean.double()) * g + bn.bias.detach().double()
return w, b
The resulting format:
"CNN1" u32 n_ops u32 out_dim u32 res | f32 mean[3] std[3] | n_ops x op 1 DENSE [k, stride, pad, cin, cout, act, kk]: kxk conv via im2col 2 PW [cin, cout, act]: 1x1 conv 3 DW [k, stride, pad, c, act]: depthwise 4 SE [c, cr, act, gate]: squeeze-excite 5 SAVE push the activation (residual) 6 ADD pop and add 7 POOL global average 8 FC [cin, out]: final projection mat(n x k): i8 w[n*k] | f32 scale[n] | f32 bias[n]
The Rust engine is a loop over this list, with the model compiled into the binary.
A register-blocked depthwise kernel
Depthwise convolutions have only k × k weights per channel, too little work for a GEMM, so they stay in f32 with their own kernel. Thanks to the channel padding, it processes 16 channels at a time and keeps those 16 accumulators in SIMD registers across all taps. The valid tap range at the image border is computed once per output row and column, so the inner loops have no bounds checks:
for cb in (0..c).step_by(16) {
let mut acc = [0f32; 16];
acc.copy_from_slice(&bias[cb..cb + 16]);
for ky in ky0..ky1 {
let iy = oy * s + ky - p;
for kx in kx0..kx1 {
let ix = ox * s + kx - p;
let xi = &x[(iy * w + ix) * c + cb..][..16];
let wk = &wt[(ky * k + kx) * c + cb..][..16];
mac16(&mut acc, xi, wk);
}
}
out[ob + cb..ob + cb + 16].copy_from_slice(&acc);
}
MobileNetV3’s activations also help: hardswish and hard sigmoid are just clamps and multiplies, with no exp.
On Functions, with random weights at the real shapes:
| Model (no conv_head) | 224 px | 192 px | 160 px |
|---|---|---|---|
| MobileNetV3-Large | 69.7 ms | 51.4 ms | 36.3 ms |
| EfficientNet-B0 | 102.8 ms | 75.2 ms | 52.4 ms |
The SIMD depthwise kernel alone took MobileNetV3 from 80.7 to 69.7 ms at 224 px.
Drop the head to fit
As trained, MobileNetV3-Large has a 960 → 1280 conv_head before the projection, and it was too big for the 4.8 MB budget at int8. I retrained without it (--nohead, 960 → 512 projection): about 3.5M parameters, 3.9 MB.
Lower resolution, same quality
The input is the OpenCLIP 224 crop, resized again to the student’s resolution. I trained 5-epoch pilots at three sizes:
| Student | Functions time | Label precision@5 | Find this item @1 | @5 |
|---|---|---|---|---|
| MobileNetV3 @ 160 px | 36 ms | 0.461 | 85.4% | 95.6% |
| MobileNetV3 @ 192 px | 51 ms | 0.460 | 85.0% | 94.8% |
| MobileNetV3 @ 224 px | 70 ms | 0.470 | 81.2% | 94.2% |
| EfficientNet-B0 @ 160 px | 52 ms | 0.458 | 86.6% | 96.6% |
| ViT-B/32 teacher | 374 ms | 0.483 | 97.2% | 99.6% |
Going from 224 to 160 px cost no measurable quality and made the encoder 1.9× faster. I chose MobileNetV3-Large at 160 px: the fastest model, and it fits the app.
Part 8: int8 needs its own training too
EfficientNet-B0 survived int8 rounding without retraining (cosine ≥ 0.996 against the float model). MobileNetV3 didn’t: 0.969–0.992, below the 0.99 gate. MobileNets are known to be sensitive to quantisation, especially in the depthwise and squeeze-and-excite layers.
Simulate the CNN engine’s int8 maths in training
As for the text encoder in Part 2, I simulated the Rust arithmetic in training. qsim folds every BatchNorm into its convolution and fake-quantises each matrix op the way cnn-core runs it: int8 per pixel for inputs, int8 per output channel for weights, and depthwise in f32:
def q_rows(x, dim):
"""Symmetric int8 per slice along `dim` (max |x| -> 127)."""
s = x.abs().amax(dim, keepdim=True).clamp_min(1e-12).detach() / 127
return ste_round(x / s).clamp(-127, 127) * s
class QConv(nn.Conv2d):
"""Conv2d with int8 weights; matrix convs also quantise their input per pixel."""
def forward(self, x):
w = q_weight(self.weight) if self.groups == 1 else self.weight
if self.groups != 1:
return F.conv2d(x, w, self.bias, self.stride, self.padding, self.dilation, self.groups)
if self.kernel_size == (1, 1):
return F.conv2d(q_rows(x, 1), w, self.bias, self.stride, self.padding)
# dense kxk conv as Rust's im2col: rows are output pixels, (ky, kx, ci) per row
# ...
Even the stem convolution is simulated as Rust runs it, via unfold (im2col), so that the quantisation groups match.
Final training: 25 epochs, then a 3-epoch QAT fine-tune at a learning rate of 3e-5:
| Model | Label precision@5 | Find this item @1 | @5 |
|---|---|---|---|
| 25 epochs, fp32 | 0.462 | 85.0% | 94.2% |
| 25 epochs, int8 without QAT | 0.465 | 83.8% | 93.2% |
| 25 epochs + QAT, int8 (shipped) | 0.461 | 85.0% | 94.6% |
| ViT-B/32 teacher | 0.483 | 97.2% | 99.6% |
QAT recovered the int8 loss completely. The Rust golden check passed: encoder cosine ≥ 0.9935 on all 16 test images, and ≥ 0.987 end to end from the JPEG.
One surprise: the 25-epoch run got closer to the teacher (higher cosine) than the 5-epoch pilot, but retrieval didn’t improve. Cosine to the teacher is a proxy. What matters is the ranking, so measure that.
Part 9: One space, one catalog, and “More like this” for free
The catalog is stored in a PCA basis: the compression rotates the vectors so that the first dimensions carry the most information and get the most bits. The text encoder already had this rotation folded into its final projection, so its output is in catalog coordinates with no extra work per query.
Fold the catalog’s rotation into the last layer
I did the same for the image student. A rotation doesn’t change cosine similarity, so it can be multiplied into the last linear layer at export time. The query never needs a separate 512 × 512 rotation at run time:
w = model.proj.weight.detach().double()
if basis is not None: # output in catalog coordinates (rotation: cosines unchanged)
w = torch.from_numpy(basis) @ w
The basis is recomputed exactly as the catalog export computes it (SVD of the normalised catalog embeddings). The export checks it by decoding the compressed catalog in numpy (mean cosine 0.995 against the original vectors).
Now text vectors, photo vectors and catalog rows are all in the same coordinates, and one search function serves all three. That also gave me a third feature at no cost. “More like this” decodes a catalog row and uses it as the query:
fn similar(id: Option<usize>, k: usize, t_total: Instant) -> Response {
let cat = miranda_core::Catalog::load(&CATALOG_BLOB.0);
// ...
let mut v = miranda_core::row(&cat, id);
let n = v.iter().map(|x| x * x).sum::<f32>().sqrt();
v.iter_mut().for_each(|x| *x /= n);
let (res, search_ms, meta_ms, count) = results(&v, k, 0.0, Some(id));
// ...
}
It takes 31 ms on Functions, almost all of it the catalog scan.
Part 10: One component
Everything compiled in
The text encoder, the tokenizer, the image encoder, the catalog and the web page are all embedded with include_bytes! and read in place. The +simd128 flag and release profile are the same as in the BERT port, and the translation post shows what happens without the flag. The only addition is a 16-byte-aligned wrapper, because three different model files are now cast in place to &[f32], &[u16] and &[u64] slices:
static CATALOG_BLOB: &Aligned<[u8]> = &Aligned(*include_bytes!("../../assets/catalog6.bin"));
static CNN_BLOB: &cnn_core::Aligned<[u8]> = &cnn_core::Aligned(*include_bytes!("../../assets/cnn6.bin"));
The final component is 51,589,983 bytes, with 838,817 bytes to spare. That’s smaller than v5, which had no image encoder.
| Piece | Size |
|---|---|
| Text encoder (3-bit, with 3-bit token table) | 25.5 MB |
| Catalog (100,000 × 212 B + levels) | 21.2 MB |
| Image encoder (int8) | 3.9 MB |
| Tokenizer (merge pairs only) | 0.47 MB |
| Code and web page | ~0.5 MB |
Endpoints
GET /search?q=… routing → BPE tokenizer → ViT-B-32 text encoder (3-bit)
POST /image (JPEG) JPEG decode → OpenCLIP 224 crop → 160 px → MobileNetV3 (int8)
GET /similar?id=N the catalog's own vector for image N
↓ all three in catalog coordinates
LUT search over 100k × 212 B → top-k image URLs + credits
Hurdles from real photos
Testing with real uploads found one bug and one risk that the test set didn’t:
- A grayscale JPEG crashed the resize.
zune-jpegreturns one channel for grayscale images even when asked for RGB. - A very large photo could exceed the 128 MB per-request memory limit while decoding, before the code could check its size.
Both are now handled in decode, which reads the header before decoding any pixels:
pub fn decode(jpeg: &[u8]) -> Result<(usize, usize, Vec<u8>), String> {
// ...
dec.decode_headers().map_err(|e| format!("jpeg: {e:?}"))?;
let (w, h) = dec.dimensions().ok_or("jpeg: no dimensions")?;
if w == 0 || h == 0 {
return Err("jpeg: empty image".into());
}
if w * h > MAX_PIXELS {
return Err(format!("image too large: {w}x{h} (max {MAX_PIXELS} pixels)"));
}
let px = dec.decode().map_err(|e| format!("jpeg: {e:?}"))?;
// grayscale JPEGs come out as one channel despite the RGB output setting
match px.len() / (w * h) {
3 => Ok((w, h, px)),
1 => Ok((w, h, px.iter().flat_map(|&v| [v, v, v]).collect())),
c => Err(format!("jpeg: unexpected {c} channels")),
}
}
The handler also rejects bodies over 2 MiB (413) and anything that doesn’t start with the JPEG marker FF D8 (415).
Let the browser do the cheap part
A phone photo can be 12 megapixels, but the encoder only looks at a 224 px centre crop. So the web page shrinks the photo before uploading it. It also applies the EXIF orientation, which the server would otherwise have to handle:
// Photo -> JPEG, shortest side at most 448 px (the encoder uses a 224 centre crop), EXIF orientation applied.
async function shrink(file) {
const bmp = await createImageBitmap(file, {imageOrientation: "from-image"});
const s = Math.min(1, 448 / Math.min(bmp.width, bmp.height));
const c = document.createElement("canvas");
// ... draw at scale s
return await new Promise(ok => c.toBlob(ok, "image/jpeg", 0.92));
}
This also means PNG, HEIC and WebP photos work on the page: the browser converts them to JPEG. Uploads are a few hundred KB and decode in 2–6 ms.
Every technique, at a glance
Size: getting three pieces into 50 MiB
| Technique | What it bought | Where |
|---|---|---|
| QAT + distillation down to 4-bit, then 3-bit weights | Text encoder from 252 MB (fp32) to 25.5 MB. At 3 bits, top-5 agreement is 0.38 without training and 0.80 after it | Parts 2, 4 |
| Hadamard rotation of weights and token table (FWHT in Rust) | Made 3-bit weights and a 3-bit table usable; freed 4.8 MB for the image encoder | Part 4 |
| 3-bit bit-plane packing (2-bit plane + 1-bit plane per 64 weights) | 3 bits per weight with a fast SIMD unpack | Part 4 |
Token table limited to ASCII-reachable tokens, with a u16 remap | 2,269 fewer rows | Part 1 |
| Tokenizer stored as merge pairs only (id = 512 + rank) | 1.8 MB → 0.47 MB | Part 1 |
| PCA + per-dimension bit allocation + Lloyd-Max levels + byte packing | Catalog from 2,048 to 212 bytes per image | Part 2 |
| Catalog-aware fine-tune of the text encoder | Won back 7 points of top-5 lost to catalog compression | Part 2 |
| Distilling CLIP’s image encoder into a pretrained MobileNetV3 | Image encoder from about 88 MB (int8) to 3.9 MB | Part 5 |
Dropping the CNN’s conv_head | Made the student fit the 4.8 MB budget | Part 7 |
| BN folding into convolutions | No BatchNorm layers at run time | Part 7 |
| int8 QAT that simulates the CNN engine | int8 image model with no quality loss | Part 8 |
include_bytes!, flat binaries read in place | No parse step, no copies (reused: BERT port) | Part 10 |
Latency: keeping each request under 100 ms
| Technique | What it bought | Where |
|---|---|---|
Running only the real tokens (causal mask + <eot> pooling) | 7.1 s → ~0.5 s per text query, identical output | Part 1 |
<eot> row only in the last block | 15.3 → 12.8 ms in wasmtime (reused idea: 1 s → 260 ms) | Part 1 |
| int8 SIMD GEMM with a 4 × 2 register tile | ~0.5 s → 24 ms (reused: 1 s → 260 ms) | Part 1 |
| Per-group (128) activation scales | Top-5 0.905 → 0.94+ at no speed cost | Part 2 |
| Lookup-table catalog search, 8 rows at a time | 100k images scored in ~30 ms with no decompression | Part 2 |
| Random-weights benchmark before training | Ruled out the ViT-B/32 image encoder (374 ms) and 4-bit image weights | Part 3 |
| Register-blocked SIMD depthwise convolution, channels padded to 16 | MobileNetV3 80.7 → 69.7 ms at 224 px | Part 7 |
| 160 px student input instead of 224 | 70 → 36 ms with the same quality | Part 7 |
| Catalog PCA rotation folded into both encoders’ last layer | One search path for text, photos and “More like this”; no extra rotation per query | Part 9 |
| Header check before decoding, browser-side shrink to 448 px | Uploads of a few hundred KB, decode in 2–6 ms, no memory blow-ups | Part 10 |
Quality: matching the original
| Technique | What it bought | Where |
|---|---|---|
| KL loss over the teacher’s ranking of the whole catalog | The student learns the order of results, which is what search uses | Parts 2, 5 |
| Bit-exact port of Pillow’s bicubic resize | Avoided a 22-point top-5 loss from preprocessing alone | Part 6 |
| Two-threshold golden check (encoder ≥ 0.99, end to end ≥ 0.98) | Caught real errors without failing on JPEG decoder differences | Part 6 |
The full picture
Quality against the original server setup
Text search, 252 retrieval queries, same 100k catalog:
| Pipeline | Top-5 same as original | Top-1 same | Label precision@5 |
|---|---|---|---|
| Original (full-precision CLIP, uncompressed catalog) | 100% | 100% | 0.398 |
| Lookup table (first edge version) | 52.3% | 41.3% | 0.416 |
| Final Version on Functions | 80.3% | 74.2% | 0.405 |
Photo search:
| Image encoder | Label precision@5 | Find this item @1 | @5 | Time per photo |
|---|---|---|---|---|
| ViT-B/32 (the catalog’s own encoder) | 0.483 | 97.2% | 99.6% | 374 ms (encoder only, on Functions) |
| On Functions (MobileNetV3, int8, compressed catalog, over HTTP) | 0.462 | 82.2% | 93.6% | 82 ms (whole request) |
Text search relevance is at least as good as the original. Photo search relevance is within about 2 points of the full ViT-B/32 image encoder. “Find this exact item” is where the gap shows: the original comes back first 82% of the time, against 97% for the teacher.
Speed on Akamai Functions
Median of 12 requests, measured from India:
| Request | Inside the Function | Round trip |
|---|---|---|
| Photo search | 82 ms (decode 2–6, resize 9–13, encode 34–40, search 29–31) | 167 ms |
| Text search | 58 ms | 124 ms |
| More like this | 31 ms |
What didn’t work
| Idea | Why it failed |
|---|---|
| Running all 77 token positions | Padding after <eot> can’t affect a causal model’s output. It was 15× wasted work |
| 4-bit weights without retraining | Top-5 agreement dropped to 0.50–0.67 |
| 2-bit text encoder | Plateaued at 0.60 top-5, even with QAT |
| Shrinking the catalog instead of the encoder | At 184 B/image, top-5 fell to 0.74. Bits in the catalog were worth more than bits in the encoder |
| 4-bit image encoder | Slower than int8 on Functions: unpacking costs more than it saves |
| The real ViT-B/32 image encoder | 374 ms per photo and about 88 MB at int8 |
| A ViT-Ti/32 student from scratch | Underfit badly (cosine 0.51) on half a million images |
| A learning rate of 1e-3 for pretrained CNNs | The features collapsed to zero |
| Bilinear resize “close enough” | Cost the real ViT-B/32 22 points of top-5 agreement with itself |
| Training longer for a higher cosine | 25 epochs raised the cosine to the teacher, but not retrieval quality |
Lessons
- Measure the platform first. A random-weights benchmark ruled out the real image encoder and 4-bit image weights before any training time was spent.
- Exploit the model’s structure. The causal mask and
<eot>pooling cut text encoding by more than 10× with an identical result. - Quantisation is free only with training. Simulate the exact integer arithmetic of your runtime in the forward pass, and distil against the full-precision model.
- Rotate before you quantise. A Hadamard rotation spreads outliers across all channels, and a fast Walsh-Hadamard transform makes it cheap at run time.
- Train for the ranking, not the vector. A KL loss over how the teacher ranks the catalog matters more than cosine to the teacher.
- A student doesn’t need the teacher’s architecture. It needs the teacher’s output space. A pretrained CNN beat a ViT trained from scratch by a wide margin.
- Preprocessing is part of the model. Port it bit-exactly, and measure the teacher against itself under your preprocessing.
- Fold fixed linear maps into the weights. The catalog’s PCA rotation costs nothing at run time once it’s inside the last layer.
- Spend the byte budget where it buys the most quality. Every byte was traded between the text encoder, the token table, the catalog and the image encoder by measuring end-to-end quality, not per-component accuracy.
- Keep a golden check with honest thresholds. Two thresholds, one for the encoder and one end to end, made the JPEG decoder difference visible instead of hiding it.
What’s next
- A catalog-aware fine-tune for the image student. The text encoder gained 7 points of top-5 from this. For images, compression costs about 2.4 points of “find this item” @1.
- Faster catalog search. At about 30 ms, scanning 100k compressed rows is now a third of a photo search.
- A SIMD Walsh-Hadamard transform and a cheaper 3-bit unpack, to win back the few milliseconds the 3-bit text encoder added.
For now, one 51.6 MB WebAssembly component on Akamai Functions runs CLIP’s full text encoder, a photo encoder trained to match CLIP’s image encoder, and search over 100,000 images. It answers a text query in about 60 ms and a photo in about 80 ms, with no GPU and no calls to any other service.
Written By

I’m an Enterprise Architect at Akamai Technologies with 15+ years across mobile engineering, edge infrastructure, security, and AI systems. Having launched 45+ apps on the App Store and Play Store (iOS, Android, Flutter, React Native), I specialize in mobile SDK internals, Frida-based security, and high-concurrency edge runtimes like Akamai EdgeWorkers, Akamai Functions and HarperDB. My current focus is optimizing small language models and porting them to constrained, CPU-only runtimes like Akamai Functions, alongside Agentic AI frameworks (LangGraph, MCP), WASM-based AI guardrails, self-hosted LLM inference, and real-time voice pipelines.