From Server Bottleneck to 100% Edge: Building a Serverless Translation Pipeline on Akamai Functions

How I built a two-stage translation pipeline that converts both Romanized text and native Indian scripts to English, why half of it initially lived on a Linode, and how replacing a 200M-parameter model with six browser-sized WebAssembly models cut latency in half and eliminated the server.

The Problem: Nobody Types in Devanagari

Customer support chat in India rarely arrives in formal, native script. Hundreds of millions of people type their native languages using the Latin alphabet, often mixed with English:

mera order abhi tak nahi aaya, please check karo!
payment fail ho gaya, refund kab milega?
enakku refund venum, I am waiting for two weeks

Standard translation engines are trained on native scripts (हिन्दी, தமிழ், తెలుగు). Handed Romanized text, they either echo the characters verbatim or produce nonsensical phonetic guesses. To build a robust service that supports both native Indic script and Romanized text seamlessly, the architecture relies on a dynamic two-step pipeline. If a user types in native script, it skips transliteration and goes straight to translation. But for Romanized inputs, the pipeline intervenes:

1. Transliterate (if needed): Convert phonetic Roman text to native script (mera order → मेरा ऑर्डर).
2. Translate: Convert native script into English (“My order hasn’t arrived yet”).

AI4Bharat publishes open-source state-of-the-art models for both stages:

Step	Model	Parameters & Weight Footprint
Romanized → Native	IndicXlit (fairseq transformer over characters, 21 languages)	11.5M parameters (~46 MB fp32, ~12.1 MiB int8)
Native → English	IndicTrans2 (indic-en-dist-200M)	200M parameters (~221 MB as CTranslate2 int8)

My target deployment environment was Akamai Functions (built on Spin / WebAssembly). But Akamai Functions enforce strict execution guardrails:

• App binary size: 50 MiB
• Memory ceiling: 128 MiB
• Execution budget: 30 seconds limit 
• Instance lifecycle: Fresh instance per request (stateless cold starts)

IndicXlit (11.5M parameters) fits neatly within these constraints. IndicTrans2 (200M parameters, 221 MB even at int8) never will.

This post details the evolution of this system: starting with a hybrid architecture (edge transliteration calling an Linode), hitting the operational and latency ceilings of that hybrid setup, and ultimately eliminating the server entirely by orchestrating six tiny WebAssembly-based Firefox Translations models across chained Akamai Functions.

Part 1: The Hybrid Architecture (Akamai Functions + Linode)

The initial design split the pipeline cleanly across the size boundary: what fits runs on the edge; what doesn’t runs on the smallest dedicated server that can host it.

Edge Transliteration: Zero-Copy Weights and Wasm SIMD128

In my previous posts on porting and optimizing BERT models in WebAssembly, I established a core rule for Akamai Functions: when every request is an isolated cold start, runtime model loading must cost zero milliseconds.

I verified the Akamai runtime behavior with a ten-line atomic probe:

static SEEN: AtomicU64 = AtomicU64::new(0);
// …
let seen = SEEN.fetch_add(1, Ordering::SeqCst) + 1; // Always returns 1

Because every request gets a fresh Wasm instance, frameworks like tract-onnx that parse and deserialize model graphs on startup impose a 1.2-second tax on every single invocation.

To bypass this entirely, I used the same zero-copy approach detailed in my earlier work: embedding the raw model blob directly into the Wasm linear memory data segment using include_bytes! and slicing pointers into it at runtime.

pub static BLOB: &[u8] = include_bytes!(“../../artifacts/xlit_q8.bin”);
pub static DICTS: [(&str, &[u8]); 6] = [
    (“hi”, include_bytes!(“../../artifacts/hi_rescore_100k.bin”)),
    // ta, te, kn, ml, mr
];
// Model::load(BLOB) slices the blob into per-layer &[i8] views; nothing is copied or parsed.

• Startup time: Dropped from 1,187 ms (tract-onnx) to 0.27 ms locally and 2 ms on Akamai.
• Quantization scheme: Symmetric int8 weights, per-row dynamic activation quantization, and exact i32 accumulation via Wasm SIMD128 dot-product intrinsics (i32x4_dot_i16x8).
• Memory footprint: Embedded weights live in the linear data segment: 12.1 MiB of model plus ~15 MiB of frequency dictionaries kept idle memory at 28.7 MiB, well within the 128 MiB boundary.

Shrinking the 342 MB Dictionary to 2.2 MiB

The official IndicXlit Python pipeline relies on a word-frequency dictionary to rescore its beam search. Without rescoring, accuracy drops sharply from 87.3% to 75.7%. However, the official Hindi frequency dictionary is a 342 MB JSON file—nearly seven times the entire Akamai deployment limit.

I converted this dictionary into a zero-copy flat binary file with an offset table, sorted UTF-8 keys, and a u16 per entry storing probabilities, searched via in-place binary search:

Zero-copy memory layout: replacing bloated JSON structures with contiguous offset-indexed binary buffers.

pub fn prob(&self, w: &str) -> Option<f64> {
    let (mut lo, mut hi) = (0usize, self.n);
    while lo < hi {
        let mid = (lo + hi) / 2;
        match self.word(mid).cmp(w.as_bytes()) {
            Less => lo = mid + 1,
            Greater => hi = mid,
            Equal => {
                let q = u16::from_le_bytes([self.q[2 * mid], self.q[2 * mid + 1]]);
                return Some((-(q as f64) / 256.0).exp()); // -ln p in 1/256 steps
            }
        }
    }
    None
}

Evaluating different dictionary pruning thresholds on actual messy customer chat revealed a counterintuitive result:

Top-N Words Retained	Binary Size	Aksharantar Benchmark	Chat Test Set Word Accuracy
None (Raw model)	0 MB	58.93%	74.30%
25,000	0.51 MB	59.87%	89.01%
100,000 (Chosen)	2.23 MB	66.26%	87.24%
1,000,000	24.98 MB	67.83%	86.96%
Full (4.22M words)	125.8 MB	68.93%	86.96%

On chat-style text, smaller dictionaries score higher. Large lexical corpora contain rare and archaic spellings that outrank the common conversational terms people actually mean. Pruning to 100k entries compressed the dictionary by 99.3% while improving conversational accuracy over the unpruned baseline.

Handling Timeouts, Tokenization, and Edge KV Caching

• Word Cache vs. Sentence Cache: An edge key-value store for individual transliterated words was slower than the model itself (~26–29 ms per KV read vs. ~21 ms for model inference). I pivoted to a full sentence-level cache. Write overheads (~330 ms) were pushed after sending the HTTP response using Spin’s raw wasi:http handler.
• Enforcing Deadlines: To prevent runaway 60-second inputs from triggering Akamai’s 30-second kill switch, transliteration halts processing at 20 seconds, returning remaining words untouched. The HTTP client calling Linode was implemented directly on wasi:http with a monotonic clock polling timer.

The Translation Server: Linode Dedicated 4 GB

On the Linode side (Dedicated 4 GB, 2 vCPU), IndicTrans2 was deployed using CTranslate2 int8 behind a lightweight Python HTTP server fronted by Caddy:

MODEL = Model(str(CKPT), device="cpu", model_type="ctranslate2") # IndicTrans2 CT2 int8
LOCK = threading.Lock() # One translation at a time; CT2 uses internal multithreading
# ...
with LOCK:
    parts = chunks(text) if len(text) > CHUNK_CHARS else [text] # Avoid silent 160-token truncation
    out = " ".join(MODEL.batch_translate(parts, src, tgt))

The CTranslate2 int8 model consumed only 368 MiB of RAM and matched quality closely. But living with this hybrid architecture in production exposed significant operational and architectural bottlenecks.

The Hidden Costs of the Hybrid Split

1. The TLS Tax on Every Call: Because each request to an Akamai Function initializes a fresh Wasm instance, there is no persistent HTTP connection pool. ~150 ms of the total 360 ms translation step was pure network handshake overhead.
2. Server Concurrency Bottleneck: Edge functions scale horizontally on demand. A 2-vCPU server running a 200M parameter model does not. The Linode saturated at ~4–5 requests/second.
3. Operational Overhead: Maintaining an always-on Linux box and the inevitable need to manage multiple Linode servers and a load balancer to support production-level concurrent users along with managing Caddy TLS certs, rotating bearer tokens, and monitoring CPU health, undermined the core premise of going serverless.

Part 2: Eliminating the Server (All-in on Akamai Functions)

The Linode only existed because IndicTrans2 (200M parameters) was too large for an Akamai Function. Why was IndicTrans2 so large? Because it was a single multilingual model designed to translate 22 languages simultaneously.

In a serverless architecture, you don’t need one monolithic model. You can run one small model per language.

Mozilla’s Firefox Translations (the Bergamot project) solved this exact constraint for in-browser client-side translation. They trained compact, language-specific Marian models optimized to run on low-power CPUs:

Language	Architecture	Footprint (with shortlist)
hi, te, kn, ml	tiny (6-layer encoder, 2-layer SSRU decoder, d=256)	~17–23 MB
ta, mr	base-memory (6-layer encoder, 4-layer SSRU decoder, d=384)	~33–37 MB

The Hindi model was just 17.1 MB of int8 weights—roughly 13 times smaller than IndicTrans2, and well below Akamai’s 50 MiB bundle ceiling.

The Runtime Breakthrough: fxtranslate

Bergamot’s native C++ codebase is tightly bound to Emscripten and browser JavaScript APIs, making standard WASI compilation impossible. Instead, I found fxtranslate, a pure-Rust implementation of the Firefox Translations engine with support for Marian SSRU decoders, zero-copy byte loading, and pure-Rust Wasm SIMD128 int8 matrix multiplication kernels.

The Serverless Chained Architecture

Rather than bundling all language models into a single massive binary, each language translation model runs in its own isolated Akamai Function. Because the pipeline naturally accepts both formats, Native Indic text bypasses the IndicXlit processing step completely:

Crucially, Function 2 operates with zero external network access:

# Function 2 spin.toml
[component.translate]
allowed_outbound_hosts = [] # Completely locked down; zero egress permitted

Edge Engineering Hurdles & Edge-Case Gotchas

1. Indic Scripts Have No Capital Letters

On the IN22-Gen benchmark, the Rust translation Function initially dropped 0.9 chrF++ below the offline Python/C++ Bergamot reference. The cause was standard Unicode sentence segmentation (ICU UAX #29). Indic scripts do not have letter cases. Every Devanagari character following a full stop is classified as “not lowercase”. As a result, common initialisms split into fragmented sentences.

Replacing ICU with an Indic-aware segmenter that explicitly handles danda (।), punctuation, and language-specific honorifics closed the quality gap.

2. The SentencePiece Byte-Fallback Leak (₹499)

When a character is missing from a SentencePiece vocabulary, it decomposes into UTF-8 byte tokens. The decoder in fxtranslate 0.5.0 failed to recombine these pieces, causing pricing in support messages to return as literal debug strings. A post-processing scanner parses consecutive hex byte pieces and reconstructs them into valid UTF-8 strings before returning.

3. The 13× Performance Cliff: A Missing Compiler Flag

During testing, Function 1’s transliteration latency suddenly spiked to 2,650 ms for a 9-word sentence on Akamai. The issue was a missing target compilation flag in .cargo/config.toml:

[target.wasm32-wasip1]
rustflags = ["-C", "target-feature=+simd128"]

Recompiling with +simd128 dropped execution latency from 2,650 ms to ~200 ms—a 13× speedup without changing a single line of logic.

4. Translation Models Require Inverted Lexicons

In the hybrid setup, ambiguous colloquial phrases like rat bite were transliterated using a phrase table to Hindi wording (चूहे के काटने) because IndicTrans2 handled contextual semantic idioms best when presented in pure Hindi. Testing that same lexicon against the Firefox Translations model resulted in a complete regression: the Firefox model interpreted काटने literally as “cut”.

Different models prefer opposite lexical representations. Lexicons are not properties of a language; they are tightly coupled to the specific translation model architecture.

5. Clause Dropping in Dravidian Languages

In mixed code-switched inputs, the Tamil (ta), Telugu (te), Kannada (kn), and Malayalam (ml) models frequently dropped or duplicated the English clauses. To fix this, Function 1 inspects the input for Dravidian targets, splits mixed clauses onto distinct lines, routes each clause as an independent translation span, and reconstructs the sentence post-inference.

Final Performance & Architecture Comparison

Latency Breakdown (Romanized Hindi Chat)

Processing Stage	Hybrid Setup (Akamai + Linode)	Pure Edge Setup (Chained Akamai Functions)
Transliteration (IndicXlit Wasm)	~212 ms	~142 ms
Network Transport / Function Hop	~360 ms (HTTPS + TLS handshake)	~86 ms (Edge internal hop)
Translation Compute	176–224 ms (Linode 2 vCPU)	~21 ms (fxtranslate SIMD128)
Translation Model Startup	0 ms (Daemon running)	~47 ms (Zero-copy slice on cold boot)
Total Client Latency (p50)	492 ms	266 ms
Total Client Latency (p95)	703 ms	458 ms

By eliminating external TLS negotiation and replacing the monolithic translator with a lightweight SIMD-accelerated binary, p50 latency dropped by 46%.

Throughput and Scalability

• Hybrid Setup: Saturated at ~4–5 requests/sec.
• Pure Edge Setup: Handled 100 concurrent connections across all six languages with 0 errors, scaling horizontally across edge nodes.

Architectural Trade-Offs

System Metric	Akamai Function + Linode	Chained Akamai Functions (Edge Only)
End-to-End Latency (p50)	492 ms	266 ms
Throughput Ceiling	~4–5 req/s (Single point of failure)	100+ concurrent clients per language
Infrastructure Overhead	Linux OS, Caddy, systemd, Bearer secrets	Zero servers, zero long-lived credentials
Translator Attack Surface	Public-facing HTTPS server	Zero egress allowed
Quality (FLORES)	Baseline	-4.0 to -5.6 chrF++
Code-Mixing Handling	Unsolved for Dravidian languages	Resolved via automated clause splitting

Key Takeaways

1. Monolithic Models Force Server Deployments: Multilingual translation models are massive primarily because they support dozens of language pairs simultaneously. Splitting the problem into dedicated, language-specific models makes running transformer inference within strict edge constraints practical.
2. Eliminating Servers Removes Latency Multipliers: The primary latency tax in the hybrid setup wasn’t model compute; it was establishing new TLS connections from fresh edge instances to a fixed origin server. Chaining edge functions keeps traffic on internal backbones and cuts connection overhead.
3. Verify Wasm Compiler Intrinsics: Never assume SIMD is enabled by default. Missing -C target-feature=+simd128 caused a 13× performance degradation with zero runtime errors or warnings.
4. Lexicons Are Model-Dependent: Translation post-processing and transliteration phrase tables cannot be treated as static linguistic data. They are tightly coupled to how a specific model interprets vocabulary, subwords, and case.
5. Zero-Copy Architecture Remains Mandatory: Embedding raw int8 weights directly into linear memory and using in-place pointer slicing is the only viable way to run neural networks within a sub-millisecond cold-start budget.