Harvey Emerges as a New AI Decacorn After Valuation Surpasses $15 Billion

The Hook

In the same 24 hours, Harvey reached a landmark valuation milestone, cementing its place as a legal-AI leader — a valuation signal that vertical AI applications are scaling hard, and every app at that scale runs on an inference backend that someone built and optimized. Today we give you the framework, the 20-minute hands-on check, and a prompt template to evaluate whether EPD disaggregation belongs in your stack.

One Tip

Today's cloud-AI skill: Encode-Prefill-Decode (EPD) Disaggregation

Most cloud engineers hit multimodal serving when a product team asks: 'Can we add image inputs to our chatbot?' You swap the model, redeploy the endpoint, and immediately notice requests with images running significantly slower than text-only requests on the same instance. That's not a bug — it's the architecture working as designed. EPD disaggregation is the fix.

The three stages, glossed:

• Encode: the vision encoder converts an image into a dense vector embedding — a numerical representation the language model can read. Compute-heavy, short in duration.
• Prefill: the model processes the full input context (your text prompt plus the image embedding) and builds a KV cache (key-value cache — a saved internal state reused during generation). Memory-bandwidth intensive.
• Decode: using the KV cache, the model generates output tokens one at a time. Latency-sensitive and iterates many times per request.

On a standard SageMaker or Bedrock endpoint, all three stages share the same GPU memory bus. A slow encode blocks the prefill queue. An oversized prefill starves the decode of cache bandwidth. At high concurrency, these queuing effects compound and throughput collapses even when your GPU utilization reads high.

What disaggregation does: it routes each stage to a dedicated worker pool — encode workers handle only vision processing, prefill workers handle only context loading, decode workers handle only generation. Each pool autoscales independently.

NVIDIA's EPD technique is available in production through NVIDIA NIM (NVIDIA Inference Microservices — a catalog of optimized model containers deployable on any cloud). If you're not building your own serving stack, checking whether your target model is available as a NIM container is the fastest path to EPD-style optimization without building the disaggregated infrastructure from scratch.

This framework matters now because Harvey's round reflects the growing ecosystem of apps built on multimodal APIs — and their engineering teams are about to hit these same serving bottlenecks. Understanding this architecture puts you a step ahead of that wave.

NVIDIA's go/no-go signals (simplified):

1. Your model takes images, video, or audio as input — not text only.
2. You're handling significant concurrent request volume at peak.
3. GPU utilization is high but tokens-per-second is still hitting a ceiling.
4. Profiling shows encode or prefill time substantially exceeds your decode time per request.

If fewer than two apply, start with speculative decoding instead — built into SageMaker's TGI container and Bedrock's inference endpoints, zero architecture change, 20–30% decode latency recovery for free.

Hands-on exercise (20 minutes):

1. Open SageMaker JumpStart in your AWS console and deploy a Llama-3.2-11B-Vision endpoint on an ml.g5.2xlarge.
2. Send 20 test requests via the built-in console — 10 with an image attachment plus a question, 10 with the same question but no image.
3. Open CloudWatch → Metrics → SageMaker/Endpoints, select your endpoint, and plot ModelLatency for both batches side by side.
4. Calculate the image-to-text latency ratio. An elevated ratio indicates a measurable encode bottleneck.
5. Record this number — it is the input to your EPD go/no-go decision as traffic grows.

You'll know it worked when: you can pull a CloudWatch metric and state, with a specific number, which stage is your bottleneck. Most engineers running multimodal endpoints can't do that today. After this exercise, you will.

One Prompt

Use this prompt to get a structured EPD evaluation for your current setup. Fill in the brackets before pasting into any AI assistant.

I'm running [MODEL_NAME — e.g. Llama-3.2-11B-Vision, Pixtral-12B, LLaVA-1.6] on [CLOUD PLATFORM + INSTANCE — e.g. AWS SageMaker ml.g5.12xlarge].

My current serving metrics:
- Input types: [text-only / image+text / video+text]
- Peak concurrent requests: [NUMBER]
- Average latency: [X ms], P99: [X ms]
- GPU utilization at peak: [X%]
- Image-to-text latency ratio: [X:1, or 'not yet measured']

Using NVIDIA's EPD (encode-prefill-decode) disaggregation framework as context:
1. Based on these metrics, is EPD disaggregation the right optimization, or should I start with speculative decoding on my existing endpoint?
2. If EPD is warranted, sketch the three hardware pools I would need and the autoscaling logic for each at my scale.
3. What three metrics should I track in CloudWatch or Prometheus to confirm a throughput improvement after the change?

Give me a go/no-go recommendation with specific reasoning.

The output becomes your technical brief for the next infrastructure conversation with your team — hand it to your manager or bring it to a design review.

Sources

1. Harvey Emerges as a New AI Decacorn After Valuation Surpasses $15 Billion — entARABI
2. When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving — developer.nvidia.com
3. ChatGPT Business: Invitation stays pending after successful login — “Content failed to load” — community.openai.com
4. SegKAN: High-Resolution Medical Image Segmentation with Long-Distance Dependencies — arxiv.org
5. Datasette 1.0a39 and 0.65.4 security releases — simonwillison.net
6. Ai Joy. To make artificial intelligence accessible to ordinary people — community.openai.com
7. WSSU’s first AI summit explores technology’s promise as researchers warn of future risks — abc45.com
8. Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer — developer.nvidia.com