For CTOs & heads of platform
Open-weight LLMs, served on your own iron
Run the models on infrastructure you own. No token egress, no per-call vendor metering, no opaque model swaps under you. You hold the weights, the GPUs, and the SLA.
We stand up production inference on open-weight models — Llama 3.x, Mistral / Mixtral, Qwen 2.5, Gemma — served with vLLM for high-throughput batched workloads or Ollama for low-friction single-node deployments. On-prem, in your VPC, or fully air-gapped. The stack is boring on purpose: OpenAI-compatible endpoints so your existing clients point at a new base URL and nothing else changes.
The serving stack
Opinionated defaults, tuned to your latency and throughput envelope:
Inference & serving
vLLM with PagedAttention and continuous batching for throughput; tensor parallelism across GPUs for models that won't fit on one card. Ollama where a single-box, llama.cpp-backed deploy is the right tool. AWQ / GPTQ / GGUF quantisation when VRAM is the constraint.
Hardware & sizing
We size GPUs to the workload — an 8B class model serves comfortably on a single 24GB card (RTX 4090 / L4); 70B-class wants 2× A100 80GB or 4× L40S with tensor parallelism. We model concurrency, context length, and KV-cache footprint before you buy a thing.
Throughput & latency
Expect tens of tokens/sec/request and thousands of tokens/sec aggregate on a well-fed node, sub-second time-to-first-token under realistic concurrency. We benchmark on your prompts, not a synthetic suite, and tune batch size, max-num-seqs, and KV-cache block allocation to your p95 target.
Integration & operations
A model server is the easy 20%. The other 80% is making it a dependable platform service:
Drop-in API surface
OpenAI-compatible /v1/chat/completions and /v1/embeddings, streaming over SSE. Existing SDKs and tools — LangChain, LlamaIndex, your own client — repoint by changing the base URL. RAG wiring against pgvector, Qdrant, or your vector store of choice.
Runs where you run
Containerised, GPU-scheduled on Kubernetes or plain systemd on bare metal. Horizontal scale behind a load balancer, model-weight caching, health and readiness probes, Prometheus metrics, and Grafana dashboards for tokens/sec, queue depth, and GPU utilisation.
Air-gapped by default if needed
Fully offline installs: mirrored weights, pinned images, no outbound egress. Reproducible from an artifact bundle so an isolated environment rebuilds without ever touching the public internet.
How we engage
Architecture review and a workload-driven sizing model up front, a benchmarked proof-of-concept on your prompts and your hardware, then a hardened production rollout with monitoring, runbooks, and an upgrade path as new open weights land. We can operate it for you or transfer it to your platform team with the docs to own it.
Have a workload and a latency budget?
Talk architecture