Bundled vLLM
Srasta deploys a GPU-backed OpenAI-compatible vLLM runtime on the selected GPU host.
Inference & Model Routing
Route model requests through a reviewed deployment profile.
Srasta separates the operator decision from the runtime plumbing. Teams choose where inference runs and what each role and workflow may use; Srasta turns that into governed LiteLLM routes, certified vLLM or MLX runtime settings, model-access controls, and smoke tests.
Routing Path
One gateway, many possible inference backends.
A caller can ask for coding, business, or general.
Those are stable model aliases. In a published supported profile, the backend is the certified local runtime.
An external OpenAI-compatible endpoint requires separate integration and data-egress review.
Provider Classes
Operators choose the inference class before choosing the model.
The installer treats inference as a deployment decision. Local inference keeps prompts and responses inside the supported single-node deployment. An external endpoint is a separately scoped option that creates an explicit data-egress decision and needs integration validation.
Bundled MLX
Apple Silicon Community runtime served host-native through Srasta's certified MLX profile.
External self-hosted
Operator-provided vLLM, NVIDIA NIM, or generic OpenAI-compatible endpoint.
Hosted API
Provider APIs such as Anthropic, OpenAI, Hugging Face, Together, or Fireworks through LiteLLM.
Certified Routes
Stable routes let models evolve without changing caller behavior.
Primary governed inference
Routes to the certified chat model selected for the installed deployment profile.
Knowledge retrieval support
Routes embeddings through the certified engine-native path for the selected hardware profile.
Role-aware access
Checks the caller's role against model access policy before any backend receives the request.
Evidence on every route
Records allowed and denied model access so operators can prove what ran and why.
Recommendation Engine
Model fit is a function of deployment profile, hardware, volume, and constraints.
The installer does more than ask whether a model can load. It evaluates fit against the operator's environment: GPU and RAM, inference class, deployment intent, expected concurrency, latency and quality targets, and cost constraints for hosted providers.
Ranking dimensions
- Hardware fit and VRAM envelope.
- Latency targets such as TTFT and token throughput.
- Workflow-specific quality requirements.
- Throughput and peak concurrency fit.
- Hosted-provider cost constraints when relevant.
- Operational consistency across the selected deployment profile.
Runtime Contracts
Correct routing includes parser and embedding decisions.
Tool-call parser
vLLM agentic tool calls require the right parser for the model family. Srasta tracks this per model so tool calls do not silently degrade into plain text.
Reasoning parser
Models with separate thinking output can declare the matching reasoning parser where supported.
Embedding route
Embeddings use the certified engine-native OpenAI-compatible path for the selected hardware profile.
Certified route
Runtime routes resolve to catalog-certified model and runtime bindings rather than ad hoc fallback chains.
Governance
Routing does not bypass Srasta's security model.
Whether a model is local, self-hosted, or hosted, access still enters through the same governed Srasta API path. Model routing is not a backdoor around identity, authorization, audit, rate limits, or license posture.
Per-role model whitelist
Roles can be granted explicit model access; unauthorized requests fail before execution.
Audit on every route
Inference requests, failures, policy denials, and external-provider calls produce audit evidence.
External egress acknowledgment
External inference is called out because prompts and responses leave the Srasta cluster.
Operator override
Recommendations are defaults; operators can change certified deployment profiles with visible tradeoffs.
FAQ
Inference questions operators usually ask first.
Where can Srasta run inference?
The public support matrix lists single-node NVIDIA/vLLM and Apple Silicon/MLX profiles. External OpenAI-compatible endpoints require a separately scoped integration and data-egress review; they are not an additional published deployment profile.
What does LiteLLM do in Srasta?
LiteLLM is the unified inference gateway. Srasta projects certified deployment-profile bindings into stable model routes while LiteLLM sends each request to the configured backend.
How does Srasta choose models?
The installer uses the licensed deployment profile, hardware profile, topology, runtime adapter, inference engine, expected volume, and operator constraints to select only certified model/runtime/hardware bindings.
How is model access governed?
Model requests still pass through Srasta API controls: identity, RBAC, per-role model whitelist, license posture, rate limits, audit, and downstream routing through LiteLLM or the configured provider.
Plan the Runtime
Start with the deployment profile, roles, and constraints, then operate models.
A useful Srasta deployment starts by mapping who will use the system, what work they do, where inference is allowed to run, and what latency, quality, cost, and data-boundary constraints matter.