Bundled vLLM
Srasta deploys a GPU-backed OpenAI-compatible vLLM runtime on the selected GPU host.
Inference & Model Routing
Srasta routes every request through certified private inference.
Srasta separates the operator decision from the runtime plumbing. Teams choose where inference runs and what each role and workflow may use; Srasta turns that into governed LiteLLM routes, certified vLLM or MLX runtime settings, model-access controls, and smoke tests.
Routing Path
One gateway, many possible inference backends.
A caller can ask for coding, business, or general.
Those are stable model aliases. The backend behind each alias is a catalog-certified local runtime
or a controlled OpenAI-compatible endpoint allowed by the deployment profile.
Provider Classes
Operators choose the inference class before choosing the model.
The installer treats inference as a deployment decision. Local inference keeps prompts and responses inside the Srasta cluster. External inference can be self-hosted or hosted, but it creates an explicit data-egress decision that operators must acknowledge.
Bundled MLX
Apple Silicon Community runtime served host-native through Srasta's certified MLX profile.
External self-hosted
Operator-provided vLLM, NVIDIA NIM, or generic OpenAI-compatible endpoint.
Hosted API
Provider APIs such as Anthropic, OpenAI, Hugging Face, Together, or Fireworks through LiteLLM.
Certified Routes
Stable routes let models evolve without changing caller behavior.
Primary governed inference
Routes to the certified chat model selected for the installed deployment profile.
Knowledge retrieval support
Routes embeddings through the certified engine-native path for the selected hardware profile.
Role-aware access
Checks the caller's role against model access policy before any backend receives the request.
Evidence on every route
Records allowed and denied model access so operators can prove what ran and why.
Recommendation Engine
Model fit is a function of deployment profile, hardware, volume, and constraints.
The installer does more than ask whether a model can load. It evaluates fit against the operator's environment: GPU and RAM, inference class, deployment intent, expected concurrency, latency and quality targets, and cost constraints for hosted providers.
Ranking dimensions
- Hardware fit and VRAM envelope.
- Latency targets such as TTFT and token throughput.
- Workflow-specific quality requirements.
- Throughput and peak concurrency fit.
- Hosted-provider cost constraints when relevant.
- Operational consistency across the selected deployment profile.
Runtime Contracts
Correct routing includes parser and embedding decisions.
Tool-call parser
vLLM agentic tool calls require the right parser for the model family. Srasta tracks this per model so tool calls do not silently degrade into plain text.
Reasoning parser
Models with separate thinking output can declare the matching reasoning parser where supported.
Embedding route
Embeddings use the certified engine-native OpenAI-compatible path for the selected hardware profile.
Certified route
Runtime routes resolve to catalog-certified model and runtime bindings rather than ad hoc fallback chains.
Governance
Routing does not bypass Srasta's security model.
Whether a model is local, self-hosted, or hosted, access still enters through the same governed Srasta API path. Model routing is not a backdoor around identity, authorization, audit, rate limits, or license posture.
Per-role model whitelist
Roles can be granted explicit model access; unauthorized requests fail before execution.
Audit on every route
Inference requests, failures, policy denials, and external-provider calls produce audit evidence.
External egress acknowledgment
External inference is called out because prompts and responses leave the Srasta cluster.
Operator override
Recommendations are defaults; operators can change certified deployment profiles with visible tradeoffs.
FAQ
Inference questions operators usually ask first.
Where can Srasta run inference?
Srasta supports local inference through catalog-certified vLLM on NVIDIA GPU deployment profiles and MLX on Apple Silicon deployment profiles, plus controlled OpenAI-compatible external endpoints when the deployment profile allows it.
What does LiteLLM do in Srasta?
LiteLLM is the unified inference gateway. Srasta projects certified deployment-profile bindings into stable model routes while LiteLLM sends each request to the configured backend.
How does Srasta choose models?
The installer uses the licensed deployment profile, hardware profile, topology, runtime adapter, inference engine, expected volume, and operator constraints to select only certified model/runtime/hardware bindings.
How is model access governed?
Model requests still pass through Srasta API controls: identity, RBAC, per-role model whitelist, license posture, rate limits, audit, and downstream routing through LiteLLM or the configured provider.
Plan the Runtime
Start with the deployment profile, roles, and constraints, then operate models.
A useful Srasta deployment starts by mapping who will use the system, what work they do, where inference is allowed to run, and what latency, quality, cost, and data-boundary constraints matter.