Connect to a vLLM Managed Inference endpoint
Send a chat completion request to a Managed Inference endpoint from the CLI or an OpenAI-compatible client.
Send a chat completion request to a vLLM Managed Inference endpoint from the CLI or any OpenAI-compatible client.
Prerequisites
Before you start, make sure that you have the following.
- A running Managed Inference Job with a serving endpoint. See Create a vLLM Managed Inference Job with the CLI.
- Your Managed Inference API key, if the endpoint requires an authorization header. See Create an API key with the CLI.
- The CosmicAC CLI installed and configured, for the CLI method. See Install the CLI.
Steps
Find your endpoint name
List your endpoints.
cosmicac models statsEach endpoint appears as Endpoint: <endpoint-name>. Copy the one you want to call.
Send a request
Use the CLI or any OpenAI-compatible client.
If you enabled Require Authorization header when you created the job, you need an API key. If you didn't, answer no to the CLI's API key prompt, and leave the Authorization header out of the request.
Start an interactive chat session.
cosmicac inference chatAnswer the prompts to start the session.
The cosmicac inference chat session asks for the following values.
- Use an API key: answer yes to authenticate, and then enter the key at the masked prompt. If the
COSMIC_API_KEYenvironment variable is set, the CLI uses that key and doesn't ask. - Endpoint ID: the endpoint name from the previous step.
The session accepts the following commands.
| Command | What it does |
|---|---|
/video <url> | Attaches a video to your next message. The URL must point to a video file the model can download in one request. A streaming URL that delivers in chunks, such as a YouTube link, doesn't work. |
/clear-video | Removes the attachment. |
/think and /no-think | Show or hide the model's reasoning in later replies. |
/exit or /quit | Ends the session, as does Ctrl+C. |
/no-think hides the reasoning and asks the model to stop producing it. Some models, such as MiniMax, ignore that request and keep spending tokens on thinking you no longer see.
Help and troubleshooting
The CLI shows "The inference proxy rejected this API key."
The CLI checks your API key before the chat session starts, and the inference proxy rejected it. Check that the key is current.
cosmicac api-keys listIf the key was revoked, create a new one. See Create an API key with the CLI.