> ## Documentation Index
> Fetch the complete documentation index at: https://hubify.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Pods

> Provision, monitor, and manage GPU compute pods, SSH access, cost tracking, and idle detection.

# Pods API

<Note>
  Shipped today: `GET /v1/pods` (list), `POST /v1/pods/{id}/start`, `POST /v1/pods/{id}/stop`. Lab cost summary lives at `GET /v1/costs`. Per-pod sub-routes documented below (`GET /v1/pods/{id}`, `/restart`, `/ssh`, `/terminate`, `/pods/costs`) are planned, not yet live. Use the web app's Pods view for SSH and termination today.
</Note>

Pods are GPU compute instances provisioned on RunPod. They boot with your lab's environment, run experiments, and tear down when finished. The Pods API gives you full control over the lifecycle, SSH access, and cost monitoring.

## Pod Status

| Status       | Description                             |
| ------------ | --------------------------------------- |
| `creating`   | Pod is being provisioned on RunPod      |
| `running`    | Pod is online and executing or idle     |
| `stopped`    | Pod has been stopped (can be restarted) |
| `terminated` | Pod has been permanently terminated     |

## Create a Pod

Provision a new GPU pod.

<ParamField body="labId" type="string" required>
  Lab this pod belongs to.
</ParamField>

<ParamField body="gpuType" type="string" required>
  GPU type: `H200`, `H100`, `A100`.
</ParamField>

<ParamField body="gpuCount" type="number" default={1}>
  Number of GPUs. Multi-GPU pods available for H100 and A100.
</ParamField>

<ParamField body="maxHours" type="number">
  Maximum runtime in hours. Pod auto-terminates after this limit. Omit for no limit (controlled by budget).
</ParamField>

<ParamField body="image" type="string" default="hubify/base:latest">
  Docker image for the pod environment.
</ParamField>

<ParamField body="env" type="object">
  Environment variables to set on the pod.
</ParamField>

<ParamField body="volumeSize" type="number" default={50}>
  Persistent volume size in GB.
</ParamField>

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://www.hubify.com/api/v1/pods \
    -H "Authorization: Bearer $HUBIFY_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{
      "labId": "j57a8k9m2n3p4q5r",
      "gpuType": "H200",
      "maxHours": 8,
      "env": {
        "COBAYA_PACKAGES": "/workspace/packages",
        "EXPERIMENT_ID": "EXP-054"
      },
      "volumeSize": 100
    }'
  ```

  ```typescript TypeScript SDK theme={null}
  const pod = await hubify.pods.create({
    labId: "j57a8k9m2n3p4q5r",
    gpuType: "H200",
    maxHours: 8,
    env: {
      COBAYA_PACKAGES: "/workspace/packages",
      EXPERIMENT_ID: "EXP-054",
    },
    volumeSize: 100,
  });
  ```
</CodeGroup>

<ResponseField name="data" type="object">
  <Expandable title="Pod object">
    <ResponseField name="_id" type="string" required>Convex document ID.</ResponseField>
    <ResponseField name="labId" type="string" required>Parent lab ID.</ResponseField>
    <ResponseField name="provider" type="string" required>Always `runpod`.</ResponseField>
    <ResponseField name="podId" type="string" required>RunPod pod identifier.</ResponseField>
    <ResponseField name="gpuType" type="string" required>GPU type.</ResponseField>
    <ResponseField name="status" type="string" required>Current status.</ResponseField>
    <ResponseField name="sshHost" type="string">SSH hostname (available once running).</ResponseField>
    <ResponseField name="sshPort" type="number">SSH port (available once running).</ResponseField>
    <ResponseField name="costPerHour" type="number">Hourly cost in USD.</ResponseField>
    <ResponseField name="createdAt" type="number" required>Creation timestamp (ms).</ResponseField>
  </Expandable>
</ResponseField>

## List Pods

<ParamField query="labId" type="string" required>
  Lab ID.
</ParamField>

<ParamField query="status" type="string">
  Filter by status: `creating`, `running`, `stopped`, `terminated`.
</ParamField>

<CodeGroup>
  ```bash curl theme={null}
  curl "https://www.hubify.com/api/v1/pods?labId=j57a8k9m2n3p4q5r&status=running" \
    -H "Authorization: Bearer $HUBIFY_TOKEN"
  ```

  ```typescript TypeScript SDK theme={null}
  const pods = await hubify.pods.list({
    labId: "j57a8k9m2n3p4q5r",
    status: "running",
  });
  ```
</CodeGroup>

## Get Pod Status

Detailed status including GPU utilization, memory, and idle time.

<CodeGroup>
  ```bash curl theme={null}
  curl https://www.hubify.com/api/v1/pods/pod_abc123 \
    -H "Authorization: Bearer $HUBIFY_TOKEN"
  ```

  ```typescript TypeScript SDK theme={null}
  const pod = await hubify.pods.get({ podId: "pod_abc123" });
  ```
</CodeGroup>

<ResponseField name="data" type="object">
  <Expandable title="Detailed pod status">
    <ResponseField name="podId" type="string" required>RunPod pod ID.</ResponseField>
    <ResponseField name="status" type="string" required>Current status.</ResponseField>
    <ResponseField name="gpuType" type="string" required>GPU type.</ResponseField>
    <ResponseField name="gpuUtilization" type="number">Current GPU utilization (0-1).</ResponseField>
    <ResponseField name="vramUsedGb" type="number">Current VRAM usage in GB.</ResponseField>
    <ResponseField name="vramTotalGb" type="number">Total VRAM in GB.</ResponseField>
    <ResponseField name="idleMinutes" type="number">Minutes since last GPU activity.</ResponseField>
    <ResponseField name="frozen" type="boolean">Whether the pod is frozen (budget limit reached).</ResponseField>
    <ResponseField name="freezeReason" type="string">Reason for freeze, if applicable.</ResponseField>
    <ResponseField name="uptimeSeconds" type="number">Total uptime in seconds.</ResponseField>
    <ResponseField name="costAccumulated" type="number">Cost accumulated so far in USD.</ResponseField>
    <ResponseField name="sshHost" type="string">SSH hostname.</ResponseField>
    <ResponseField name="sshPort" type="number">SSH port.</ResponseField>
  </Expandable>
</ResponseField>

## SSH Configuration

Once a pod is running, SSH credentials are available in the pod details.

<CodeGroup>
  ```bash curl theme={null}
  curl https://www.hubify.com/api/v1/pods/pod_abc123/ssh \
    -H "Authorization: Bearer $HUBIFY_TOKEN"
  ```

  ```typescript TypeScript SDK theme={null}
  const ssh = await hubify.pods.getSSH({ podId: "pod_abc123" });
  // { host: "205.196.19.52", port: 11452, user: "root" }
  ```
</CodeGroup>

```bash theme={null}
# Connect via SSH
ssh root@205.196.19.52 -p 11452
```

## Stop a Pod

Stop a running pod. The persistent volume is preserved. The pod can be restarted later.

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://www.hubify.com/api/v1/pods/pod_abc123/stop \
    -H "Authorization: Bearer $HUBIFY_TOKEN"
  ```

  ```typescript TypeScript SDK theme={null}
  await hubify.pods.stop({ podId: "pod_abc123" });
  ```
</CodeGroup>

## Restart a Pod

Restart a stopped pod with the same configuration.

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://www.hubify.com/api/v1/pods/pod_abc123/restart \
    -H "Authorization: Bearer $HUBIFY_TOKEN"
  ```

  ```typescript TypeScript SDK theme={null}
  await hubify.pods.restart({ podId: "pod_abc123" });
  ```
</CodeGroup>

## Terminate a Pod

Permanently terminate a pod. The persistent volume is retained for 7 days, then deleted.

<Warning>
  Termination is irreversible. Make sure experiment results have been synced back to the lab before terminating.
</Warning>

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST https://www.hubify.com/api/v1/pods/pod_abc123/terminate \
    -H "Authorization: Bearer $HUBIFY_TOKEN"
  ```

  ```typescript TypeScript SDK theme={null}
  await hubify.pods.terminate({ podId: "pod_abc123" });
  ```
</CodeGroup>

## GPU Types and Pricing

| GPU      | VRAM   | Cost/hr (approx.) | Best For                                           |
| -------- | ------ | ----------------- | -------------------------------------------------- |
| **H200** | 141 GB | \$4.00            | Large MCMC, foundation models, multi-survey sweeps |
| **H100** | 80 GB  | \$2.50            | Training runs, medium MCMC, anomaly detection      |
| **A100** | 80 GB  | \$1.50            | General GPU compute, smaller models                |

<Note>
  Prices are approximate and depend on RunPod availability. Actual costs are tracked in the `cost_tracking` table and visible via the cost endpoints.
</Note>

## Cost Tracking

### Get Current Month Costs

<CodeGroup>
  ```bash curl theme={null}
  curl "https://www.hubify.com/api/v1/pods/costs?labId=j57a8k9m2n3p4q5r&month=current" \
    -H "Authorization: Bearer $HUBIFY_TOKEN"
  ```

  ```typescript TypeScript SDK theme={null}
  const costs = await hubify.pods.getCosts({
    labId: "j57a8k9m2n3p4q5r",
    month: "current",
  });
  ```
</CodeGroup>

<ResponseField name="data" type="object">
  <Expandable title="Cost summary">
    <ResponseField name="gpuCost" type="number">Total GPU compute cost in USD.</ResponseField>
    <ResponseField name="llmCost" type="number">Total LLM API cost in USD.</ResponseField>
    <ResponseField name="storageCost" type="number">Storage cost in USD.</ResponseField>
    <ResponseField name="totalCost" type="number">Combined total cost.</ResponseField>
    <ResponseField name="gpuHours" type="number">Total GPU hours consumed.</ResponseField>
    <ResponseField name="experimentsRun" type="number">Experiments run this month.</ResponseField>
    <ResponseField name="budgetRemaining" type="number">Remaining budget in USD.</ResponseField>
    <ResponseField name="budgetPercent" type="number">Percentage of budget used (0-1).</ResponseField>
  </Expandable>
</ResponseField>

## Idle Detection

Hubify monitors GPU utilization and alerts you when pods are sitting idle. This is accessible through the pod status endpoint (the `idleMinutes` field) and triggers notifications when a configurable threshold is exceeded.

<Note>
  An idle GPU is a violation of the Hubify Labs philosophy. The system will suggest queued experiments to deploy on idle pods, and auto-deploy if you have auto-schedule enabled.
</Note>

### Configure Idle Threshold

<Note>
  `PATCH /api/v1/labs/{slug}/settings` is planned — not yet shipped. Configure idle thresholds via Settings in the web app. The auto-stop cron (shipped in v1.3.1) uses a default 6-hour threshold.
</Note>

<CodeGroup>
  ```bash curl theme={null}
  curl -X PATCH https://www.hubify.com/api/v1/labs/dark-energy/settings \
    -H "Authorization: Bearer $HUBIFY_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"idleAlertMinutes": 15, "autoScheduleEnabled": true}'
  ```

  ```typescript TypeScript SDK theme={null}
  await hubify.labs.updateSettings({
    slug: "dark-energy",
    idleAlertMinutes: 15,
    autoScheduleEnabled: true,
  });
  ```
</CodeGroup>