Investigate pricing information from Lambda Labs

eliribble commented

2026-07-19 22:38:17 +00:00

Owner

Our current implementation of vision_workers for Nidus Reveal attempts to track cost internally by keeping track of the price of an instance and how long we think it stays up. We control this, but it may introduce some timekeeping error compared to what our provider measures.

I'd like you to look at the Lambda Labs API documentation and determine if it would be possible to use their API to track price. This would allow our systems to be exactly in sync. If it's possible, I'd like a detailed writeup of how it would be done.

Our current implementation of vision_workers for Nidus Reveal attempts to track cost internally by keeping track of the price of an instance and how long we think it stays up. We control this, but it may introduce some timekeeping error compared to what our provider measures. I'd like you to look at [the Lambda Labs API documentation](https://docs-api.lambda.ai/api/cloud) and determine if it would be possible to use their API to track price. This would allow our systems to be exactly in sync. If it's possible, I'd like a detailed writeup of how it would be done.

ned was assigned by eliribble

2026-07-19 22:38:17 +00:00

ned commented

2026-07-19 22:40:10 +00:00

Member

Investigation: Lambda Labs API for Price Tracking

I dove into the Lambda Labs Cloud API documentation and examined how our current vision_worker cost tracking works.

Current Approach

The vision_worker table has a cost_per_hour column that an operator sets manually when registering a worker. The system tracks first_poll_at, last_heartbeat, and shutdown_at timestamps, then the total cost is time × this manually-entered price. The concern is that if our internal timekeeping drifts from what the provider measures, we get a mismatch.

Lambda Labs API: What It Offers

The API provides pricing through GET /api/v1/instance-types, which returns every instance type with its authoritative price_cents_per_hour. For example:

{
  "data": {
    "gpu_8x_h100_sxm5gdr": {
      "instance_type": {
        "name": "gpu_8x_h100_sxm5gdr",
        "description": "8x H100 (80 GB SXM5)",
        "price_cents_per_hour": 3592,
        "specs": { "vcpus": 208, "memory_gib": 1800, "storage_gib": 24780, "gpus": 8 }
      },
      "regions_with_capacity_available": [...]
    }
  }
}

Each running instance returned by GET /api/v1/instances also includes the full instance_type object with price_cents_per_hour baked in.

What the API does NOT provide:

No billing/usage history endpoint
No per-instance accrued cost or "billed seconds" field
No invoice or charges API
No way to query what the provider billed for a specific instance after termination

The only lifecycle statuses are: booting, active, unhealthy, terminated, terminating, preempted — but there is no uptime or "total cost so far" exposed in the instance object.

Can We Use It for Price Tracking?

Yes, partially. There are three areas where the API improves our current approach:

1. Authoritative Instance-Type Pricing (Eliminates Manual Entry)

Instead of having an operator enter cost_per_hour during registration, we can call GET /api/v1/instance-types and look up the real price_cents_per_hour for the instance type. This eliminates the risk of stale or incorrect operator-entered prices.

Implementation sketch:

When creating a worker, store the Lambda Labs instance type name (we already have provider + instance_type in the DB)
When the worker phone-homes (first poll or on a schedule), call Lambda Labs API to get the current price_cents_per_hour and update vision_worker.cost_per_hour
This keeps pricing synced to whatever Lambda Labs currently charges (even if prices change)

2. Instance Lifecycle Tracking via Provider ID

The API exposes instance status changes, but it is poll-based — you poll /instances or /instances/{id} to see current status. The statuses include preempted, terminated, terminating, which you could use to detect when billing stops.

Limitation: The API doesn't expose exact timestamps for state transitions (only current state). So you still need to record the time when you see the termination to calculate duration.

3. Instance Type ↔ Price Mapping

Since GET /api/v1/instances/{id} includes the full instance_type with price_cents_per_hour, you could periodically confirm that the price you're tracking matches the API. This works as a cross-check that catches pricing surprises.

What Can't Be Done

The API does not provide:

Lambda's own billing measurement of uptime per instance
A "billed seconds" or "compute time used" field
Granular billing line items via API

Since Lambda bills in wall-clock time (like most cloud GPU providers — you pay per hour or partial hour that the instance is active), our internal first_poll_at → shutdown_at tracking using the same instance lifecycle events as Lambda would give us the same wall-clock duration Lambda uses. The remaining gap would be partial-hour rounding, which we'd handle the same way Lambda does.

Summary

Aspect	Before (Manual)	After (API-driven)
Price source	Operator enters cost_per_hour manually	Fetched from GET /api/v1/instance-types
Price freshness	Stale until operator updates	Fresh at start time, re-fetchable on demand
Duration tracking	Our internal timestamps	Same (no API alternative exists)
Cross-check	None	Can verify when polling instance status

Bottom line: Using the Lambda Labs API to look up pricing is definitely possible and eliminates the manual-entry error. It gets us the authoritative price that Lambda charges, which is the biggest source of inaccuracy. The duration tracking has to remain our own because Lambda doesn't expose billed time, but we can mitigate any discrepancy by using the same wall-clock semantics as the provider (i.e., price_cents_per_hour × wall-clock hours the instance was active).

If you'd like me to create a feature branch with the implementation, I can start by:

Adding a Lambda Labs API client that calls GET /api/v1/instance-types
Updating the worker registration flow to look up price from the API
Adding a periodic price-verify step to our existing worker heartbeat/status loop

Let me know how deep you want me to go on this!

## Investigation: Lambda Labs API for Price Tracking I dove into the Lambda Labs Cloud API documentation and examined how our current vision_worker cost tracking works. ## Current Approach The vision_worker table has a cost_per_hour column that an operator sets manually when registering a worker. The system tracks first_poll_at, last_heartbeat, and shutdown_at timestamps, then the total cost is time × this manually-entered price. The concern is that if our internal timekeeping drifts from what the provider measures, we get a mismatch. ## Lambda Labs API: What It Offers The API provides pricing through GET /api/v1/instance-types, which returns every instance type with its authoritative price_cents_per_hour. For example: ```json { "data": { "gpu_8x_h100_sxm5gdr": { "instance_type": { "name": "gpu_8x_h100_sxm5gdr", "description": "8x H100 (80 GB SXM5)", "price_cents_per_hour": 3592, "specs": { "vcpus": 208, "memory_gib": 1800, "storage_gib": 24780, "gpus": 8 } }, "regions_with_capacity_available": [...] } } } ``` Each running instance returned by GET /api/v1/instances also includes the full instance_type object with price_cents_per_hour baked in. **What the API does NOT provide:** - No billing/usage history endpoint - No per-instance accrued cost or "billed seconds" field - No invoice or charges API - No way to query what the provider billed for a specific instance after termination The only lifecycle statuses are: booting, active, unhealthy, terminated, terminating, preempted — but there is no uptime or "total cost so far" exposed in the instance object. ## Can We Use It for Price Tracking? **Yes, partially.** There are three areas where the API improves our current approach: ### 1. Authoritative Instance-Type Pricing (Eliminates Manual Entry) Instead of having an operator enter cost_per_hour during registration, we can call GET /api/v1/instance-types and look up the real price_cents_per_hour for the instance type. This eliminates the risk of stale or incorrect operator-entered prices. Implementation sketch: - When creating a worker, store the Lambda Labs instance type name (we already have provider + instance_type in the DB) - When the worker phone-homes (first poll or on a schedule), call Lambda Labs API to get the current price_cents_per_hour and update vision_worker.cost_per_hour - This keeps pricing synced to whatever Lambda Labs currently charges (even if prices change) ### 2. Instance Lifecycle Tracking via Provider ID The API exposes instance status changes, but it is poll-based — you poll /instances or /instances/{id} to see current status. The statuses include preempted, terminated, terminating, which you could use to detect when billing stops. Limitation: The API doesn't expose exact timestamps for state transitions (only current state). So you still need to record the time when you see the termination to calculate duration. ### 3. Instance Type ↔ Price Mapping Since GET /api/v1/instances/{id} includes the full instance_type with price_cents_per_hour, you could periodically confirm that the price you're tracking matches the API. This works as a cross-check that catches pricing surprises. ## What Can't Be Done The API does not provide: - Lambda's own billing measurement of uptime per instance - A "billed seconds" or "compute time used" field - Granular billing line items via API Since Lambda bills in wall-clock time (like most cloud GPU providers — you pay per hour or partial hour that the instance is active), our internal first_poll_at → shutdown_at tracking using the same instance lifecycle events as Lambda would give us the same wall-clock duration Lambda uses. The remaining gap would be partial-hour rounding, which we'd handle the same way Lambda does. ## Summary | Aspect | Before (Manual) | After (API-driven) | |--------|-----------------|-------------------| | Price source | Operator enters cost_per_hour manually | Fetched from GET /api/v1/instance-types | | Price freshness | Stale until operator updates | Fresh at start time, re-fetchable on demand | | Duration tracking | Our internal timestamps | Same (no API alternative exists) | | Cross-check | None | Can verify when polling instance status | **Bottom line:** Using the Lambda Labs API to look up pricing is definitely possible and eliminates the manual-entry error. It gets us the authoritative price that Lambda charges, which is the biggest source of inaccuracy. The duration tracking has to remain our own because Lambda doesn't expose billed time, but we can mitigate any discrepancy by using the same wall-clock semantics as the provider (i.e., price_cents_per_hour × wall-clock hours the instance was active). If you'd like me to create a feature branch with the implementation, I can start by: 1. Adding a Lambda Labs API client that calls GET /api/v1/instance-types 2. Updating the worker registration flow to look up price from the API 3. Adding a periodic price-verify step to our existing worker heartbeat/status loop Let me know how deep you want me to go on this!

eliribble commented

2026-07-19 23:11:14 +00:00

Author

Owner

Okay, thanks for digging in to this. Here's what I'm thinking we want to do:

During the 'worker-instance-start' background job, after the instance has been successfully started, capture the pricing information and the start time of the instance
During the 'worker-software-install' when we get notification that the worker is alive and we install its basic operating software, we confirm that the pricing is at it was before. If there's a change, emit a warning in the log
During the 'worker-instance-shutdown' event capture the stop time of the instance, and one more time, confirm the pricing per unit time. Then we calculate the total running time and therefore the total price of the instance.

I'd like you to create a pull request to implement this

Okay, thanks for digging in to this. Here's what I'm thinking we want to do: * During the 'worker-instance-start' background job, after the instance has been successfully started, capture the pricing information and the start time of the instance * During the 'worker-software-install' when we get notification that the worker is alive and we install its basic operating software, we confirm that the pricing is at it was before. If there's a change, emit a warning in the log * During the 'worker-instance-shutdown' event capture the stop time of the instance, and one more time, confirm the pricing per unit time. Then we calculate the total running time and therefore the total price of the instance. I'd like you to create a pull request to implement this

ned referenced this issue

2026-07-19 23:16:19 +00:00

Lambda Labs pricing tracking #146

ned commented

2026-07-19 23:16:24 +00:00

Member

PR #146 is ready with the implementation:

Adds DB migration 00195 — adds started_at and total_cost columns to vision_worker
worker-instance-start — after CreateMachine succeeds, calls EstimateCost() on the provider driver, stores the authoritative cost_per_hour and sets started_at
worker-software-install — re-fetches pricing and compares; logs a warning if the price changed
worker-instance-shutdown — final price check, then computes (shutdown_at - started_at) × cost_per_hour, stores total_cost, and logs the summary
computeUptime updated in both platform/ and resource/ to prefer started_at over created_at when available

The pricing capture is provider-agnostic (both Lambda Labs and DigitalOcean implement EstimateCost()), so it works for both without special-casing.

PR #146 is ready with the implementation: - **Adds DB migration 00195** — adds `started_at` and `total_cost` columns to `vision_worker` - **worker-instance-start** — after CreateMachine succeeds, calls `EstimateCost()` on the provider driver, stores the authoritative `cost_per_hour` and sets `started_at` - **worker-software-install** — re-fetches pricing and compares; logs a warning if the price changed - **worker-instance-shutdown** — final price check, then computes `(shutdown_at - started_at) × cost_per_hour`, stores `total_cost`, and logs the summary - **computeUptime** updated in both `platform/` and `resource/` to prefer `started_at` over `created_at` when available The pricing capture is provider-agnostic (both Lambda Labs and DigitalOcean implement `EstimateCost()`), so it works for both without special-casing.

Rows
Columns

Investigate pricing information from Lambda Labs #145

Investigation: Lambda Labs API for Price Tracking

Current Approach

Lambda Labs API: What It Offers

Can We Use It for Price Tracking?

1. Authoritative Instance-Type Pricing (Eliminates Manual Entry)

2. Instance Lifecycle Tracking via Provider ID

3. Instance Type ↔ Price Mapping

What Can't Be Done

Summary