Guides

Observability

Logs, live GPU metrics, request traces, cost per output, and exporting to your monitoring stack.

Logs

Terminal

clockingcloud logs product-shots --follow
clockingcloud logs product-shots --since 15m --grep error

Logs include setup output, your process stdout and stderr, and system events such as scale-up, scale-down and idle stop. Logs are retained for 7 days.

Live view

Terminal

clockingcloud top
clockingcloud top product-shots

Shows GPU utilization, memory, queue depth, replicas and the current cost per second. The same view is available in the console under Observability.

Request traces

Terminal

clockingcloud trace req_01JA7X9K2M

Stage	What it measures
queue	Time waiting for a free replica.
cold_start	GPU attach and instance start. Zero when warm.
weight_load	Reading model weights from the node cache or the registry.
denoise	The model steps themselves.
decode	Decoding latents to pixels or frames.
write_back	Writing output to your volume or the response.

Each stage reports seconds and cost, and the stages add up to the request total. Use it to find out whether to raise scale.min, move data closer, or change GPU.

Metrics

Metric	Meaning
gpu_utilization	Percent of time the GPU was busy.
gpu_memory_used_bytes	Memory in use per GPU.
requests_total	Requests received, labeled by outcome.
request_duration_seconds	End-to-end latency histogram (p50, p95, p99).
queue_depth	Requests waiting.
replicas	Running replicas.
cold_starts_total	Replica starts caused by traffic.
cost_usd_total	Spend, accumulating per second of GPU time.

Export

clockingcloud metrics product-shots --export prometheus
clockingcloud metrics product-shots --since 24h -o json

Something unclear or missing? Email hello@clockingcloud.com.