<div id="readability-page-1" class="page"><div><img src="https://devstarsj.github.io/assets/https://images.unsplash.com/photo-1611974789855-9c2a0a7236a3?w=1600&amp;auto=format&amp;fit=crop&amp;q=80"><h2 id="introduction">Introduction</h2>Cloud costs are rarely anyone’s favorite topic — until they’re everyone’s crisis. AI workloads have introduced a new category of cloud spend that grows exponentially and is often poorly understood by finance teams, misattributed in dashboards, and invisible until the bill arrives.The 2026 reality: GPU instances and LLM API calls are now the fastest-growing line items in most tech company cloud budgets. Teams that ship AI features without FinOps discipline are one viral moment away from a six-figure surprise.This post is the practical guide to AI cloud cost management that most organizations are still figuring out.<img src="https://images.unsplash.com/photo-1551288049-bebda4e38f71?w=1200&amp;auto=format&amp;fit=crop&amp;q=80" alt="Financial data visualization on a screen">Photo by <a href="https://unsplash.com/@lukechesser">Luke Chesser</a> on Unsplash<hr><h2 id="the-ai-cost-landscape">The AI Cost Landscape</h2><h3 id="where-the-money-goes">Where the Money Goes</h3>LLM API calls (pay-per-token models)<ul><li>Input tokens + output tokens billed separately</li><li>Cost varies 10–100× between model tiers</li><li>Prompt caching can reduce costs 60–90% for repeated contexts</li></ul>GPU compute (self-hosted inference/training)<ul><li>NVIDIA H100/H200: $25–35/hour on-demand</li><li>A100: $10–20/hour on-demand</li><li>Spot/preemptible: 60–80% discount, interrupted workloads</li><li>Reserved/committed: 40–60% discount for predictable workloads</li></ul>Vector databases and embeddings<ul><li>Often overlooked; high-volume RAG systems generate millions of embedding calls</li><li>Storage costs for large vector indices</li></ul>Serving infrastructure<ul><li>Load balancers, auto-scaling overhead, storage for model weights</li><li>Often 20–30% on top of pure compute</li></ul><hr><h2 id="llm-api-cost-optimization">LLM API Cost Optimization</h2><h3 id="the-model-selection-matrix">The Model Selection Matrix</h3>The most impactful decision is which model to call. For most production use cases, you don’t need the most powerful model:<div><pre><code>class ModelRouter:
 """Route requests to appropriate model tier based on complexity."""
 
 MODELS = {
 "simple": "claude-haiku-3-5", # ~$0.25/1M tokens
 "standard": "claude-sonnet-4", # ~$3/1M tokens 
 "complex": "claude-opus-4", # ~$15/1M tokens
 }
 
 def route(self, task_type: str, context_length: int) -&gt; str:
 # Simple classification, short context → cheapest model
 if task_type in ("classify", "extract") and context_length &lt; 2000:
 return self.MODELS["simple"]
 
 # Complex reasoning, code generation → top model
 if task_type in ("reason", "code") and context_length &gt; 10000:
 return self.MODELS["complex"]
 
 return self.MODELS["standard"]
</code></pre></div>A typical production system using smart routing saves 40–70% vs. sending everything to the frontier model.<h3 id="prompt-caching">Prompt Caching</h3>Anthropic, OpenAI, and Google all now support prompt caching — paying once to cache a long system prompt or context, then paying only for the cached portion on subsequent calls:<div><pre><code># Anthropic: mark cacheable content explicitly
response = anthropic.messages.create(
 model="claude-sonnet-4",
 messages=[
 {
 "role": "user",
 "content": [
 {
 "type": "text",
 "text": long_document,
 "cache_control": {"type": "ephemeral"} # Cache this!
 },
 {
 "type": "text",
 "text": user_question
 }
 ]
 }
 ]
)
</code></pre></div>Cache hit: ~10% of input token cost. For RAG systems that prepend the same documents to every query, this is a massive saving.<h3 id="output-token-control">Output Token Control</h3>Output tokens are often 3–5× the cost of input tokens. Control output length:<div><pre><code># Bad: open-ended generation
prompt = "Summarize this document: ..."

# Better: constrained generation
prompt = """Summarize this document in exactly 3 bullet points.
Each bullet should be under 20 words.
Do not include introductory phrases.

Document: ..."""
</code></pre></div>For structured outputs (JSON extraction, classification), use structured output APIs rather than free-text:<div><pre><code>response = openai.beta.chat.completions.parse(
 model="gpt-4o-mini", # Cheaper model for structured tasks
 messages=[...],
 response_format=SentimentResult, # Pydantic model
)
# Structured outputs generate less tokens than "I have analyzed the text and determined..."
</code></pre></div><hr><h2 id="gpu-cost-optimization">GPU Cost Optimization</h2><h3 id="workload-aware-instance-selection">Workload-Aware Instance Selection</h3><div><pre><code>Training runs:
 → Reserved/committed GPU instances (40-60% discount)
 → Spot for distributed training with checkpointing

Batch inference (non-realtime):
 → Spot instances with queue + retry
 → Process during off-peak hours (30-50% spot discount improvement)

Online inference (realtime):
 → On-demand for baseline, auto-scale with spot for bursts
 → Explore inference-optimized instances (Inf2 on AWS, TPUs on GCP)
</code></pre></div><h3 id="model-optimization-techniques">Model Optimization Techniques</h3>Before scaling up infrastructure, optimize the model:Quantization: Reduce precision from FP32 to INT8 or INT4<div><pre><code>from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# 4-bit quantization: ~4× memory reduction, ~5-10% quality impact
quantization_config = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_compute_dtype=torch.float16,
 bnb_4bit_use_double_quant=True,
 bnb_4bit_quant_type="nf4"
)

model = AutoModelForCausalLM.from_pretrained(
 "meta-llama/Llama-3-70b-instruct",
 quantization_config=quantization_config,
 device_map="auto"
)
# 70B model: ~140GB FP16 → ~35GB INT4
# Fits on 2× A100 40GB instead of requiring 4×
</code></pre></div>Batching: Maximize GPU utilization by batching requests<div><pre><code># Continuous batching (vLLM model)
from vllm import LLM, SamplingParams

llm = LLM(
 model="meta-llama/Llama-3-8b-instruct",
 max_num_seqs=256, # Batch up to 256 requests simultaneously
 tensor_parallel_size=2,
)
</code></pre></div>vLLM’s continuous batching typically achieves 2–4× GPU throughput vs. naive single-request inference.<hr><h2 id="observability-see-your-ai-costs">Observability: See Your AI Costs</h2>You can’t optimize what you can’t see. Tag and track AI costs specifically:<h3 id="cost-attribution-by-feature">Cost Attribution by Feature</h3><div><pre><code>class TrackedLLMClient:
 def __init__(self, client, cost_tracker):
 self.client = client
 self.cost_tracker = cost_tracker
 
 async def complete(
 self,
 messages: list,
 feature: str, # "search", "summarize", "chat", etc.
 user_id: str,
 ) -&gt; str:
 start = time.monotonic()
 response = await self.client.complete(messages)
 latency_ms = (time.monotonic() - start) * 1000
 
 cost = self.calculate_cost(response.usage)
 
 # Emit to your observability platform
 self.cost_tracker.record({
 "feature": feature,
 "user_id": user_id,
 "model": response.model,
 "input_tokens": response.usage.input_tokens,
 "output_tokens": response.usage.output_tokens,
 "cost_usd": cost,
 "latency_ms": latency_ms,
 })
 
 return response.content
</code></pre></div>Build dashboards showing:<ul><li>Cost per feature (which feature is the most expensive?)</li><li>Cost per user (power users subsidized by others?)</li><li>Cost per request over time (are prompts growing?)</li><li>Cache hit rates</li></ul><h3 id="budget-alerts">Budget Alerts</h3>Every major cloud provider and LLM API now supports budget alerts. Use them:<div><pre><code># AWS Budgets for GPU compute
aws budgets create-budget \
 --account-id $ACCOUNT_ID \
 --budget '{
 "BudgetName": "GPU-Monthly",
 "BudgetLimit": {"Amount": "10000", "Unit": "USD"},
 "TimeUnit": "MONTHLY",
 "BudgetType": "COST",
 "CostFilters": {
 "Service": ["Amazon EC2"],
 "InstanceType": ["p4d.24xlarge", "p3.16xlarge"]
 }
 }' \
 --notifications-with-subscribers '[{
 "Notification": {
 "NotificationType": "ACTUAL",
 "ComparisonOperator": "GREATER_THAN",
 "Threshold": 80
 },
 "Subscribers": [{"SubscriptionType": "EMAIL", "Address": "platform@company.com"}]
 }]'
</code></pre></div><hr><h2 id="the-finops-maturity-model-for-ai">The FinOps Maturity Model for AI</h2>Crawl: Basic tagging, monthly cost review, budget alerts → Know what you’re spendingWalk: Cost per feature attribution, model routing, caching → Understand why you’re spending itRun: Predictive budgeting, automated optimization, showback/chargeback to teams → Optimize continuouslyMost organizations in 2026 are between Crawl and Walk. The teams at Run level treat AI cost optimization as a product capability — with dedicated tooling, SLOs for cost per request, and engineering time allocated to efficiency.<hr><h2 id="quick-wins-checklist">Quick Wins Checklist</h2><ul><li>Enable prompt caching for any LLM call with a system prompt &gt; 1000 tokens</li><li>Audit model tier usage: are you calling GPT-4o/Claude Opus for tasks that don’t need it?</li><li>Set per-user or per-feature rate limits and budget caps</li><li>Move batch workloads to off-peak hours or spot instances</li><li>Add token budget alerts via your LLM provider’s dashboard</li><li>Tag all AI-related cloud resources with <code>team</code>, <code>feature</code>, and <code>environment</code></li><li>Review your top 5 most expensive prompts — can any be shortened or cached?</li></ul><hr><h2 id="conclusion">Conclusion</h2>AI spend is here to stay and will only grow. The organizations that build FinOps discipline for AI workloads now — observability, attribution, optimization loops — will have significant cost advantages as models become more capable and usage scales.The good news: the optimizations aren’t exotic. Prompt caching, model routing, quantization, and batching together can often cut AI infrastructure costs by 50–70% with bounded engineering effort. The bad news: none of this happens automatically. You have to build it.Start with visibility, then optimize. The tools and APIs are mature. The blocker is almost always attention.<hr>Further reading:<ul><li>FinOps Foundation: Cloud FinOps</li><li>Anthropic: Prompt Caching documentation</li><li>vLLM: Production inference optimization</li></ul> <div>이 글이 도움이 되셨다면 공감 및 광고 클릭을 부탁드립니다 :) </div></div></div>

Dev Note

Introduction
Cloud costs are rarely anyone’s favorite topic — until they’re everyone’s crisis. AI workloads have introduced a new category of cloud spend that grows exponentially and is often poorly understood by finance teams, misattributed in dashboards, and invisible until the bill arrives.
The 2026 reality: GPU instances and LLM API calls are now the fastest-growing line items in most tech company cloud budgets. Teams that ship AI features without FinOps discipline are one viral moment away from a six-figure surprise.
This post is the practical guide to AI cloud cost management that most organizations are still figuring out.
Photo by Luke Chesser on Unsplash
The AI Cost Landscape
Where the Money Goes
LLM API calls (pay-per-token models)
Input tokens + output tokens billed separately
Cost varies 10–100× between model tiers
Prompt caching can reduce costs 60–90% for repeated contexts
GPU compute (self-hosted inference/training)
NVIDIA H100/H200: $25–35/hour on-demand
A100: $10–20/hour on-demand
Spot/preemptible: 60–80% discount, interrupted workloads
Reserved/committed: 40–60% discount for predictable workloads
Vector databases and embeddings
Often overlooked; high-volume RAG systems generate millions of embedding calls
Storage costs for large vector indices
Serving infrastructure
Load balancers, auto-scaling overhead, storage for model weights
Often 20–30% on top of pure compute
LLM API Cost Optimization
The Model Selection Matrix
The most impactful decision is which model to call. For most production use cases, you don’t need the most powerful model:
class ModelRouter:
 """Route requests to appropriate model tier based on complexity."""
 
 MODELS = {
 "simple": "claude-haiku-3-5", # ~$0.25/1M tokens
 "standard": "claude-sonnet-4", # ~$3/1M tokens 
 "complex": "claude-opus-4", # ~$15/1M tokens
 }
 
 def route(self, task_type: str, context_length: int) -> str:
 # Simple classification, short context → cheapest model
 if task_type in ("classify", "extract") and context_length < 2000:
 return self.MODELS["simple"]
 
 # Complex reasoning, code generation → top model
 if task_type in ("reason", "code") and context_length > 10000:
 return self.MODELS["complex"]
 
 return self.MODELS["standard"]
A typical production system using smart routing saves 40–70% vs. sending everything to the frontier model.
Prompt Caching
Anthropic, OpenAI, and Google all now support prompt caching — paying once to cache a long system prompt or context, then paying only for the cached portion on subsequent calls:
# Anthropic: mark cacheable content explicitly
response = anthropic.messages.create(
 model="claude-sonnet-4",
 messages=[
 {
 "role": "user",
 "content": [
 {
 "type": "text",
 "text": long_document,
 "cache_control": {"type": "ephemeral"} # Cache this!
 },
 {
 "type": "text",
 "text": user_question
 }
 ]
 }
 ]
)
Cache hit: ~10% of input token cost. For RAG systems that prepend the same documents to every query, this is a massive saving.
Output Token Control
Output tokens are often 3–5× the cost of input tokens. Control output length:
# Bad: open-ended generation
prompt = "Summarize this document: ..."
# Better: constrained generation
prompt = """Summarize this document in exactly 3 bullet points.
Each bullet should be under 20 words.
Do not include introductory phrases.
Document: ..."""
For structured outputs (JSON extraction, classification), use structured output APIs rather than free-text:
response = openai.beta.chat.completions.parse(
 model="gpt-4o-mini", # Cheaper model for structured tasks
 messages=[...],
 response_format=SentimentResult, # Pydantic model
)
# Structured outputs generate less tokens than "I have analyzed the text and determined..."
GPU Cost Optimization
Workload-Aware Instance Selection
Training runs:
 → Reserved/committed GPU instances (40-60% discount)
 → Spot for distributed training with checkpointing
Batch inference (non-realtime):
 → Spot instances with queue + retry
 → Process during off-peak hours (30-50% spot discount improvement)
Online inference (realtime):
 → On-demand for baseline, auto-scale with spot for bursts
 → Explore inference-optimized instances (Inf2 on AWS, TPUs on GCP)
Model Optimization Techniques
Before scaling up infrastructure, optimize the model:
Quantization: Reduce precision from FP32 to INT8 or INT4
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
# 4-bit quantization: ~4× memory reduction, ~5-10% quality impact
quantization_config = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_compute_dtype=torch.float16,
 bnb_4bit_use_double_quant=True,
 bnb_4bit_quant_type="nf4"
)
model = AutoModelForCausalLM.from_pretrained(
 "meta-llama/Llama-3-70b-instruct",
 quantization_config=quantization_config,
 device_map="auto"
)
# 70B model: ~140GB FP16 → ~35GB INT4
# Fits on 2× A100 40GB instead of requiring 4×
Batching: Maximize GPU utilization by batching requests
# Continuous batching (vLLM model)
from vllm import LLM, SamplingParams
llm = LLM(
 model="meta-llama/Llama-3-8b-instruct",
 max_num_seqs=256, # Batch up to 256 requests simultaneously
 tensor_parallel_size=2,
)
vLLM’s continuous batching typically achieves 2–4× GPU throughput vs. naive single-request inference.
Observability: See Your AI Costs
You can’t optimize what you can’t see. Tag and track AI costs specifically:
Cost Attribution by Feature
class TrackedLLMClient:
 def __init__(self, client, cost_tracker):
 self.client = client
 self.cost_tracker = cost_tracker
 
 async def complete(
 self,
 messages: list,
 feature: str, # "search", "summarize", "chat", etc.
 user_id: str,
 ) -> str:
 start = time.monotonic()
 response = await self.client.complete(messages)
 latency_ms = (time.monotonic() - start) * 1000
 
 cost = self.calculate_cost(response.usage)
 
 # Emit to your observability platform
 self.cost_tracker.record({
 "feature": feature,
 "user_id": user_id,
 "model": response.model,
 "input_tokens": response.usage.input_tokens,
 "output_tokens": response.usage.output_tokens,
 "cost_usd": cost,
 "latency_ms": latency_ms,
 })
 
 return response.content
Build dashboards showing:
Cost per feature (which feature is the most expensive?)
Cost per user (power users subsidized by others?)
Cost per request over time (are prompts growing?)
Cache hit rates
Budget Alerts
Every major cloud provider and LLM API now supports budget alerts. Use them:
# AWS Budgets for GPU compute
aws budgets create-budget \
 --account-id $ACCOUNT_ID \
 --budget '{
 "BudgetName": "GPU-Monthly",
 "BudgetLimit": {"Amount": "10000", "Unit": "USD"},
 "TimeUnit": "MONTHLY",
 "BudgetType": "COST",
 "CostFilters": {
 "Service": ["Amazon EC2"],
 "InstanceType": ["p4d.24xlarge", "p3.16xlarge"]
 }
 }' \
 --notifications-with-subscribers '[{
 "Notification": {
 "NotificationType": "ACTUAL",
 "ComparisonOperator": "GREATER_THAN",
 "Threshold": 80
 },
 "Subscribers": [{"SubscriptionType": "EMAIL", "Address": "platform@company.com"}]
 }]'
The FinOps Maturity Model for AI
Crawl: Basic tagging, monthly cost review, budget alerts
→ Know what you’re spending
Walk: Cost per feature attribution, model routing, caching
→ Understand why you’re spending it
Run: Predictive budgeting, automated optimization, showback/chargeback to teams
→ Optimize continuously
Most organizations in 2026 are between Crawl and Walk. The teams at Run level treat AI cost optimization as a product capability — with dedicated tooling, SLOs for cost per request, and engineering time allocated to efficiency.
Quick Wins Checklist
Enable prompt caching for any LLM call with a system prompt > 1000 tokens
Audit model tier usage: are you calling GPT-4o/Claude Opus for tasks that don’t need it?
Set per-user or per-feature rate limits and budget caps
Move batch workloads to off-peak hours or spot instances
Add token budget alerts via your LLM provider’s dashboard
Tag all AI-related cloud resources with team, feature, and environment
Review your top 5 most expensive prompts — can any be shortened or cached?
Conclusion
AI spend is here to stay and will only grow. The organizations that build FinOps discipline for AI workloads now — observability, attribution, optimization loops — will have significant cost advantages as models become more capable and usage scales.
The good news: the optimizations aren’t exotic. Prompt caching, model routing, quantization, and batching together can often cut AI infrastructure costs by 50–70% with bounded engineering effort. The bad news: none of this happens automatically. You have to build it.
Start with visibility, then optimize. The tools and APIs are mature. The blocker is almost always attention.
Further reading:
FinOps Foundation: Cloud FinOps
Anthropic: Prompt Caching documentation
vLLM: Production inference optimization

FinOps in the Age of AI Workloads: Taming GPU and LLM Inference Costs

댓글