AWS reclaimed my only GPU seven times in 21 hours
Spot can take the box behind the product with two minutes of notice. Seven reclaims in 21 hours proved out the six design decisions that make that boring.
The Koleslaw Blog
Notes from running a fine-tuned 30B model in production: quantization, spot GPUs, failure design, and what it all costs.
4 hours and 37 minutes of serving nothing
Running a 30B model on spot GPUs cut the bill from $588 to $245 a month. First, spot emptied the fleet for 4 hours and 37 minutes, because an armed 100%-spot ASG is never passive.
My "4-bit" quant was 6.2 bits per weight
Nemotron's hidden dimension is 2688, which is not divisible by 256. That one fact quietly inflated a 4-bit quant by a third and cost a weekend.