www.awsgood.com
ParallelCluster is essentially a CloudFormation template plus the Slurm scheduler plus a pile of lifecycle scripts. There are no license fees, so every cent can be matched to a specific resource in Billing.
The head node runs 7×24 by default, hosting slurmctld and the pcluster CLI. An m5.xlarge on-demand is about 0.19 USD/hour, which comes to just over 140 USD a month, roughly 1,000 RMB. It is the single point of the whole cluster; don't swap it for a t3.small to save money—it bogs down once the configuration grows.
Compute nodes are started and stopped dynamically by Slurm based on queue demand, and the default scaledown_idletime is 10 minutes. After a job finishes, a node still idles for 10 minutes before shutting down. For batch queues I usually set it to 3 minutes; for interactive parameter tuning I keep the default.
Storage splits into two parts. Shared EBS on gp3 is about 0.08 USD/GB-month; a 1.2 TB FSx for Lustre Scratch file system is roughly 100+ USD a month, with a minimum capacity and minimum billing duration—once it's on, you're billed by the hour. There are also two commonly forgotten items: NAT Gateway bills by the hour plus data processed, and CloudWatch Logs charges per GB ingested; for a medium cluster, tens of dollars a month from just these two is common.
| Resource | Billing method | Unit price magnitude | Monthly magnitude (720h) | Notes |
|---|---|---|---|---|
| m5.xlarge head node | On-demand | ~0.19 USD/h | ~140 USD (about 1,000 RMB) | Always on, 7×24 |
| c5.2xlarge compute node | On-demand | ~0.34 USD/h | ~245 USD | Billed only during jobs |
| c5.2xlarge compute node | Spot | 60%–90% discount | ~25–100 USD | May be reclaimed |
| c5n.18xlarge | On-demand | ~3.9 USD/h | ~2,800 USD (about 20,000 RMB) | Large-scale MPI |
| FSx for Lustre 1.2TB Scratch | By capacity | ~0.145 USD/GB-month | ~170 USD | Has a minimum billing period |
| gp3 shared storage 500GB | By capacity | ~0.08 USD/GB-month | ~40 USD | Remember to clean up when deleting the cluster |
A p4d.24xlarge on-demand is about 32 USD/hour, so a single day runs 700+ USD. Using Spot plus checkpoints for training jobs is standard practice, but if checkpoints are written to FSx too frequently, FSx write throughput becomes the bottleneck first; tune the interval to your actual fault-tolerance granularity.
Teams in China mostly put their clusters in ap-northeast-1 or ap-southeast-1, where unit prices are noticeably higher than us-east-1. After switching the console to Chinese, several field names in Budgets don't match the English docs; searching the official docs with English keywords is more accurate.
The first layer is Budgets, capping the total at the service dimension. Many people fill in just one total amount, and then EC2 stays under while FSx goes over—and the alarm never makes a sound.
cat > budget.json <<'EOF'
{
"BudgetName": "pcluster-monthly",
"BudgetLimit": {"Amount": "1500", "Unit": "USD"},
"TimeUnit": "MONTHLY",
"BudgetType": "COST",
"CostFilters": {
"Service": [
"Amazon Elastic Compute Cloud - Compute",
"Amazon FSx",
"Amazon Elastic Block Store",
"Amazon Virtual Private Cloud"
]
}
}
EOF
aws budgets create-budget
--account-id 123456789012
--budget file://budget.json
--notifications-with-subscribers file://notify.json
I set thresholds at two levels: an email at 50% of actual spend, an SMS at 85%. Add one more at 100% of the forecast—this one saves you earlier than actuals because it projects forward at the current rate.
The second layer is CloudWatch alarms watching cumulative EC2 running hours. Budgets updates daily, with significant lag; CloudWatch metrics are minute-level and can catch things like "a queue didn't scale down last night." Pull `CPUUtilization` and instance-count curves per AutoScalingGroup—the line that won't come down is the culprit.
The third layer is automatic shutdown. The first two layers rely on someone looking; nobody watches email at 2 a.m. Use EventBridge Scheduler to trigger a Lambda every day at 20:00 that stops EC2 instances tagged with `parallelcluster:cluster-name`, and start them again at 8:00 the next day.
Note: don't apply this directly to production queues—stopping the head node cuts off running jobs. Start by applying it only to the head node and let Slurm handle compute node scale-down itself.
## Hands-on test and pitfalls: FSx stuck on deletion, billing kept running for two extra weeks
In November 2025 I was doing managed operations for a team doing simulation computing. They had deleted their test cluster, yet the next month's bill still had 100+ USD of FSx running. Checking it, the fs-xxx in `aws fsx describe-file-systems` had status AVAILABLE, with a creation time from two months earlier.
At the time, `pcluster delete-cluster` reported this line:
Deleting cluster failed. Cluster stack my-hpc-cluster did not delete successfully.
Reason: Resource deletion failed. The FSx file system fs-0a1b2c3d4e5f67890
cannot be deleted while it has mount targets.
FSx was attached to ENIs in two subnets; CloudFormation deleted the security groups first, the ENIs were left behind, and the file system got stuck in the deletion chain. The mistake in the first version of the configuration was assuming that deleting the cluster would take FSx with it—it isn't in the stack's lifecycle, and `DeletionPolicy` doesn't apply to it.
The verification method is crude but effective: Billing → Cost Explorer, group by Service=FSx, stretch the time range to 30 days; that steady hourly billing curve shows it was alive the whole time. Deleting the ENIs by hand and then the file system took 40 minutes end to end.
After that I added a rule to every cluster: before deleting a cluster, confirm with `aws fsx describe-file-systems`, then come back to Cost Explorer the next day to see whether costs have gone to zero. For troubleshooting this kind of deletion residue and budget misconfiguration, our managed operations at awsgood.com has a fixed checklist; going through it item by item beats relying on memory.
## When not to use ParallelCluster
- Work that finishes in two hours on a single machine. Spending half a day configuring a cluster and tuning queues costs far more time than the machine money saved.
- Pure parameter sweeps where jobs barely communicate. Batch plus Step Functions is less hassle, and Slurm's scheduling advantages don't show.
- Nobody on the team writes Slurm scripts and nobody wants to maintain it. The operational burden of the cluster will outweigh the cost benefit.
- Projects shorter than a week. Setting up the cluster, configuring FSx, and tuning queues takes about as long as running the jobs.
## FAQ
**Does ParallelCluster itself charge a fee?** No. It's an open-source tool, and the entire bill comes from the underlying AWS resources.
**How do I save money if jobs only run during the day?** Configure scheduled start/stop for the head node, use Spot for compute nodes, and set `scaledown_idletime` to 3 minutes.
**Are jobs lost when Spot is reclaimed?** Slurm will requeue the job, provided the script can resume from a checkpoint. Don't put work that can't resume into a Spot queue.
One thing you can do tonight: create the Budget from the JSON above, set the amount to 1.2× your actual spend last month, copy the `Service` list as-is, and come back tomorrow to look at the forecast line.
## References
- [What is AWS ParallelCluster](https://docs.aws.amazon.com/parallelcluster/latest/ug/what-is-aws-parallelcluster.html)
- [Working with clusters (cluster creation and deletion)](https://docs.aws.amazon.com/parallelcluster/latest/ug/working-with-clusters.html)
- [Managing costs with AWS Budgets](https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html)
- [Amazon FSx for Lustre pricing](https://aws.amazon.com/fsx/lustre/pricing/)