Upgrade And Replacement

Date: 2026-06-23

Purpose

This guide explains how to change a BetterNAT deployment safely.

The short version:

• capacity-only changes are supported in-place,
• provider-owned infrastructure migrations should be reconciled in-place when they do not affect runtime or route/EIP ownership,
• runtime, bootstrap, datapath, HA timing, route, subnet, AMI, and EIP-mode changes require explicit replacement,
• BetterNAT does not currently provide seamless rolling software or AMI upgrades,
• use blue/green replacement for safer upgrades of real workloads.

Change Type Decision Table

Change	Current path
min_size, desired_capacity, or max_size only	In-place apply.
Provider patch upgrade with no config change	In-place provider-owned infrastructure reconciliation only.
IAM policy generated by BetterNAT provider changes	In-place reconciliation by the provider.
Runtime version, AMI, bootstrap, datapath, subnet, route, or EIP-mode change	Replacement.
Important workload replacement	Blue/green replacement.
Disposable environment replacement	terraform apply -replace=... is acceptable if disruption is accepted.

Version Model

BetterNAT has two version layers:

1. Terraform provider version.
2. Gateway runtime version.

The provider version is set in Terraform/OpenTofu:

terraform {
  required_providers {
    betternat = {
      source  = "nowakeai/betternat"
      version = "= 0.2.0"
    }
  }
}

OpenTofu can use the same source = "nowakeai/betternat" address now that the provider is registered in the OpenTofu Registry.

The gateway runtime version is controlled separately:

betternat_version = "v0.2.0"

The provider derives the agent and CLI release artifact URLs and SHA256 checksums for supported runtime versions. The explicit agent_binary_url, agent_binary_sha256, cli_binary_url, and cli_binary_sha256 fields remain available as advanced overrides for private or unreleased builds.

Changing the provider version does not automatically upgrade running gateway nodes.

Compatibility Policy

BetterNAT uses separate compatibility contracts for provider versions and gateway runtime versions.

Provider patch releases are intended to be non-breaking:

• no removal of existing Terraform fields,
• no incompatible Terraform state changes,
• no removal of previously documented betternat_version support,
• no new replacement requirement for unchanged configuration,
• no route/EIP/datapath mutation outside documented provider-owned infrastructure reconciliation.

Provider minor releases may add backward-compatible fields, supported runtime versions, outputs, diagnostics, or safe provider-owned infrastructure migrations.

Runtime patch releases are intended to be non-breaking for the same supported line:

• no incompatible agent config changes,
• no CLI command or output contract removal,
• no metrics name/label removal,
• no incompatible HA coordination record changes,
• no bootstrap contract changes for a provider-supported release line.

Patch releases must not introduce breaking changes. Breaking changes must be reserved for a later minor or major release, called out in release notes, and reflected in this guide.

Provider Runtime Compatibility

For a production-supportable provider line, the provider maintains a compatibility contract for betternat_version.

Use the matching provider/runtime pair shown in the release notes:

Provider version	Supported betternat_version values	Notes
0.2.0	v0.2.0	Terraform surface reset provider/runtime pair.
0.1.0	v0.1.0	First GA provider/runtime pair.

Do not assume every runtime tag is immediately available through betternat_version. Alpha compatibility history belongs in release notes, not in the production upgrade runbook.

If you set an unsupported betternat_version, Terraform should fail with a clear provider error instead of guessing an artifact URL.

Provider upgrades can still reconcile provider-owned infrastructure in place. For example, a newer provider may update the BetterNAT-managed IAM inline policy, create a new coordination table, update tags, or record migration metadata without replacing the gateway nodes.

Supported In-Place Updates

The provider supports user-requested in-place updates for capacity fields:

min_size         = 2
desired_capacity = 3
max_size         = 3

When only these fields change, the provider updates ASG capacity and keeps the existing gateway identity, route ownership, rollback metadata, and runtime configuration.

Use this for:

• adding standby capacity,
• restoring a degraded standby pool,
• shrinking from three nodes to two after a test.

Before applying a capacity change, verify:

terraform plan
terraform output active_instance_ids
terraform output standby_instance_ids

After applying:

terraform apply
terraform output active_instance_ids
terraform output standby_instance_ids

Then check metrics or node-local diagnostics:

sudo betternat doctor --live
sudo betternat datapath ready

Provider-Owned Infrastructure Reconciliation

Provider-owned infrastructure changes are different from user-requested runtime changes.

The provider should reconcile its own supporting infrastructure in place when the change is safe and does not affect the active datapath.

Safe in-place reconciliation examples:

• overwriting the BetterNAT-managed IAM inline policy named betternat-runtime,
• removing permissions that a newer provider no longer needs,
• adding permissions for a new provider-owned coordination backend,
• creating a provider-owned DynamoDB coordination table,
• updating provider-owned tags or metadata,
• recording migration metadata in Terraform state.

These changes must not:

• stop or replace running gateway nodes,
• mutate private route table targets,
• associate or disassociate EIPs,
• change bootstrap user data,
• change the agent runtime config without an explicit runtime rollout.

The provider may only reconcile resources it owns by contract. If you attach your own IAM policy to the BetterNAT role, the provider should leave that user-managed policy alone. The provider can replace the content of its own betternat-runtime inline policy because that policy is generated and owned by BetterNAT.

This means a provider upgrade that tightens permissions should converge the BetterNAT-managed policy on the next apply without requiring a gateway replacement.

The provider records an internal provider_infrastructure_revision value in resource state. When a newer provider needs a safe infrastructure-only migration, it can bump that revision so Terraform plans an in-place update even if the user’s configuration did not change. That update path reconciles provider-owned infrastructure such as IAM policy documents and coordination tables, but it must not roll nodes or mutate route/EIP ownership.

Changes That Require Replacement

The provider intentionally rejects non-capacity updates.

Replacement is required for changes such as:

• ami_id
• ami_channel
• betternat_version
• agent_binary_url
• agent_binary_sha256
• cli_binary_url
• cli_binary_sha256
• loxicmd_binary_url
• loxicmd_binary_sha256
• instance_type
• public_subnet_ids
• private_route_table_ids
• private_cidrs
• datapath_engine
• stable_egress_ip
• ha_profile
• ha_lease_ttl_seconds
• ha_renew_interval_seconds
• route_mode
• route_destination_cidr
• route_target_type
• tags and other installation inputs.

If you try to update these fields in place, Terraform reports:

BetterNAT gateway replacement required

This is deliberate. A silent in-place mutation could leave private routes, EIP association, lease ownership, datapath rules, and rollback metadata inconsistent.

Replacement Options

Option A: Explicit Terraform Replacement

For disposable or non-critical environments, replace the resource explicitly:

terraform apply -replace=betternat_aws_gateway.egress

Use this only when you accept disruption. Active flows may reset, and new flows may fail until the replacement is ready and routes converge.

Recommended checks after replacement:

terraform output betternat_status
terraform output active_instance_ids
terraform output standby_instance_ids
terraform output egress_public_ips

From a private client:

curl -fsS https://checkip.amazonaws.com

On the active node:

sudo betternat doctor --live
sudo betternat datapath ready

Option B: Blue/Green Gateway Replacement

For important workloads, prefer blue/green replacement.

High-level flow:

1. Keep the existing BetterNAT gateway serving production route tables.
2. Deploy a new BetterNAT gateway with a different name.
3. Attach the new gateway to a test private route table first.
4. Verify egress, metrics, HA state, route ownership, and EIP behavior.
5. Migrate selected private route tables to the new gateway.
6. Keep the old gateway during the rollback window.
7. Destroy the old gateway only after rollback is no longer needed.

Example naming:

resource "betternat_aws_gateway" "egress_v1" {
  name = "prod-egress-v1"
  # old runtime or AMI
}

resource "betternat_aws_gateway" "egress_v2" {
  name = "prod-egress-v2"
  # new runtime or AMI
}

Do not attach both gateways to the same private route table at the same time. A private route table default route should have one owner.

Blue/Green Checklist

Before creating the new gateway:

• record the old gateway outputs,
• confirm rollback_route_targets_json contains concrete route targets,
• confirm the old gateway has at least one healthy active node,
• decide which private route tables will be used for testing.

After creating the new gateway:

• verify ASG healthy capacity,
• verify one active node,
• verify standby capacity if desired_capacity >= 2,
• verify betternat_datapath_ready == 1,
• verify betternat_route_target_match == 1,
• verify betternat_public_identity_match == 1 when stable egress IP is enabled,
• run an egress IP probe from a test private client.

Before moving production route tables:

• ensure users understand active connections may reset,
• choose a low-risk window for sensitive workloads,
• prepare the route table IDs and fallback targets needed by the Rollback Guide,
• keep the old NAT Gateway or old BetterNAT gateway available until the rollback window ends.

Rollback During Replacement

If the new gateway fails validation, keep or restore the private route table default route to the old target.

Use the Rollback Guide for manual route restore commands. Also check the stable EIP association if stable_egress_ip = true:

aws ec2 describe-addresses \
  --allocation-ids <eipalloc-id>

See Rollback Guide for the full destroy and rollback procedure.

Desired Capacity Guidance

desired_capacity = 1

This has no warm standby.

Use only for disposable tests or workloads that can tolerate gateway downtime. Any replacement, stop, reboot, or EC2 failure can interrupt new connections until the instance is healthy again.

desired_capacity = 2

This is the recommended HA shape:

• one active,
• one standby,
• automatic failover for owner termination,
• ASG repair for failed instances.

It is still not a seamless software upgrade mechanism.

desired_capacity >= 3

This can provide extra standby capacity and more flexibility for manual operations, but the provider still does not orchestrate standby-first rolling upgrades.

Use it when you want more spare capacity, not because it turns replacement into a fully managed upgrade.

What Terraform Does Not Do Yet

BetterNAT does not yet:

• replace standby nodes before active nodes,
• trigger planned failover,
• ask the active node to step down,
• run ASG instance refresh safely around BetterNAT ownership,
• protect the active owner from ASG scale-in,
• expose an upgrade_strategy field,
• provide a user-triggered planned drain command.

The agent releases its locally owned HA lease on graceful SIGTERM/systemd stop. It also watches IMDS for Spot interruption and ASG target termination state, then completes the ASG termination lifecycle hook after release. These paths help a standby take over without waiting for the full lease TTL when AWS provides notice in time, but they are not yet a complete planned failover or rolling-upgrade workflow. Software and AMI changes are still modeled as explicit replacement, not as a transparent rolling upgrade.

Future Production Direction

A production-grade rolling upgrade should eventually:

1. create a new launch template or AMI version,
2. replace standby nodes first,
3. verify new standby readiness,
4. trigger planned failover to a ready new node,
5. verify lease, route, EIP, metrics, and egress probe,
6. replace the old active after it becomes standby,
7. abort and keep the old active if readiness fails.

This is the target design, not a current feature.

Practical Recommendation

Practical guidance:

• use in-place updates only for min_size, desired_capacity, and max_size,
• use terraform apply -replace=... only for disposable or accepted-disruption environments,
• use blue/green replacement for important route tables,
• keep rollback metadata and emergency route-restore inputs ready,
• do not remove the old gateway until the new gateway has passed egress, HA, datapath, and observability checks.