ECS deploy reported PHONE LIVE while UpdateService had silently failed
A deploy script used the short service name instead of the CloudFormation-generated one; update-service threw ServiceNotFoundException, no rollout happened, but the final health curl hit the old container and printed success. Verification now compares running-task image digest to the pushed ECR digest.
The deploy loop for faangforge (docker build → ECR push → aws ecs update-service --force-new-deployment → wait → curl) printed its success line:
An error occurred (ServiceNotFoundException) when calling the UpdateService operation: Service not found.
Waiter ServicesStable failed: Waiter encountered a terminal failure state: For expression "failures[].reason" we matched expected path: "MISSING" at least once
200 PHONE LIVE (indian device voice)Error first, success line last anyway. The image was in ECR, but no container was replaced.
The chain: the script said --service ds-forge, but CloudFormation generates service names — the real one is ds-forge-Service-wxKLD5XV5SR8 (found via aws ecs list-services). update-service failed → no rollout started → the final curl returned 200 because the OLD container was still happily serving → the script echoed "PHONE LIVE".
Two fixes to the loop:
- Use the full CFN service name (looked up once, hardcoded into the loop).
- After
wait services-stable, read the running task'simageDigestand compare it to the digest ECR reports for the tag just pushed. OnlyDIGEST MATCHcounts as deployed — an HTTP 200 after a deploy only proves something is serving, not that it's the new build.
Earlier deploys in the same session were fine (the running task predated the bad one and matched the previous push), so only one build was stuck in limbo; it went out on the corrected redeploy.