GKE Artifact Registry Auth Expiry — The Silent Push Failure
Docker's exit code after a push is not a reliable success signal when GCP auth tokens expire mid-build. The only ground truth is a registry-side digest check.
The problem
Google Artifact Registry auth tokens expire after one hour. If you run a long build session — say, a Docker build that takes 20 minutes followed by a push — and your gcloud auth configure-docker was done at the start of the session, the push can fail with denied: Unauthenticated request even though the build succeeded. The worse case: docker push reports a successful layer push but the final manifest push fails, and the overall command exits 0 because the layers were cached. The image appears to push. It didn't.
The downstream symptom is ImagePullBackOff on kubectl rollout — a confusing error that reads like a pull problem rather than a push problem.
The approach
The reliable pattern is to re-authenticate immediately before every push, not once at the start of a session:
gcloud auth configure-docker asia-southeast1-docker.pkg.dev --quiet
gcloud auth print-access-token | docker login \
-u oauth2accesstoken \
--password-stdin \
asia-southeast1-docker.pkg.dev
docker push asia-southeast1-docker.pkg.dev/project/repo/image:tag
# Verify the push actually landed — don't trust exit code alone
gcloud artifacts docker images describe \
asia-southeast1-docker.pkg.dev/project/repo/image:tag \
--format="value(image_summary.digest)"The gcloud auth print-access-token | docker login approach uses a fresh short-lived token rather than the credential helper, which can cache a stale token. The final gcloud artifacts docker images describe confirms the digest is live in the registry — this is the only reliable proof the push succeeded.
For kubectl set image deployments, always follow with kubectl rollout status and watch for ImagePullBackOff before declaring success:
kubectl set image deployment/my-app container=image:tag
kubectl rollout status deployment/my-app --timeout=3mWhat I learned
Docker's exit code after a push is not a reliable success signal when auth issues are involved. The layers are pushed first (and they may already be cached in the registry), and the final manifest write is what requires a valid auth token. If the token expired between layer push and manifest commit, docker may report partial success or exit 0 depending on the exact timing and caching state. The only ground truth is a registry-side digest check. This is especially relevant in background build jobs where the auth was established before the job started.