Answer a question

I'm trying to push a multi-arch docker image to docker hub and run into errors (opened docker issue at https://github.com/docker/distribution/issues/3100):

❯ docker build -t andig/evcc:latest-amd64 -f linux-amd64-.Dockerfile .
❯ docker build -t andig/evcc:latest-arm.v6 -f linux-arm-6.Dockerfile .
❯ docker push andig/evcc:latest-amd64
❯ docker push andig/evcc:latest-arm.v6
❯ docker manifest create --amend andig/evcc:latest andig/evcc:latest-amd64 andig/evcc:latest-arm.v6
❯ docker manifest annotate andig/evcc:latest andig/evcc:latest-amd64 --os=linux --arch=amd64
❯ docker manifest annotate andig/evcc:latest andig/evcc:latest-arm.v6 --os=linux --arch=arm --variant=v6
❯ docker manifest push andig/evcc:latest

failed to put manifest docker.io/andig/evcc:latest: errors:
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry
manifest blob unknown: blob unknown to registry

Searching around shows error might be related to "foreign layers" (https://issues.sonatype.org/browse/NEXUS-11865) but it's unclear how that might happen building an image based on Alpine.

As the docker github ticket and a forum question didn't get any response- what else could I do for further analysis? E.g., how could I find out which layer it's complaining about?

Answers

As I suspected a problem with a somehow corrupted manifest that I might have pushed to docker hub I've started retrying the process with a clean, local registry. Following the steps led me to the same error message.

At this stage I've started another round of googling which not only suggested to use --purge for cleaning the manifest locally (which didn't help with this issue) but also to remove the local manifest cache by deleting ~/.docker/manifests.

After removing the manifests folder both registry pushes- local and docker hub- started working.