Summary

We have a multi-arch cluster. This cluster have both amd64 nodes (m6a.4xlg) and aarch64 nodes (m6g.4xlg).

We noticed, for Builds, if scheduled on amd64 nodes, are completed successfully. If scheduled on aarch64 nodes, are mostly failed. This is independent to the specific node the pod is scheduled to (i.e., we tried multiple aarch64 nodes).

The error messages look like:

error: mkdir /var/lib/containers/storage/overlay/metacopy-check1398259516/l2: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: changing permissions on file for metacopy check: chmod /var/lib/containers/storage/overlay/metacopy-check271055033/merged/f: no space left on device
error: changing permissions on file for metacopy check: chmod /var/lib/containers/storage/overlay/metacopy-check3798390120/merged/f: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: changing permissions on file for metacopy check: chmod /var/lib/containers/storage/overlay/metacopy-check2121835199/merged/f: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device

Goal

Identify the root cause and find solutions to the issue.

Ref: https://redhat-internal.slack.com/archives/CK1AE4ZCK/p1708356005736639

duplicates

OCPBUGS-29712 Build from same source code works well in amd64 but stuck on aarch64

Assignee:: Kevin Hannon

Reporter:: Bear Chen

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2024/02/20 12:50 PM

Updated:: 2024/02/21 9:22 PM

Resolved:: 2024/02/20 4:21 PM

Details

Description

Summary

Goal

Attachments

Issue Links

Activity

People

Dates

Hide