CI should test that distribution build works

Currently the CI does not test whether the `-Pdistribution` profile successfully builds or not. As we want master to always be releasable, we should test this on PRs.