org.apache.tools.ant.taskdefs.Java setOutput method call doesn't work in runSWTBotInstallRoutine in testInstall.groovy script. It means that the output from the process is not redirected to the temporary output file. The temporary output file is then empty and the errors cannot be detected. And it also means that the testInstall script can return SUCCESS even if there are errors.

This issue is very well hidden, because testInstall.groovy script is mainly used in jenkins jobs and all of the jobs publish JUnit reports which will change the job status into yellow according to the produced xml report.
But if you uncheck the "Publish JUnit report" option, then the job can be blue even if there are test errors.