making sure you see all your failed tests in Jenkins

Jenkins runs it's build shell scripts with /bin/sh -xe which means any command that runs tests that has failing tests will abort the entire build and you likely won't see the JUnit reports for your entire test suite.

I've come up with some simple shell functions that will allow all of our test commands to run and then still allow the build to still fail like it's supposed to when we are ready for it.

Below is my current build script for Test-Driven Development with Python

It's two shell functions: test_wrapper which shuts off the -e, runs the command and sets a global variable if we have a non zero exit. and test_status which simply returns that variable as an exit status. If everything goes well then nothing happens.

If we have even one failing test run we will still fail the build but it will be after we have run the test commands (unit, QUnit and functional_tests) I do not wrap anything that is not test related so its more obvious that I have build failures.

The settings I use for the xUnit plugin will still fail the build when it sees the failing tests but with this setup I will be able to see a complete report of all failing tests instead of just the reports up until the failing test command.

I still have plans to move everything to a Jenkinsfile and do pipelines but until then this isn't the world's worst stopgap.

global_failure_state=0
#run test with the error trap disabled
#and if we get an error set a flag to be used later
test_wrapper () {
    set +e
    $*
    tmp=$?
    if [ $tmp -gt 0 ]; then
        global_failure_state=$tmp
    fi
    set -e
    return 0
}

#run this at the end of the testing phase to cause
#the build to fail if any of our tests caught an error.
test_status () {
    return $global_failure_state
}

rm -rf python-tdd-book/tmp
mkdir -p python-tdd-book/tmp
cd python-tdd-book
pip install -r requirements.txt
pip install fabric3 unittest-xml-reporting
XUNIT_REPORT=1
export XUNIT_REPORT
test_wrapper python manage.py test lists accounts
#big ball of wax to support older branches... 
if [ -f Gruntfile.js ]; then
   npm install
   test_wrapper ./node_modules/grunt/bin/grunt test 
fi
test_wrapper python manage.py test functional_tests
test_status