A better way to run QUnit tests.

July 25, 2018, 10:49 p.m. wschaub django TDD

There is a much simpler and better way to run QUnit tests for Test-Driven Development with Python

After trying to use the xUnit plugin for Jenkins (which I will be writing about more in depth soon) I ran into the problem of generating JUnit reports for QUnit tests. This article is about my solution to that problem.

I'm currently working on a blog post about configuring Jenkins for the book. Part of that configuration is adding nice test reports to Jenkins with the Jenkins xUnit plugin.(Thanks to John Fitzpatrick's comment on Chapter 24) QUnit presented special problems, particularly in the way that the headless testing was run in the book with a special QUnit phantomjs runner. I tried looking at a JUnit plugin for QUnit that is displayed on the home page of QUnit but that lead to me modifying the PhantomJS runner and was a bit hacky and not very satisfying.

While looking for other JUnit plugins for QUnit I found a much simpler way to do it using Grunt.js using a much nicer plugin for same called grunt-qunit-junit

The biggest obstacle in my way however happened to be this bug report for the Jenkins xUnit plugin since neither python nor any JUnit report generator I could find for QUnit would generate XML that the plugin considered suitable.

This started a dive into XSLT (Thank you very much larssjk for attaching an XSLT template for pytest to that bug report!) so that I could transform the XML for the QUnit reports in a similar way to the pytest example. This is what I came up with after struggling with XSLT for about a day It's essentially the pytest XSL with handling for an enclosing <testsuites> </testsuites> and we remove special handling for the classname attribute on testcase tags.

I promise you this was the only unpleasant thing involved in the whole process, mainly because XML and XSLT were not something I've previously done much with. You can find my repo on GitHub here that I fed into Jenkins to look over the test results (failing on purpose and otherwise)

grunt-qunit-junit.xsl

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xs="http://www.w3.org/2001/XMLSchema" exclude-result-prefixes="xs" version="1.0">
  <xsl:output method="xml" indent="yes" encoding="UTF-8"
    cdata-section-elements="system-out system-err failure"/>
 <xsl:decimal-format decimal-separator="." grouping-separator=","/>
  <xsl:template match = "testsuites">
     <testsuites>
                <xsl:apply-templates select = "testsuite" />
    </testsuites> 
   </xsl:template>  
    <xsl:template match="testsuite">
    <testsuite>
      <xsl:if test="@name">
        <xsl:attribute name="name">
          <xsl:value-of select="@name"/>
        </xsl:attribute>
      </xsl:if>
      <xsl:if test="@tests">
        <xsl:attribute name="tests">
          <xsl:value-of select="@tests"/>
        </xsl:attribute>
      </xsl:if>
      <xsl:if test="@errors">
        <xsl:attribute name="errors">
          <xsl:value-of select="@errors"/>
        </xsl:attribute>
      </xsl:if>
      <xsl:if test="@failures">
        <xsl:attribute name="failures">
          <xsl:value-of select="@failures"/>
        </xsl:attribute>
      </xsl:if>
      <xsl:if test="@skips">
        <xsl:attribute name="skipped">
          <xsl:value-of select="@skips"/>
        </xsl:attribute>
      </xsl:if>
      <xsl:if test="@time">
        <xsl:attribute name="time">
          <xsl:value-of select="@time"/>
        </xsl:attribute>
      </xsl:if>

      <xsl:apply-templates select="testcase"/>
    </testsuite>
  </xsl:template>

  <xsl:template match="testcase">
    <testcase>
      <xsl:if test="@classname">
        <xsl:attribute name="classname">
         <xsl:value-of select="@classname"/>
        </xsl:attribute>
      </xsl:if>
      <xsl:if test="@name">
        <xsl:attribute name="name">
          <xsl:value-of select="@name"/>
        </xsl:attribute>
      </xsl:if>
      <xsl:if test="@time">
        <xsl:attribute name="time">
          <xsl:value-of select="@time"/>
        </xsl:attribute>
      </xsl:if>
      <xsl:if test="error">
        <xsl:copy-of select="error"/>
      </xsl:if>
      <xsl:if test="failure">
        <xsl:copy-of select="failure"/>
      </xsl:if>
    </testcase>
  </xsl:template>
</xsl:stylesheet>

Now onto the simple stuff. instead of calling phantomjs lists/static/tests/runner.js lists/static/tests/tests.html we can simply have a file called Gruntfile.js and a file called package.json inside the root of our django project. and we can add more tests to the Gruntfile as we go along. it simplifies generating our test reports for Jenkins and makes adding tests much simpler without having to constantly modify our Jenkins config. (Trust me you really don't want to have to change that very often if you can avoid it. )

Here are the changes I've done to my testinggoat repo (it's in a private project so I can't link to it. )

deleted lists/static/tests/runner.js
added Gruntfile.js
added package.json
checked in pytest-xunit.xsl
checked in grunt-qunit-junit.xsl
modified lists/static/tests/tests.html to give a module name in our QUnit tests so we can more easily see the QUnit tests in our reports page in Jenkins.

lists/static/tests/tests.html

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width">
  <title>Javascript tests</title>
  <link rel="stylesheet" href="qunit-2.6.1.css">
</head>
<body>
  <div id="qunit"></div>
  <div id="qunit-fixture">
    <form>
      <input name="text">
      <div class="has-error">Error Text</div>
    </form>
  </div>
  <script src="../jquery-3.3.1.js"></script>
  <script src="../list.js"></script>
  <script src="qunit-2.6.1.js"></script>
  <script>
QUnit.module( "QUnit/lists_page_QUnit_Tests" );
QUnit.test("errors should be hidden on keypress", function(assert) {
  window.Superlists.initialize();
  $('input[name="text"]').trigger('keypress');
  assert.equal($('.has-error').is(':visible'), false);
});
QUnit.test("errors aren't hidden if there is no keypress", function(assert) {
  window.Superlists.initialize();
  assert.equal($('.has-error').is(':visible'), true);
});
  </script>
</body>
</html>

The important part is QUnit.module( "QUnit/lists_page_QUnit_Tests" ); This essentially labels our test in the reports.

Gruntfile.js

This file defines a task called test (called with grunt test on the command line) that runs our qunit tests for us and produces JUnit reports in ./tmp/QUnit/.xml Our regular python tests are in ./tmp/.xml but I will cover that in a different post. You can get get details on getting the reports for python in the comments section on Chapter 24.

To add more tests you simply add a new element to the all: array in the qunit: section pointing to the html file that contains your QUnit tests. reportts are saved in files named TEST-filename.xml inside the QUnit folder in tmp so it will be a good idea to name the file something more descriptive than just tests.html in the future.

module.exports = function( grunt ) {
    grunt.initConfig( {
        qunit: {

            all: [
                    'lists/static/tests/tests.html'
                 ]
        },
        qunit_junit: {
            options: {
                dest: './tmp/QUnit'
                // Task-specific options go here.
            }
        },
    } );
    grunt.loadNpmTasks( "grunt-contrib-qunit" );
    grunt.loadNpmTasks('grunt-qunit-junit');
    grunt.registerTask('test', [ 'qunit_junit', 'qunit']);
};

package.json

{
    "devDependencies": {
        "grunt": "^1.0.1",
        "grunt-contrib-qunit": "^1.2.0",
        "grunt-qunit-junit": "^0.3.1"
    }
}

This file is for the benefit of npm. when we define our build on the jenkins server (or if you want to run the tests locally on your own computer after having ran npm -g install grunt-cli ) you simply run npm install inside of your django project directory (the one that has package.json, Gruntfile.js and manage.py) and then run grunt test on the Jenkins server I just run grunt from inside the node_modules/grunt/bin folder because I haven't installed the grunt command globally there.

example of running grunt test locally

wschaub@T530:~/skunkworks/meow/testinggoat/python-tdd-book$ grunt test
Running "qunit_junit" task
>> XML reports will be written to ./tmp/QUnit

Running "qunit:all" (qunit) task
Testing lists/static/tests/tests.html ..OK
>> 2 tests completed with 0 failed, 0 skipped, and 0 todo. 
>> 2 assertions (in 59ms), passed: 2, failed: 0

Done.

the part of my Jenkins build script that does this looks like:

Virtualenv Builder script snippet

if [ -f Gruntfile.js ]; then
   npm install
   ./node_modules/grunt/bin/grunt test 
fi

we also add rm -rf tmp to our jenkins build to be sure that we remove test reports from previous runs.

Finally here's a screenshot of what the xUnit report config looks like on my Superlists project in Jenkins. it's worth nothing that if you followed the book instructions to the letter you will need to remove the python-tdd-book part of the paths you see here. I decided not to put the django root inside the root of my git repo because I have a Vagrantfile and some ansible provisioning stuff inside of there for spinning up a dev environment that's isolated from my main Linux workstation.

Finally a look at what the test reports look like:

Superlists test results main page

Superlists test results Qunit page

This is what happens when you click the Qunit link on the main page:

grunt-test project's page with a bunch of passing and purposely failing tests.

(I used thisproject to test what I've documented here inside jenkins)

I hope someone finds this information useful. I'm working on a much more in depth article on Jenkins but I wanted to share this part while it was still fresh in my head.

Blog Search

Tags
TDD django python python-tdd-book bootstrap-blog wagtail github testinggoat jenkins Grunt.js QUnit XSLT xrandr KDE xorg pdp8 blinkenbone PLATO NOS irata.online cyber1.org dtcyber CYBIS pytorch AI avx2