Moc is the Meta Object Compiler: responsible for the code generated that does the signals and slots stuff and much more. Rcc is the resource compiler: creates a c compilable file to embed binary files like icons (not suite for big files and not the purpose). The Qt VS Tools allows programmers to create, build, debug and run Qt applications from within non-Express versions of Microsoft Visual Studio 2013 and later. The add-in contains project wizards, Qt project import/export support, integrated Qt resource manager and automated build setup for the Qt Meta-Object Compiler, User Interface Compiler.
As of November 2007 we require all new features going into master to beaccompanied with a unit test. Initially we have limited this requirement toqgis_core, and we will extend this requirement to other parts of the code baseonce people are familiar with the procedures for unit testing explained in thesections that follow.
Unit testing is carried out using a combination of QTestLib (the Qt testinglibrary) and CTest (a framework for compiling and running tests as part of theCMake build process). Lets take an overview of the process before we delve intothe details:
There is some code you want to test, e.g. a class or function. Extremeprogramming advocates suggest that the code should not even be written yetwhen you start building your tests, and then as you implement your code you canimmediately validate each new functional part you add with your test. Inpractice you will probably need to write tests for pre-existing code in QGISsince we are starting with a testing framework well after much applicationlogic has already been implemented.
You create a unit test. This happens under
<QGISSourceDir>/tests/src/corein the case of the core lib. The test is basically a client that creates aninstance of a class and calls some methods on that class. It will check thereturn from each method to make sure it matches the expected value. If anyone of the calls fails, the unit will fail.You include QtTestLib macros in your test class. This macro is processed bythe Qt meta object compiler (moc) and expands your test class into arunnable application.
You add a section to the CMakeLists.txt in your tests directory that willbuild your test.
You ensure you have
ENABLE_TESTINGenabled in ccmake / cmakesetup. Thiswill ensure your tests actually get compiled when you type make.You optionally add test data to
<QGISSourceDir>/tests/testdataif yourtest is data driven (e.g. needs to load a shapefile). These test data shouldbe as small as possible and wherever possible you should use the existingdatasets already there. Your tests should never modify this data in situ,but rather make a temporary copy somewhere if needed.You compile your sources and install. Do this using normal
make&&(sudo)makeinstallprocedure.You run your tests. This is normally done simply by doing
maketestafter themakeinstallstep, though we will explain other approaches that offermore fine grained control over running tests.
Right with that overview in mind, we will delve into a bit of detail. We’vealready done much of the configuration for you in CMake and other places in thesource tree so all you need to do are the easy bits - writing unit tests!
Creating a unit test is easy - typically you will do this by just creating asingle .cpp file (no .h file is used) and implement all yourtest methods as public methods that return void. We’ll use a simple test class forQgsRasterLayer throughout the section that follows to illustrate. By conventionwe will name our test with the same name as the class they are testing butprefixed with ‘Test’. So our test implementation goes in a file calledtestqgsrasterlayer.cpp and the class itself will be TestQgsRasterLayer.First we add our standard copyright banner:
Next we start our includes needed for the tests we plan to run. There isone special include all tests should have:
Beyond that you just continue implementing your class as per normal, pullingin whatever headers you may need:
Since we are combining both class declaration and implementation in a singlefile the class declaration comes next. We start with our doxygen documentation.Every test case should be properly documented. We use the doxygen ingroupdirective so that all the UnitTests appear as a module in the generated Doxygendocumentation. After that comes a short description of the unit test andthe class must inherit from QObject and include the Q_OBJECT macro.
All our test methods are implemented as private slots. The QtTest frameworkwill sequentially call each private slot method in the test class. There arefour ‘special’ methods which if implemented will be called at the start of theunit test (initTestCase), at the end of the unit test(cleanupTestCase). Before each test method is called, the init()method will be called and after each test method is called the cleanup()method is called. These methods are handy in that they allow you to allocateand cleanup resources prior to running each test, and the test unit as a whole.
Then come your test methods, all of which should take no parameters andshould return void. The methods will be called in order of declaration. Weare implementing two methods here which illustrate two types of testing.
In the first case we want to generally test if the various parts of the class are working,We can use a functional testing approach. Once again, extreme programmerswould advocate writing these tests before implementing the class. Then asyou work your way through your class implementation you iteratively run yourunit tests. More and more test functions should complete successfully as yourclass implementation work progresses, and when the whole unit test passes, yournew class is done and is now complete with a repeatable way to validate it.
Typically your unit tests would only cover the public API of your class,and normally you do not need to write tests for accessors and mutators. If itshould happen that an accessor or mutator is not working as expected you wouldnormally implement a regression test to check for this.
Next we implement our regression tests. Regression tests should beimplemented to replicate the conditions of a particular bug. For example:
We received a report by email that the cell count by rasters was off by1, throwing off all the statistics for the raster bands.
We opened a bug report (ticket #832)
We created a regression test that replicated the bug using a smalltest dataset (a 10x10 raster).
We ran the test, verifying that it did indeed fail(the cell count was 99 instead of 100).
Then we went to fix the bug and reran the unit test and the regression testpassed. We committed the regression test along with the bug fix. Now ifanybody breakes this in the source code again in the future, we canimmediately identify that the code has regressed.
Better yet, before committing any changes in the future, running our testswill ensure our changes don’t have unexpected side effects - like breakingexisting functionality.
There is one more benefit to regression tests - they can save you time. If youever fixed a bug that involved making changes to the source, and then runningthe application and performing a series of convoluted steps to replicate theissue, it will be immediately apparent that simply implementing your regressiontest before fixing the bug will let you automate the testing for bugresolution in an efficient manner.
To implement your regression test, you should follow the naming convention ofregression<TicketID> for your test functions. If no ticket exists for theregression, you should create one first. Using this approach allows the personrunning a failed regression test easily go and find out more information.
Finally in your test class declaration you can declare privately any datamembers and helper methods your unit test may need. In our case we will declarea QgsRasterLayer* which can be used by any of our test methods. The rasterlayer will be created in the initTestCase() function which is run before anyother tests, and then destroyed using cleanupTestCase() which is run after alltests. By declaring helper methods (which may be called by various testfunctions) privately, you can ensure that they won’t be automatically run by theQTest executable that is created when we compile our test.
That ends our class declaration. The implementation is simply inlined in thesame file lower down. First our init and cleanup functions:
The above init function illustrates a couple of interesting things.
We needed to manually set the QGIS application data path so thatresources such as
srs.dbcan be found properly.Secondly, this is a data driven test so we needed to provide away to generically locate the
tenbytenraster.ascfile. This wasachieved by using the compiler defineTEST_DATA_PATH. Thedefine is created in theCMakeLists.txtconfiguration file under<QGISSourceRoot>/tests/CMakeLists.txtand is available to allQGIS unit tests. If you need test data for your test, commit itunder<QGISSourceRoot>/tests/testdata. You should only commitvery small datasets here. If your test needs to modify the testdata, it should make a copy of it first.
Qt also provides some other interesting mechanisms for data driventesting, so if you are interested to know more on the topic, consultthe Qt documentation.
Next lets look at our functional test. The isValid() test simply checks theraster layer was correctly loaded in the initTestCase. QVERIFY is a Qt macrothat you can use to evaluate a test condition. There are a few other usemacros Qt provide for use in your tests including:
QCOMPARE ( actual, expected )
QEXPECT_FAIL ( dataIndex, comment, mode )
QFAIL ( message )
QFETCH ( type, name )
QSKIP ( description, mode )
QTEST ( actual, testElement )
QTEST_APPLESS_MAIN ( TestClass )
QTEST_MAIN ( TestClass )
QTEST_NOOP_MAIN ()
QVERIFY2 ( condition, message )
QVERIFY ( condition )
QWARN ( message )
Some of these macros are useful only when using the Qt framework for datadriven testing (see the Qt docs for more detail).
Normally your functional tests would cover all the range of functionality ofyour classes public API where feasible. With our functional tests out the way,we can look at our regression test example.
Since the issue in bug #832 is a misreported cell count, writing our test issimply a matter of using QVERIFY to check that the cell count meets theexpected value:
With all the unit test functions implemented, there’s one final thing we need toadd to our test class:
The purpose of these two lines is to signal to Qt’s moc that this is a QtTest(it will generate a main method that in turn calls each test function. The lastline is the include for the MOC generated sources. You should replacetestqgsrasterlayer with the name of your class in lower case.
Rendering images on different environments can produce subtle differences due toplatform-specific implementations (e.g. different font rendering and antialiasingalgorithms), to the fonts available on the system and for other obscure reasons.
When a rendering test runs on Travis and fails, look for the dash link at thevery bottom of the Travis log. This link will take you to a cdash page whereyou can see the rendered vs expected images, along with a “difference” imagewhich highlights in red any pixels which did not match the reference image.
The QGIS unit test system has support for adding “mask” images, which are usedto indicate when a rendered image may differ from the reference image.A mask image is an image (with the same name as the reference image,but including a _mask.png suffix), and should be the same dimensions as thereference image. In a mask image the pixel values indicate how much thatindividual pixel can differ from the reference image, so a black pixel indicatesthat the pixel in the rendered image must exactly match the same pixel in thereference image. A pixel with RGB 2, 2, 2 means that the rendered image can varyby up to 2 in its RGB values from the reference image, and a fully white pixel(255, 255, 255) means that the pixel is effectively ignored when comparing theexpected and rendered images.
A utility script to generate mask images is available asscripts/generate_test_mask_image.py. This script is used by passing it thepath of a reference image (e.g. tests/testdata/control_images/annotations/expected_annotation_fillstyle/expected_annotation_fillstyle.png)and the path to your rendered image.
E.g.
You can shortcut the path to the reference image by passing a partial part ofthe test name instead, e.g.
(This shortcut only works if a single matching reference image is found.If multiple matches are found you will need to provide the full path to thereference image.)
The script also accepts http urls for the rendered image, so you can directlycopy a rendered image url from the cdash results page and pass it to the script.
Be careful when generating mask images - you should always view the generatedmask image and review any white areas in the image. Since these pixels areignored, make sure that these white images do not cover any important portionsof the reference image – otherwise your unit test will be meaningless!
Similarly, you can manually “white out” portions of the mask if you deliberatelywant to exclude them from the test. This can be useful e.g. for tests which mixsymbol and text rendering (such as legend tests), where the unit test is notdesigned to test the rendered text and you don’t want the test to be subject tocross-platform text rendering differences.
To compare images in QGIS unit tests you should use the classQgsMultiRenderChecker or one of its subclasses.
Qt Meta Object Compiler
To improve tests robustness here are few tips:
Qt Meta Object Compiler Install Flash Card
Disable antialiasing if you can, as this minimizes cross-platform renderingdifferences.
Make sure your reference images are “chunky”… i.e. don’t have 1 px widelines or other fine features, and use large, bold fonts (14 points or moreis recommended).
Sometimes tests generate slightly different sized images (e.g. legendrendering tests, where the image size is dependent on font rendering size -which is subject to cross-platform differences). To account for this,use
QgsMultiRenderChecker::setSizeTolerance()and specify the maximumnumber of pixels that the rendered image width and height differ from thereference image.Don’t use transparent backgrounds in reference images (CDash does notsupport them). Instead, use
QgsMultiRenderChecker::drawBackground()to draw a checkboard pattern for the reference image background.When fonts are required, use the font specified in
QgsFontUtils::standardTestFontFamily()(“QGIS Vera Sans”).
If travis reports errors for new images (for instance due toantialiasing or font differences), the scriptparse_dash_results.pycan help you when you are updating the local test masks.
Adding your unit test to the build system is simply a matter of editing theCMakeLists.txt in the test directory, cloning one of the existing testblocks, and then replacing your test class name into it. For example:
We’ll run through these lines briefly to explain what they do, but if you arenot interested, just do the step explained in the above section.
Let’s look a little more in detail at the individual lines. First we define thelist of sources for our test. Since we have only one source file (following themethodology described above where class declaration and definition are in thesame file) its a simple statement:
Since our test class needs to be run through the Qt meta object compiler (moc)we need to provide a couple of lines to make that happen too:
Next we tell cmake that it must make an executable from the test class.Remember in the previous section on the last line of the class implementation weincluded the moc outputs directly into our test class, so that will give it(among other things) a main method so the class can be compiled as anexecutable:
Next we need to specify any library dependencies. At the moment, classes havebeen implemented with a catch-all QT_LIBRARIES dependency, but we will beworking to replace that with the specific Qt libraries that each class needsonly. Of course you also need to link to the relevant qgis libraries asrequired by your unit test.
Next we tell cmake to install the tests to the same place as the qgis binariesitself. This is something we plan to remove in the future so that the tests canrun directly from inside the source tree.
Finally the above uses ADD_TEST to register the test with cmake / ctest.Here is where the best magic happens - we register the class with ctest. If yourecall in the overview we gave in the beginning of this section, we are usingboth QtTest and CTest together. To recap, QtTest adds a main method to yourtest unit and handles calling your test methods within the class. It alsoprovides some macros like QVERIFY that you can use as to test forfailure of the tests using conditions. The output from a QtTest unit test is anexecutable which you can run from the command line. However when you have asuite of tests and you want to run each executable in turn, and better yetintegrate running tests into the build process, the CTest is what we use.
To build the unit test you need only to make sure that ENABLE_TESTS=truein the cmake configuration. There are two ways to do this:
Run
ccmake..( orcmakesetup..under windows) and interactively settheENABLE_TESTSflag toON.Add a command line flag to cmake e.g.
cmake-DENABLE_TESTS=true..
Other than that, just build QGIS as per normal and the tests should build too.
The simplest way to run the tests is as part of your normal build process:
The maketest command will invoke CTest which will run each test that wasregistered using the ADD_TEST CMake directive described above. Typical outputfrom maketest will look like this:
If a test fails, you can use the ctest command to examine more closely why itfailed. Use the -R option to specify a regex for which tests you want to runand -V to get verbose output:
C++ tests are ordinary applications. You can run them from the build folderlike any executable.
These tests also take command line arguments.This makes it possible to run a specific subset of tests:
For C++ unit tests, QtCreator automatically adds run targets, so you can startthem from the debugger.
If you go to Projects and there to the Build & Run –> Desktop Run tab, you canalso specify command line parameters that will allow a subset of the tests to be runinside a .cpp file in the debugger.
It’s also possible to start Python unit tests from QtCreator with GDB. Forthis, you need to go to Projects and choose Run underBuild & Run.Then add a new Runconfiguration with the executable /usr/bin/python3and the Command line arguments set to the path of the unit test python file,e.g./home/user/dev/qgis/QGIS/tests/src/python/test_qgsattributeformeditorwidget.py.
Now also change the RunEnvironment and add 3 new variables:
Variable | Value |
PYTHONPATH | [build]/output/python/:[build]/output/python/plugins:[source]/tests/src/python |
QGIS_PREFIX_PATH | [build]/output |
LD_LIBRARY_PATH | [build]/output/lib |
Replace [build] with your build directory and [source] withyour source directory.
Well that concludes this section on writing unit tests in QGIS. We hope youwill get into the habit of writing test to test new functionality and to checkfor regressions. Some aspects of the test system (in particular theCMakeLists.txt parts) are still being worked on so that the testing frameworkworks in a truly platform independent way.