Testing TWiki the Perl way
In the current release, we stumbled over a problem which only manifests itself on Windows (
POSIX::tzset
not available). Well, one could argue that running TWiki on Windows is a bad idea anyway, or should be done using the TWiki VM. On the other hand,
TWikiPersonal works quite fine, and I wouldn't want to abandon it.
Running the test suite on Windows would have revealed the problem, because
TimeTests.pm
contains the bad call to
tzset
, so we know that it hasn't been done prior to the release. And here's why: It is way to tedious. TWiki testing requires a lot of environment which isn't easily transferred to Windows. Right now I haven't found a way to get the unit tests running outside an
SVN checkout, and I am definitely not going to run a complete checkout on Windows.
On the other hand, Strawberry perl already brings the Perl test tools to Windows, for example the command
prove
. I do other Perl stuff with
prove
, so I would love to run
prove
on a functional TWiki installation, not only on Windows, but also on
SVN.
The first shot
How many lines of code does it take at minimum to run the complete
TWikiSuite
through
prove
in a
SVN checkout? Answer: four.
Create a file
.../core/t/TWikiSuite.t
:
use Test::Simple tests => 1;
chdir "test/unit";
my $result = `perl ../bin/TestRunner.pl TWikiSuite.pm 2>&1`;
ok($result =~ /All tests passed \(\d+\)/s, $result);
Then, from the
core
directory, run:
prove -vl
The
-v
switch makes
prove
log the tests. The
-l
switch adds
./lib
to
@INC
, exactly what we need when we're in the
core
directory of a
SVN checkout, or in the root of an installation. That sorta
works, but isn't very friendly because it runs the whole suite without giving feedback. If all tests pass, that's fine, but if there's a failure, it is too difficult to find out which of the tests failed. With a few more lines, mimicking the collection of tests in today's
TWikiSuite.pm
, this can be fixed.
Next steps
I plan to implement a reasonably fast way to create a "testable windows installation", leveraging the already existing
twikidev collect
and
prove
. In my own environment, I have a CIFS mounted windows drive to which I can collect the installation, so this would be a two-command trick:
- From SVN, collect TWiki plus the unit tests to a directory on CIFS
- On windows, start
prove
from the root of this directory.
Would that be interesting for others, i.e. a candidate for the TWiki
SVN?
--
Contributors: Harald Jörg - 2015-12-11
Discussion
Standardization is good.
--
Peter Thoeny - 2015-12-17
As ever so often, the second step turns out to be considerably harder than the first. Up to now I've encountered a challenge: In the current implementation, TWiki's unit test environment needs to be deeply nested with the TWiki runtime.
- It explicitly requires
setlib.cfg
- but why? We are running tests, not a TWiki. I accept that the test cases need to get access to the TWiki locations. There are several (too many) mechanisms for that in different areas (setlib.cfg
, environment variables TWIKI_LIBS
and TWIKI_ROOT
), which is messy enough. But the test driver should not depend on code of the test object.
- It explicitly requires
TWiki.pm
. In addition to the "why" question from above, and to the huge amount of stuff this imports, this creates another dependency: TWiki.pm
requires LocalSite.cfg
. So I need a working installation before I can run unit tests? It should definitely be the other way around. Even for a regression test suite, the test driver should be independent of the test object.
- Importing
LocalSite.cfg
from a "running" environment creates some randomness on the actual configuration under test. Again and again test cases need to be fixed to make local changes to the configuration (most notably: switch off CryptToken
).
In full
SVN installations, the usual practice is to pseudo-install
UnitTestContrib
and
TextFixturePlugin
, which is ugly enough. There's still some way to go before the TWiki modules (the test ibject) and the test driver can sit on independent locations. I think I'll start committing after a successful run of the (maybe modified) suite under Windows, and on Raspberry.
--
Harald Jörg - 2016-01-08
The next experiment in this area has been motivated by a support question (
Comment Plugin: Use emailto-parameter) and a bug report (
Bugs:Item7716). I wanted to test the suspicious
sub expandEmail
, but I did
not want to have an email sent for every test case, nor did I want to register users in my
SVN TWiki, nor did I want to install
SendMailPlugin
. So this is how it works:
- Create a directory
.../trunk/SendMailPlugin/t
- Write a CPAN:Test::More conforming test case
.../trunk/SendMailPlugin/t/expandEmail.t
To run the test, change directory to
.../trunk/SendMailPlugin
and say to your computer:
-
prove -lv -I../core/lib
- Explanation of the options:
-
-l
: shorthand for -Ilib
. As in Perl, add the lib
directory, in this case, .../trunk/SendMailPlugin/lib
, to @INC
.
-
-v
: verbose. Show the short comment provided for each test case (can be omitted).
-
-I../core/lib
: Picks also the core
library (plus installed plugins) to @INC
.
More variations:
-
prove -lv -I../core/lib t/expandEmail.t
- Yes, you can run an individual test file as well.
-
perl -d -Ilib -I../core/lib t/expandEmail.t
- If a test case runs inexplicably wrong, just run it through the debugger.
On the plus side,
- These are what I call unit tests. Running them still uses practically all of TWiki modules, though.
- I find the invocation, and especially the debugging, a lot easier than with TWiki's
Test::Unit
stuff. I haven't even figured out yet how to run unit tests for an extension.
On the minus side,
- To get the tests to run, some TWiki functions need to be simulated by mock functions. For this first step, I just wrote them to the test case, maybe this should be moved to a test library.
- Writing
Test::More
stuff is different from writing Test::Unit
stuff. For core, or for extensions which already have a Test::Unit
suite, the duplication of text fixtures might bring more effort than profit.
I'll commit the test file. It serves to verify that
SendMailPlugin
had a bug before revision 29909, and that it is fixed now. For me, it is sort of an alpha stage whether using traditional Perl testing is a good idea - maybe we'll find out that it isn't.
--
Harald Jörg - 2016-01-20
As a next step, I've attached four test cases which I am using a lot for the experiments to run TWiki with new Perl versions. In a
SVN checkout, create a directory named
t
under your
core
, put these files in, and then from
core
run
prove -vl
.
- TWikiSuite.t is a wrapper around
TWikiSuite.pm
(1186 test cases - this takes some time)
- compile_all.t compiles everything under
core
. This needs the Perl module Test::Compile
plus a boatload of Perl modules required to compile the stuff we have in core/lib/CPAN/lib
.
- compile_warnings.t tries to capture warnings in the compilation step and has been very helpful to chase the "unescaped { in regex" stuff coming with Perl 5.24. It does not detect warnings which occur only during runtime.
- search_bugs.t is a minimalistic unit test to underpin the diagnosis for Bugs:Item7823 and Bugs:Item7824.
Later, this should be pushed to
SVN... either directly or as a developer-only contrib.
--
Harald Jörg - 2017-10-20
Nice progress Harald!
--
Peter Thoeny - 2017-10-22