author | Brendan Cully <brendan@kublai.com> |
Wed, 27 Sep 2006 09:10:28 -0700 | |
changeset 3166 | 3466bd7b9754 |
parent 2977 | f63667f694de |
child 3192 | 096f1c73cdc3 |
permissions | -rw-r--r-- |
331 | 1 |
A simple testing framework |
2 |
||
332 | 3 |
To run the tests, do: |
4 |
||
5 |
cd tests/ |
|
2207
8a2a7f7d9df6
Delete the shell version of run-tests
Stephen Darnell <stephen@darnell.plus.com>
parents:
1933
diff
changeset
|
6 |
python run-tests.py |
332 | 7 |
|
331 | 8 |
This finds all scripts in the test directory named test-* and executes |
9 |
them. The scripts can be either shell scripts or Python. Each test is |
|
10 |
run in a temporary directory that is removed when the test is complete. |
|
11 |
||
12 |
A test-<x> succeeds if the script returns success and its output |
|
13 |
matches test-<x>.out. If the new output doesn't match, it is stored in |
|
14 |
test-<x>.err. |
|
15 |
||
16 |
There are some tricky points here that you should be aware of when |
|
17 |
writing tests: |
|
18 |
||
2283
e506c14382fd
deprecate 'update -m'. use 'merge' instead.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2207
diff
changeset
|
19 |
- hg commit and hg merge want user interaction |
331 | 20 |
|
761
0fb498458905
Change all references to -t --text commit message to -m and --message.
Andrew Thompson <andrewkt@aktzero.com>
parents:
350
diff
changeset
|
21 |
for commit use -m "text" |
2283
e506c14382fd
deprecate 'update -m'. use 'merge' instead.
Vadim Gelfer <vadim.gelfer@gmail.com>
parents:
2207
diff
changeset
|
22 |
for hg merge, set HGMERGE to something noninteractive (like true or merge) |
331 | 23 |
|
24 |
- changeset hashes will change based on user and date which make |
|
25 |
things like hg history output change |
|
26 |
||
1933
7544700fd931
Use 'hg ci -d "1000000 0"' in tests to circumvent problem with leading zero.
Thomas Arendsen Hein <thomas@intevation.de>
parents:
814
diff
changeset
|
27 |
use commit -m "test" -u test -d "1000000 0" |
331 | 28 |
|
29 |
- diff will show the current time |
|
30 |
||
2935 | 31 |
use hg diff | sed -e "s/\(+++ [a-zA-Z0-9_/.-]*\).*/\1/" \ |
32 |
-e "s/\(--- [a-zA-Z0-9_/.-]*\).*/\1/" |
|
33 |
to strip dates |
|
2976
61fcd9fac434
Make tests append settings to $HGRCPATH instead of $HGTMP/.hgrc
Thomas Arendsen Hein <thomas@intevation.de>
parents:
2935
diff
changeset
|
34 |
|
61fcd9fac434
Make tests append settings to $HGRCPATH instead of $HGTMP/.hgrc
Thomas Arendsen Hein <thomas@intevation.de>
parents:
2935
diff
changeset
|
35 |
- You can append your own hgrc settings to the file that the environment |
61fcd9fac434
Make tests append settings to $HGRCPATH instead of $HGTMP/.hgrc
Thomas Arendsen Hein <thomas@intevation.de>
parents:
2935
diff
changeset
|
36 |
variable HGRCPATH points to. This file is cleared before running a test. |
2977 | 37 |
|
2968
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
38 |
You also need to be careful that the tests are portable from one platform |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
39 |
to another. You're probably working on Linux, where the GNU toolchain has |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
40 |
more (or different) functionality than on MacOS, *BSD, Solaris, AIX, etc. |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
41 |
While testing on all platforms is the only sure-fire way to make sure that |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
42 |
you've written portable code, here's a list of problems that have been |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
43 |
found and fixed in the tests. Another, more comprehensive list may be |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
44 |
found in the GNU Autoconf manual, online here: |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
45 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
46 |
http://www.gnu.org/software/autoconf/manual/html_node/Portable-Shell.html |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
47 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
48 |
sh: |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
49 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
50 |
The Bourne shell is a very basic shell. /bin/sh on Linux is typically |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
51 |
bash, which even in Bourne-shell mode has many features that Bourne shells |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
52 |
on other Unix systems don't have (and even on Linux /bin/sh isn't |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
53 |
guaranteed to be bash). You'll need to be careful about constructs that |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
54 |
seem ubiquitous, but are actually not available in the least common |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
55 |
denominator. While using another shell (ksh, bash explicitly, posix shell, |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
56 |
etc.) explicitly may seem like another option, these may not exist in a |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
57 |
portable location, and so are generally probably not a good idea. You may |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
58 |
find that rewriting the test in python will be easier. |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
59 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
60 |
- don't use pushd/popd; save the output of "pwd" and use "cd" in place of |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
61 |
the pushd, and cd back to the saved pwd instead of popd. |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
62 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
63 |
- don't use math expressions like let, (( ... )), or $(( ... )); use "expr" |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
64 |
instead. |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
65 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
66 |
grep: |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
67 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
68 |
- don't use the -q option; redirect stdout to /dev/null instead. |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
69 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
70 |
- don't use extended regular expressions with grep; use egrep instead, and |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
71 |
don't escape any regex operators. |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
72 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
73 |
sed: |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
74 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
75 |
- make sure that the beginning-of-line matcher ("^") is at the very |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
76 |
beginning of the expression -- it may not be supported inside parens. |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
77 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
78 |
echo: |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
79 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
80 |
- echo may interpret "\n" and print a newline; use printf instead if you |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
81 |
want a literal "\n" (backslash + n). |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
82 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
83 |
false: |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
84 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
85 |
- false is guaranteed only to return a non-zero value; you cannot depend on |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
86 |
it being 1. On Solaris in particular, /bin/false returns 255. Rewrite |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
87 |
your test to not depend on a particular return value, or create a |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
88 |
temporary "false" executable, and call that instead. |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
89 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
90 |
diff: |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
91 |
|
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
92 |
- don't use the -N option. There's no particularly good workaround short |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
93 |
of writing a reasonably complicated replacement script, but substituting |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
94 |
gdiff for diff if you can't rewrite the test not to need -N will probably |
4cdb68d7eb92
patch queue: portability.notes
Danek Duvall <danek.duvall@sun.com>
parents:
2935
diff
changeset
|
95 |
do. |