Mercurial > repos > blastem

annotate comparetests.py @ 1383:18cf08cd67e0

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Updated changelog for 0.5.0 release
author Michael Pavone <pavone@retrodev.com>
date Tue, 30 May 2017 23:39:09 -0700
parents 0f32f52fc98e
children 18555c44a5e7
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
1 #!/usr/bin/env python
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
2 from glob import glob
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
3 import subprocess
220
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
4 from sys import exit,argv
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
5
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
6 prefixes = []
224
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
7 skip = set()
220
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
8 for i in range(1, len(argv)):
224
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
9 if '.' in argv[i]:
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
10 f = open(argv[i])
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
11 for line in f:
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
12 parts = line.split()
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
13 for part in parts:
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
14 if part.endswith('.bin'):
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
15 skip.add(part)
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
16 f.close()
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
17 print 'Skipping',len(skip),'entries from previous report.'
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
18 else:
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
19 prefixes.append(argv[i])
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
20
572
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
21 def print_mismatch(path, b, m):
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
22 blines = b.split('\n')
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
23 mlines = m.split('\n')
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
24 if len(blines) != len(mlines):
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
25 print '-----------------------------'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
26 print 'Unknown mismatch in', path
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
27 print 'blastem output:'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
28 print b
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
29 print 'musashi output:'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
30 print m
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
31 print '-----------------------------'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
32 return
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
33 prevline = ''
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
34 differences = []
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
35 flagmismatch = False
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
36 regmismatch = False
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
37 for i in xrange(0, len(blines)):
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
38 if blines[i] != mlines[i]:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
39 if prevline == 'XNZVC':
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
40 differences.append((prevline, prevline))
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
41 flagmismatch = True
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
42 else:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
43 regmismatch = True
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
44 differences.append((blines[i], mlines[i]))
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
45 prevline = blines[i]
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
46 if flagmismatch and regmismatch:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
47 mtype = 'General'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
48 elif flagmismatch:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
49 mtype = 'Flag'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
50 elif regmismatch:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
51 mtype = 'Register'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
52 else:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
53 mtype = 'Unknown'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
54 print '-----------------------------'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
55 print mtype, 'mismatch in', path
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
56 for i in xrange(0, 2):
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
57 print 'musashi' if i else 'blastem', 'output:'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
58 for diff in differences:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
59 print diff[i]
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
60 print '-----------------------------'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
61
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
62
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
63
224
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
64 for path in glob('generated_tests/*/*.bin'):
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
65 if path in skip:
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
66 continue
220
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
67 if prefixes:
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
68 good = False
224
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
69 fname = path.split('/')[-1]
220
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
70 for prefix in prefixes:
224
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
71 if fname.startswith(prefix):
220
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
72 good = True
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
73 break
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
74 if not good:
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
75 continue
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
76 try:
440
306986209cba Fix 68K test harness
Mike Pavone <pavone@retrodev.com>
parents: 224
diff changeset
77 b = subprocess.check_output(['./trans', path])
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
78 try:
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
79 m = subprocess.check_output(['musashi/mustrans', path])
217
acd29e2664c6 Added testcases file. Some fixes to test generator for dealing with indexed mode with base and index reg the same. Added support for blastem headless mode in test runner.
Mike Pavone <pavone@retrodev.com>
parents: 214
diff changeset
80 #_,_,b = b.partition('\n')
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
81 if b != m:
572
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
82 print_mismatch(path, b, m)
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
83
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
84 else:
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
85 print path, 'passed'
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
86 except subprocess.CalledProcessError as e:
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
87 print '-----------------------------'
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
88 print 'musashi exited with code', e.returncode, 'for test', path
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
89 print 'blastem output:'
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
90 print b
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
91 print '-----------------------------'
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
92 except subprocess.CalledProcessError as e:
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
93 print '-----------------------------'
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
94 print 'blastem exited with code', e.returncode, 'for test', path
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
95 print '-----------------------------'
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
96