Mercurial > repos > blastem

annotate comparetests.py @ 2488:bfd09d3367ba default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Fix crash when enabling VGM recording while running Pico or Copera software
author Michael Pavone <pavone@retrodev.com>
date Mon, 15 Apr 2024 23:07:18 -0700
parents 18555c44a5e7
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2446
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
1 #!/usr/bin/env python3
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
2 from glob import glob
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
3 import subprocess
220
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
4 from sys import exit,argv
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
5
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
6 prefixes = []
224
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
7 skip = set()
220
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
8 for i in range(1, len(argv)):
224
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
9 if '.' in argv[i]:
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
10 f = open(argv[i])
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
11 for line in f:
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
12 parts = line.split()
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
13 for part in parts:
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
14 if part.endswith('.bin'):
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
15 skip.add(part)
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
16 f.close()
2446
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
17 print('Skipping',len(skip),'entries from previous report.')
224
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
18 else:
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
19 prefixes.append(argv[i])
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
20
572
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
21 def print_mismatch(path, b, m):
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
22 blines = b.split('\n')
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
23 mlines = m.split('\n')
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
24 if len(blines) != len(mlines):
2446
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
25 print('-----------------------------')
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
26 print('Unknown mismatch in', path)
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
27 print('blastem output:')
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
28 print(b)
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
29 print('clean output:')
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
30 print(m)
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
31 print('-----------------------------')
572
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
32 return
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
33 prevline = ''
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
34 differences = []
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
35 flagmismatch = False
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
36 regmismatch = False
2446
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
37 for i in range(0, len(blines)):
572
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
38 if blines[i] != mlines[i]:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
39 if prevline == 'XNZVC':
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
40 differences.append((prevline, prevline))
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
41 flagmismatch = True
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
42 else:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
43 regmismatch = True
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
44 differences.append((blines[i], mlines[i]))
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
45 prevline = blines[i]
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
46 if flagmismatch and regmismatch:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
47 mtype = 'General'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
48 elif flagmismatch:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
49 mtype = 'Flag'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
50 elif regmismatch:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
51 mtype = 'Register'
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
52 else:
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
53 mtype = 'Unknown'
2446
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
54 print('-----------------------------')
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
55 print(mtype, 'mismatch in', path)
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
56 for i in range(0, 2):
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
57 print('clean' if i else 'blastem', 'output:')
572
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
58 for diff in differences:
2446
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
59 print(diff[i])
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
60 print('-----------------------------')
572
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
61
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
62
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
63
224
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
64 for path in glob('generated_tests/*/*.bin'):
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
65 if path in skip:
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
66 continue
220
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
67 if prefixes:
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
68 good = False
224
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
69 fname = path.split('/')[-1]
220
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
70 for prefix in prefixes:
224
f7ff02eeec2f Added testcases for move and roxl/roxr. Made some small improvements to test tools.
Mike Pavone <pavone@retrodev.com>
parents: 220
diff changeset
71 if fname.startswith(prefix):
220
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
72 good = True
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
73 break
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
74 if not good:
cb72780e17b1 Add support for picking random numbers in a larger range in test generator. Add support for running a subset of tests in runner. Added testcases for bit and rotate instructions.
Mike Pavone <pavone@retrodev.com>
parents: 217
diff changeset
75 continue
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
76 try:
2446
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
77 b = subprocess.check_output(['./trans', path]).decode('utf-8')
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
78 try:
2446
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
79 m = subprocess.check_output(['../blastem_clean/trans', path]).decode('utf-8')
217
acd29e2664c6 Added testcases file. Some fixes to test generator for dealing with indexed mode with base and index reg the same. Added support for blastem headless mode in test runner.
Mike Pavone <pavone@retrodev.com>
parents: 214
diff changeset
80 #_,_,b = b.partition('\n')
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
81 if b != m:
572
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
82 print_mismatch(path, b, m)
0f32f52fc98e Make some small changes in trans so that it is more likely to produce the same output as mustrans when given misbehaving programs. Add lea to testcases.txt. Improve the output of comparetest.py so that known issues can easily be separated from new ones.
Michael Pavone <pavone@retrodev.com>
parents: 440
diff changeset
83
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
84 else:
2446
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
85 print(path, 'passed')
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
86 except subprocess.CalledProcessError as e:
2446
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
87 print('-----------------------------')
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
88 print('clean exited with code', e.returncode, 'for test', path)
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
89 print('blastem output:')
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
90 print(b)
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
91 print('-----------------------------')
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
92 except subprocess.CalledProcessError as e:
2446
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
93 print('-----------------------------')
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
94 print('blastem exited with code', e.returncode, 'for test', path)
18555c44a5e7 Update comparetests to python3 and change it to use a "clean" build of a core from blatem for comparison
Michael Pavone <pavone@retrodev.com>
parents: 572
diff changeset
95 print('-----------------------------')
214
9126c33cc33c Add test generator, builder and runner
Mike Pavone <pavone@retrodev.com>
parents:
diff changeset
96