| Stage | What | Command | Who |
|---|---|---|---|
| 1. MPF only | game logic (MPF's test harness), the real mpf booting clean, every slide/widget/sound/show/switch a config names exists, random switch storms (fuzz), the registry |
.venv/Scripts/python tools/test/stage1.py (~15 s; CI runs the same) |
automatic, every change |
| 2. Playtest, no hardware | the real MPF + GMC (Godot window, screens, sound) playing scripted scenarios | .venv/Scripts/python tools/test/playtest.py --scenario <attract\|period_1\|period_2\|period_3\|period_4\|period_5\|small_ending\|full> --speed <fast\|watch\|step\|N> |
automatic; watch/step to follow along |
| 3. Hardware check | every wired switch, coil and LED on the real boards, guided on the machine's display | .venv/Scripts/python tools/test/playtest.py --scenario hardware_check --hardware |
the owner at the cabinet (flipper buttons answer) |
| 4. Play by hand | Godot + MPF + MPF Monitor, keys for every shot, notes with screenshots | .venv/Scripts/python tools/test/session.py [--hardware] |
the owner |
Stage 1 - tests/ (the unit tests, plus test_boot.py, test_config_refs.py, test_fuzz.py)
and check_registry.py, run by tools/test/stage1.py. MPF's own advice is tests that play the
game plus fuzz tests that "hit all the switches randomly" (Testing your game,
archived mpf-docs-testing); MPF 0.80 ships no fuzz tool, so test_fuzz.py is ours (fixed
seeds; checked that the storms really score and that an injected exception fails the test).
Stage 2 - built on MPF's Platform Integration Test runner, mpf -pit <test.py> ("Run a
platform integration test", mpf game,
archived mpf-docs-game-command): MPF loads the scenario class from machinefolder/playtests/
and runs it inside the real, running MPF, with GMC connected. A scenario presses switches, waits
for the events that should follow, checks modes and player variables, confirms each screen by
GMC's own slide_<name>_active event, asks for screenshots and lists the sounds GMC must play
(playtests/base.py). The runner (tools/test/playtest.py) starts Godot and MPF, adds a
temporary capture autoload (never committed), restores MPF's runtime data afterwards and writes a
run folder (playtest-runs/, not committed) with report.html, a contact sheet and the logs. It
fails on a failed check, a missing sound, a Godot SCRIPT ERROR or an MPF ERROR.
--speed sets the pacing: fast (~85 s for full), watchstep (Enter before each action, showing what's next), or seconds--setting NAME=VALUE (repeatable) sets an operator setting before the scenario,--setting sign_style=2 for the signs' icon-only look; the run folder gets-sign_style2 added, and the scenario checks that every sign shown is that look (CHANGES #93).name (BCP sends the eventname=); MPF doesn't send an event that came in over BCP back to BCP clients (soattract_cycle_hold) while a scenarioStage 3 - the hardware_check scenario on the real boards (--hardware asks for a typed
yes). Each step is a slide on the machine's own display (modes/hwcheck); the owner answers
with the flipper buttons (right = yes, left = no / skip). Order: the two answer buttons, every
wired switch area by area with the coil power OFF (the plunger-lane switch and the launch
button make MPF eject a ball, so the walk only starts once the owner confirms the power is off),
the coils one at a time after the owner confirms the power is back on (the trough eject last:
it puts a ball in the shooter lane), every LED lit white, then MPF's audit of switches never hit.
Throughout, every ball device's auto_fire_on_unexpected_ball is switched off, so nothing
launches by itself. A hardware run keeps MPF's audit data (a virtual run's is restored). --auto-answer dry-runs the
whole flow on the virtual platform. Only ever started with the owner at the cabinet.
Stage 4 - tools/test/session.py starts Godot, MPF (virtual, or --hardware) and MPF
Monitor: the playfield picture with the switches and LEDs on it - "Left-click on a switch to
'tap' it ... Right-click ... to 'toggle' it", LEDs live (MPF Monitor,
archived mpf-docs-monitor). Install it once with tools/test/setup_monitor.py (its 1.0.0
package leaves out its .ui files; the script restores them). tools/test/monitor_layout.py
places the devices by registry area on a drawn picture until a photo of the playfield replaces
machinefolder/monitor/playfield.jpg (the tool never redraws over a photo - it checks the drawn
picture's fingerprint); positions dragged in MPF Monitor are kept. --hardware asks for a typed
yes: on the real boards, key 1/3 and clicking a trough or plunger-lane switch in MPF Monitor
act like a real ball, so MPF may pulse the trough eject or the autolauncher. Keys in the
Godot window: machinefolder/gmc.cfg (q-p the major shots, F5/F6 periods 2/3, F7/F8 hold/resume
the attract pages, F12 a screenshot). Notes typed in the session terminal are saved with the time
and a screenshot in the session's notes.md.
The sections below are the original investigation (2026-08-27 onward) and the details behind
the stages.
The resumption roadmap (CHANGES.md #3, now finished) called for hardware checkpoints on every
phase, but that shouldn't be the only way to validate config/rules changes — especially once
Phase 2 (rules architecture) and Phase 3 (feature modes) start landing config-heavy changes that
are cheap to get subtly wrong. This is a concrete, verified design for testing this MPF machine
without needing the physical cabinet connected. It superseded the one-line "Unit tests" bullet in
the roadmap's Verification section.
The local MPF install was broken when this investigation started:
C:\Users\dorus\.local\bin\mpf.exe (on PATH) was a 0-byte stub file — nothing worked, virtual or
real hardware. Fixed as Tier 0 below. Every tier past Tier 0 has been run for real against this
repo's actual machinefolder/ config (not just read about) — two claims from initial doc research
turned out to be wrong for the installed MPF 0.80.0 and are corrected inline below.
.venv/ at the repo root (gitignored), MPF 0.80.0 installed via pip install mpf==0.80.0config_version: 6 and the "MPF 0.80" comment in config.yaml, and supports./.venv/Scripts/mpf --version → Mission Pinball Framework v0.80.0.machinefolder/addons/mpf-gmc, not installed via pip. (Kivy does still matter forVerified command, run from machinefolder/:
../.venv/Scripts/mpf -X -t -b
-X forces the smart_virtual platform regardless of the committedhardware: platform: opp in config.yaml — nothing to revert, no risk of a "test mode" config-x gives plain virtual; -X is the smarter one that-t disables MPF's ASCII text UI (an asciimatics-based curses-style console). This is-b skips the BCP connection attempt, for a pure backend run with no Godot needed. Drop-b to also run the Godot mpf-gmc app alongside for a full display+audio playtest loop — itinit_done, machine_reset_phase_1-3, and intoattract mode run loop, with zero ERROR/WARNING lines in the run or in the generatedmachinefolder/logs/*.log file.gmc.cfg's [keyboard] block covers manual switch presses for anything smart_virtuals-trough1, s-start, s-plunger-lane, s-toplane1). Extend thisThis tier is the fast-iteration replacement for "play it on the machine" during active
config/rules work — not a replacement for the real hardware checkpoint, but the thing to reach for
constantly in between.
Correction from initial doc research: the mpf test CLI command unconditionally imports
kivy (mpf/commands/test.py) — it's built for MPF's own internal doc-test suite tied to the
legacy Kivy-based mpf-mc, and fails with ModuleNotFoundError: No module named 'kivy' in this
project (which correctly has no mpf-mc/kivy dependency). Don't install kivy just to unblock
this — MpfTestCase/MpfGameTestCase themselves have no kivy dependency at all; just run tests
the plain Python way instead:
./.venv/Scripts/python -m unittest discover tests
tests/test_bringup.py — starts a real game against the actualmachinefolder/config/config.yaml, fills the trough, confirms ball 1 is in play, hits s-toplane1,base.yaml's variable_player config). VerifiedRan 1 test in 0.271s / OK.mpf.tests.MpfTestCase/MpfGameTestCase sourceget_machine_path() / get_config_file() — override per test class. get_machine_path()tests/test_bringup.py computes it fromos.path.dirname(__file__)) — a relative path gets resolved against the installed mpfget_platform() — override to return 'smart_virtual' (default is plain 'virtual').force_platform from this method regardless ofconfig.yaml's committed platform: opp — tests can never accidentally touch real hardware.hit_switch_and_run(name, delta) / release_switch_and_run(name, delta) — simulate switchmock_event(name) / assertEventCalled(name) / assertEventNotCalled(name) /post_event(name, run_time, **kwargs) — event-based assertions.MpfGameTestCase adds fill_troughs(), start_game(), add_player(),assertBallNumber(n), drain_one_ball()/drain_all_balls(), assertGameIsRunning(), etc.MpfFakeGameTestCase exists for teststests/test_ball_save.py,tests/test_lanes.py, tests/test_flippers.py, tests/test_portal.py), each exercising thetests/ has a filetest_story_p1.py, test_pa_callouts.py, test_full_game.py and more.test_ball_save.py; there's no test_flippers.py: the flippers are plain MPF.github/workflows/ci.yml) runs it on every push and pull request.Correction from initial doc research: there is no mpf format/lint command in the installed
MPF 0.80.0 — the actual command set is both, build, core, create_config, diagnosis,
game, hardware, init, service, test, wire (verified by listing
mpf/commands/*.py directly). wire is FAST-platform-specific wiring-diagram tooling (not
relevant, this project uses OPP), and none of the others do YAML/schema linting.
In practice this doesn't leave a gap: MPF's config loader validates schema/typos as part of a
normal boot, so Tier 1's clean-boot check already is the fast config-validation step — a
bad config value fails immediately on mpf -X -t -b, before a game can even start. No separate
tool needed.
Real findings from writing ~30 additional tests across the feature/bonus/multiball/wizard-gate
work, kept here so they're not rediscovered the hard way again:
MpfGameTestCase.fill_troughs() overfills relative to balls_installed. It activatesball_switch configured on a trough device — for this project's bd-trough that's 7s-trough1-6 + s-trough-jam, a real trough capacity larger than the 4 balls actuallyhardware-basic.yaml's balls_installed: 4 and confuses thevirtual_platform_start_active_switches:for name in ("s-trough1", "s-trough2", "s-trough3", "s-trough4"):
self.hit_switch_and_run(name, 0)
self.advance_time_and_run()
bd-plunger uses mechanical_eject, so a ball sitss-launch is hit. Calling drain_all_balls()playfield.available_balls, and every later ball_starting hangs foreverBallController.wait_until_playfields_are_empty() (a silent hang, not an exception — watchbd-plunger's eject onto the (switchless) playfield has no confirming switch, soconfirm_eject_type: target falls back to its ~10s default eject timeout beforeplayfield.balls actually updates. A test that checks playfield ball count right after aadvance_time_and_run(10+) first, not just a couple seconds.bonus mode adds ~6s between a ball draining and the next one starting once enableddisplay_delay_ms × 3 steps at MPF's 2000ms default — bonus_start → entry → total → end).bonus was enabled needs its post-drainadvance_time_and_run() budget bumped accordingly.disable_on_complete defaults to true for every logic_block (counters:/accruals:/sequences: — mpf/config_spec.yaml's shared logic_blocks_common). A logic block that needsdisable_on_complete: false explicitly, or it silently caps at one completion, ever, per modepersist_state defaults to false for every logic_block. Progress that needs to survivepersist_state: true explicitly — the default resets every ball like a normal per-ballachievements: DOES default a clean per-stage completion eventevents_when_completed → achievement_<name>_state_completed, filled in byvalidate_and_parse_config whenever not set explicitly) — don't assume only the genericachievement_<name>_changed_state event exists without checking the installed source first.achievement_group is for player-selectable "pick one" mechanics (rotation/random-select),achievements: by their own default completionachievement_group.MpfTestCase.assertLightOn/assertLightOff are broken for multi-channel subtype: ledlight.hw_driver (singular), which doesn't exist onAttributeError: 'Light' object has no attribute 'hw_driver' — it's hw_drivers,assertLightColor(name, "off") / assertLightColor(name, "on") instead for anyGMCMedia (mpf-gmc's sound/slide registry) methods directly outside its normalsound_player: filenames resolve correctly in Godot (the audio equivalent of the slideMPF's own boot check (Tier 1) validates slide_player:/show_player: config syntax but never
actually renders a .tscn slide — that's mpf-gmc/Godot's job. Godot 4.3 is installed on this
machine (C:\Program Files (x86)\Godot\Godot_v4.3-stable_win64_console.exe), so a slide can be
rendered for real and reviewed as a screenshot:
Godot_v4.3-stable_win64_console.exe --path machinefolder --script capture.gd
where capture.gd (a SceneTree-extending script, not committed to the repo — write it to the
scratchpad per-use) loads the .tscn, instantiates it, waits a couple of process_frames, and
saves get_root().get_texture().get_image() as a PNG. Do not pass --headless — that forces
a null/dummy renderer and produces blank textures; the plain (non-headless) console binary still
runs to completion non-interactively and renders for real via Vulkan. This caught one genuine
rendering bug during the original 10-slide art pass (Line2D closed-loop rectangles
triangulating incorrectly — switched to ColorRect) and was used again to verify the multiball
slide and iterate on an icon's scale/position (2026-08-27), and again on period 1's chamber sign
(tools/display/story_p1_slides.py, CHANGES.md #63), where it showed that mpf-gmc's
MPFConditional hides itself whenever its variable is 0 or unset.
This only verifies slides (visual). The equivalent check for sounds does not currently exist
— see the GMCMedia finding in the Tier 2 addendum above.
mpf run (2026-10-08)The render above shows one scene in one state. To see screens change during play - widgets
staying up across slides, conditionals switching as player variables change - run the real
display against a virtual mpf and drive it. Done by hand on 2026-10-08 (CHANGES #72); now
automated as test stage 2 (tools/test/playtest.py, above) - the steps below are what it does.
Godot_v4.3-stable_win64_console.exe --headless --path machinefolder --import. (--editor --quit aborts its file scan before importing anything.).import files it creates next to the assets.mpf, without --headless: GMC listens on port 5050 and mpfmpftools/mpf-session.ps1 -Action Start -Virtual (no -NoBcp).mpf also listens on port 5051. A small socket client answershello/reset and sends switch?name=s-start&state=int:1 (then 0); trigger?name=<event>story_period_2 to skip ahead. monitor_start?category=player_varsmachinefolder/override.cfg[autoload] + ZzCapture="*res://zz_capture_tmp.gd") adds a node that, when a requestget_viewport().get_texture().get_image() afterRenderingServer.frame_post_draw, and sets get_window().size to 1900x1000. Screen grabsdata/audits.yaml afterwards (the run's audits are noise).Every slide at once, without mpf: for a generated set like the 50 attract pages, a
SceneTree script (--script, not --headless) can set sample values in
MPF.game.machine_vars, instantiate each .tscn in turn, call its action_update({}, {}) (what
MPF's slide player does - without it no conditional is evaluated, docs/gmc-bugs.md #6), and save
the viewport after frame_post_draw - one PNG per slide, then a contact sheet to look at them together. Done
2026-10-08 (design/research/live-screen-test/attract-pages-2026-10-08/).
It found two bugs the unit tests and the single-scene renders couldn't: the chamber sign vanished
with the first chapter card, and its conditionals never updated (design/SCREENS.md, "Seen
live"). GMC's log (stdout) also lists every sound it plays (Bus<voice> : Playing sound ...), a
first check that sound names resolve - the gap noted in the Tier 2 addendum - though not that
they sound right.
MPF ships a fuzz-testing mode (mpf/tests includes fuzz-test infrastructure) that randomly hits
switches to shake out crashes/hangs that deliberate Tier 2 test cases wouldn't think to try. Worth
adding once the Tier 2 suite has enough real coverage to be worth stress-testing — not a day-one
item, not yet independently verified.
Confirmed real: mpf game --help shows a --vpx flag (force_platform: virtual_pinball),
so bridging to a Visual Pinball X table for actual ball-physics simulation is a real, one-flag-away
MPF capability — the most realistic non-cabinet test possible. It still requires building and
maintaining a VPX table file matching this machine's actual layout, which is a real design/asset
task in its own right. Flagged as a future option, intentionally not built as part of this design.
.venv/ created and MPF 0.80.0 installed (gitignored via the repo-root .gitignore addedvenv module also drops its own internal .gitignore inside.venv/, so it was already excluded either way).tests/test_bringup.py — first passing example test (Tier 2).README.md — "Running it" section corrected to match what was actually verified here (the venvmpf -X -t -b, python -m unittest discover tests), replacing the previous text whichmpf install that didn't actually exist.TODO.md — broken mpf.exe stub checked off as fixed; the mpf test/kivy and mpf format-doesn't-exist corrections added so they're not rediscovered the hard way again../.venv/Scripts/mpf --version → Mission Pinball Framework v0.80.0. Done.mpf -X -t -b from machinefolder/ boots to the attract mode run loop with nopython -m unittest discover tests from the repo root passestests/test_bringup.py::test_game_starts_and_scores in well under a second. Done — extend