Skip to content

Run oeqa Tests in QEMU

This guide covers running the ToloMEO runtime test suite (tlm_systemd, tlm_platform, tlm_cloud) and the on-target ptests against a QEMU image locally, under TCG (no /dev/kvm required).

Prerequisites

  • The devcontainer (provides kas and the disk tools sfdisk, mkfs.ext4, mksquashfs).
  • Device credentials at certs/agent.env, consumed by the tlm_cloud checks.
  • Network access from the container, needed only by the tlm_cloud reachability probe.

Build the Test Image

Build image-devel. The image itself carries ptest-runner and tolomeo-qemu-netcfg; the kas-ci/testimage.yml fragment is needed only when running the tests, where it enables the testimage task and sets TEST_SUITES:

kas build kas/tolomeo-qemux86-64_tolomeo-devel_image-devel.yml

The boot artifacts land in build/tmp-glibc/deploy/images/tolomeo-qemux86-64/. For a from-scratch build (empty local sstate, remote mirror only), run make clean-sstate first.

The checkout also applies the kas/patches/poky/ patches, which backport resulttool junit from oe-core to scarthgap.

Provision the Second QEMU Disk

The image boots with a second virtio disk (GPT A/B rootfs slots, data, and a certs partition) plus a shared folder. Provision them at the build TMPDIR, flashing certs/agent.env onto the certs partition:

meta-tolomeo-qemu/scripts/provision-qemu-disk.sh build/tmp-glibc certs/agent.env

This creates build/tmp-glibc/qemu-disk.img and build/tmp-glibc/shared/.

Run the Tests

Run the tests with either of two equivalent approaches.

Option A — In-Build Task

Run the testimage task; it boots QEMU and runs the fragment's TEST_SUITES:

kas shell kas/tolomeo-qemux86-64_tolomeo-devel_image-devel.yml:kas-ci/testimage.yml \
  -c 'bitbake image-devel -c testimage'

Option B — Standalone Runner

Boot the already-built image with runqemu and run oe-test against it — no BitBake. Pass the deploy directory; the suite list is optional and defaults to the full set:

meta-tolomeo-qemu/scripts/run-oe-test.sh \
  build/tmp-glibc/deploy/images/tolomeo-qemux86-64 \
  ping ssh ptest tlm_systemd tlm_platform tlm_cloud

A successful run ends with:

runtime - OK - All required tests passed (successes=..., skipped=0, failures=0, errors=0)
[run-oe-test] oe-test exit: 0

Know What the Suites Cover

Suite Checks
ping, ssh Reachability and SSH into the guest
ptest On-target shell ptests (ota-monitor, system-update-postboot, swupdate, hwrevision)
tlm_systemd Per-unit enable-state and ordering (including start after connhex-edge-agent); nothing failed
tlm_platform Platform mounts and overlays (/app, /data, /certs, /shared, overlayfs-etc)
tlm_cloud network-online.target reached, connhex-edge-agent active, /certs/agent.env present, egress

Read the Reports

Both approaches write a machine-readable testresults.json (same oeqa schema) plus detailed logs — no extra flags needed. Archive the testresults.json, and the log directory for detail.

Option A — In-Build Task

Artifact Path
JSON report build/tmp-glibc/log/oeqa/testresults.json (per-image copy under image-devel/)
Detailed logs (SSH command log, QEMU boot log) build/tmp-glibc/work/tolomeo_qemux86_64-tolomeo-linux/image-devel/*/testimage/

Option B — Standalone Runner

Artifact Path
JSON report runtime-results/testresults.json, relative to the directory you ran the script from
Console / detail log runtime-results.log and timestamped runtime-results-<YYYYMMDDhhmmss>.log, same place

Option B accumulates runs

runtime-results/testresults.json keeps every run — each adds a top-level key runtime_image-devel_tolomeo-qemux86-64_<timestamp>. Delete runtime-results/ first for a single-run report.

Both files share the oeqa schema, so resulttool reads either — for a summary, or to diff two runs:

sources/poky/scripts/resulttool report     build/tmp-glibc/log/oeqa/   # Option A
sources/poky/scripts/resulttool report     runtime-results/            # Option B
sources/poky/scripts/resulttool regression <baseline-dir> <new-dir>    # diff two runs

JUnit Report

resulttool junit converts a testresults.json to JUnit XML, the format CI publishes as the test report. It writes junit.xml next to the JSON unless -j names another path; -a also writes the ptest logs to a test-logs/ subfolder and references them from the XML:

sources/poky/scripts/resulttool junit build/tmp-glibc/log/oeqa/testresults.json   # Option A
sources/poky/scripts/resulttool junit runtime-results/testresults.json            # Option B
sources/poky/scripts/resulttool junit build/tmp-glibc/log/oeqa/testresults.json -j junit.xml -a

The command prints a summary line of the form total -> tests: …, success: …, error: …, failures: …, skipped: ….

Troubleshooting

The boot times out before the login prompt

TCG boots are slow. Raise BOOT_TIMEOUT for run-oe-test.sh, or TEST_QEMUBOOT_TIMEOUT for the in-build task.

The cloud checks skip or the credential check fails

certs/agent.env is missing or empty. The reachability probe is opt-in and non-fatal (it skips without egress), but tlm_cloud still checks that /certs/agent.env is present — re-run the provisioning step with a valid file.

resulttool has no junit subcommand

The sources/poky checkout predates the kas/patches/poky/ backport, or kas skipped the patches because the checkout has local changes. Stash or discard the changes in sources/poky, then check out again:

kas checkout kas/tolomeo-qemux86-64_tolomeo-devel_image-devel.yml
test -f sources/poky/scripts/lib/resulttool/junit.py && echo "resulttool junit available"

runqemu cannot find a QEMU binary

The build tree is incomplete, so the native QEMU under recipe-sysroot-native is absent. Rebuild the image (first step) so the native tools exist.

See also