Debugging

This page has two halves. Tools is the standing kit — how to get logs, run the simulator, read system health, and step with GDB. Playbooks is symptom-driven: start there when something is already broken.

Tools

RTT Logging

flash.sh flashes the firmware and streams RTT log output via defmt and probe-rs. Log levels are controlled by DEFMT_LOG (set to debug by default in .cargo/config.toml):

[env]
DEFMT_LOG = "debug"

Override it per-invocation by exporting DEFMT_LOG before flash.sh (e.g. DEFMT_LOG=trace). On hardware, Log.i(TAG, msg) arrives over RTT as TAG: msg; in the simulator the same call prints [TAG] msg.

Stack traces and shrunk logs

An uncaught exception prints a Java-style trace — the exception class and message, then one frame per call in Android’s StackTraceElement spelling:

Exception in thread "main" java.lang.RuntimeException: kaboom
    at tracedemo.TraceDemo.deepest(TraceDemo.java:41)
    at tracedemo.TraceDemo.middle(TraceDemo.java:37)

That is what the simulator prints: it is built with the line-numbers cargo feature, which keeps LineNumberTable and SourceFile in the class files. Every device firmware, debug or release, leaves the tables out of flash and prints the bytecode offset instead, at tracedemo.TraceDemo.deepest(pc=9): the tables and the JVM’s line-number paths cost a W board’s debug image about 70 KB of its program region (the RP2040 dropped them first, on 2026-09-26; every board followed on 2026-10-14). PICODROID_LINE_NUMBERS=1 builds them in on any board with the room, and flash.sh prints a one-line reminder whenever it flashes a firmware without them. Resolve (pc=N) frames on the host from the class trees this checkout compiled:

./scripts/retrace.sh --app tracedemo < device.log      # (pc=N) -> (TraceDemo.java:41)

A class without a SourceFile prints (Unknown Source:41); overloads whose lines disagree list every candidate (TraceDemo.java:12|40). The trees must come from the same build the device runs, or a line can be off.

The same script un-shrinks a --shrink log. A shrunk firmware prints the mapped names everywhere — Class.getName(), exception banners, frames (a.DK.uQ where a no-shrink build says picodroid.view.View.setText), pdb output — and retrace.sh substitutes the originals back using the active release map. A PAPK built with --shrink-app also renames the app’s own classes (c.A.qZ); retrace it with the map build-apk.sh wrote next to the PAPK:

./scripts/sim.sh --app foo --shrink 2>&1 | ./scripts/retrace.sh
./scripts/retrace.sh build/apks/foo.shrink-map.toml --app foo < device.log

examples/tracedemo throws a three-deep uncaught exception so you can see each form. What is renamed, the map format and the compatibility rules are on the Shrinker page.

Host Simulator

The host simulator lets you run apps on your development machine without hardware. Hardware calls are stubbed with logged output, making it useful for testing app logic and debugging JVM behaviour.

./scripts/sim.sh --app helloworld
./scripts/sim.sh --app blinky          # loops forever — Ctrl-C to stop
./scripts/sim.sh --app benchmark       # JVM performance benchmark
./scripts/sim.sh --app gcstress        # GC stress test
./scripts/sim.sh --app displaydemo     # opens a 320x240 graphical window
./scripts/sim.sh --board pico_enviro_mon --app picoenvmon   # a four-button board

For display apps, the simulator opens a graphical window (via minifb) that renders the LVGL widget tree with mouse-as-touch input. On a board with buttons the host keyboard presses them: the Up and Down arrows are PREV and NEXT, Enter is ENTER, Backspace is BACK, and 1–4 are the first to fourth declared button. Close the window or press Escape to exit.

The simulator models the board’s memory, not the host’s: the heap is capped at the board’s FreeRTOS arena and the LVGL pool is sized from its lv_mem_kb, so an app that runs out on the device runs out here. Two checks are on by default — the handle sanitizer, which stops the run with a backtrace when a freed widget is used (--no-sanitize-handles turns it off), and the UI build budget, which prints screen build: … took N ms when a screen build holds the UI tick for 250 ms or more, and jank: … when steady-state handlers of 50 ms or more cost fifteen frames in a second (on a device too).

Three environment switches change what the simulator boots into, without a rebuild:

Variable	Effect
`PICODROID_DONT_KEEP_ACTIVITIES=1`	Destroy every Activity the moment it is covered, as Android’s developer option does — finds state kept in fields instead of the `Bundle`
`PICODROID_BOOT=launcher` (with `--system-apps`)	Boot the launcher, with the app under test installed beside it; see Launcher
`PICODROID_SIM_NET=down`	Boot a network board with its link down; `net up` on the control channel raises it

To drive the UI without a local display (over SSH, in CI, or for scripted QA), see Driving the simulator headlessly below.

System Monitor (pdb sysmon)

The pdb sysmon command queries runtime system health over the device’s USB CDC port without reflashing or adding debug prints:

pdb -s /dev/cu.usbmodem102 sysmon

This reports:

Heap: free bytes, minimum-ever free bytes (high-water mark)
Uptime: tick count and wall-clock seconds
Task table: every FreeRTOS task with name, state, priority, stack high-water mark, and CPU %
JVM block (a mem-diag firmware only): live bytes, the post-GC floor, total allocations and the largest free block

CPU % is computed from the delta between consecutive queries — run it twice with a few seconds in between. The first query shows CPU % as N/A, and so does the simulator (pdb -s sim sysmon), whose kernel keeps no run-time statistics.

Under the hood this uses xPortGetFreeHeapSize(), xPortGetMinimumEverFreeHeapSize(), and uxTaskGetSystemState() from FreeRTOS, with run-time stats driven by the hardware microsecond timer (TIMERAWL register). There is no background sampling task — stats are collected on-demand when the host sends the query, so there is zero impact on power consumption or scheduling.

The min-ever-free figure is your heap high-water mark across the whole run; it is the single most useful number for chasing memory pressure (see Out of memory / heap exhaustion).

Injecting input (pdb input)

pdb input drives a real device from the host — the picodroid analog of adb shell input tap|swipe|keyevent. It is the hardware counterpart of the sim control channel: the same Android verbs, sent over the device’s USB CDC port, so a script or an AI agent can exercise an app end-to-end without touching the board.

pdb input keyevent KEYCODE_DPAD_DOWN   # press+release a key (name or number, e.g. 20)
pdb input keyevent --longpress 23      # hold past the long-press timeout (Android's flag)
pdb input keyevent --down 20           # press only; `--up 20` releases (hold a key across commands)
pdb input dpad up                      # up|down|left|right|center — D-pad shorthand
pdb input back                         # KEYCODE_BACK shorthand
pdb input tap 120 80                   # touch tap at (x, y)
pdb input swipe 40 60 200 60 300       # swipe (x1 y1)->(x2 y2) over [ms] (default 300)

The injection happens at the HAL layer on the device (a GPIO edge for keys, the touch sampler for tap/swipe), so the full on-device pipeline runs unchanged — the EditMode filter, LVGL focus navigation, KeyEvent/MotionEvent dispatch, and BACK routing all behave exactly as they do for a physical press. This mirrors Android’s InputManager.injectInputEvent, where the on-device program builds the event and injection is privileged (here, reachable only over PDB).

Keycode→pin resolution happens on the device against the board’s button table, so the host stays board-agnostic — a keycode with no matching button returns ERR (no such key), and tap/swipe on a board with no touchscreen returns ERR (no touch panel). On success the command prints nothing and exits 0.

Diagnostics builds (mem-diag, sched-diag)

Two opt-in cargo features compile in monitors that report once a second; with the feature off, none of their code is in the binary.

Feature	Watches	Simulator	Firmware
`mem-diag`	Heap growth, churn, fragmentation; offensive checks for corruption	`./scripts/sim.sh --app <app> --mem-diag`	`PICODROID_EXTRA_FEATURES=mem-diag ./scripts/flash.sh -b <board> -a <app>`
`sched-diag`	A real-time task holding its core (`HOG`), a starved task (`STARVE`), sleep-polling (`POLL`), a delay that burns cycles (`BUSYDELAY`), a long `spin_until!` (`SPIN`)	`./scripts/sim.sh --app <app> --sched-diag`	`PICODROID_EXTRA_FEATURES=sched-diag ./scripts/flash.sh -b <board> -a <app>`

A diagnostics image says so at boot (memdiag: ACTIVE, scheddiag: ACTIVE). The memory monitor prints one [memmon] line per window (memmon: over RTT): floor is the post-GC live floor, the leak signal; lv=used/total is the LVGL pool; and LEAK? is the growth sentinel tripping. In a simulator built with --mem-diag, ./scripts/sim-ctrl.sh memstats prints a snapshot on demand and heapcensus the live set by class. The scheduling monitor prints one [schedmon] line per window and a line per finding.

On the RP2040 both are manual opt-ins: each adds flash the program region may not have room for. Output formats and knobs are in docs/memory-diagnostics.md and docs/scheduling-diagnostics.md.

HardFault

A firmware fault logs the stacked exception frame over RTT before it halts:

[fault] HardFault pc=0x10012345 lr=0x10011f21 r0=... r1=... r2=... r3=... r12=... xpsr=...

Resolve pc and lr against the ELF that was flashed: arm-none-eabi-addr2line -e target/thumbv6m-none-eabi/debug/picodroid -i 0x10012345. The RP2040’s Cortex-M0+ has no fault status registers, so these two addresses are the whole record. One catch: probe-rs (0.31) catches hard faults by default and halts at the vector before the handler runs, so the line appears only under probe-rs run --no-catch-hardfault. With the catch on, the backtrace probe-rs prints names whatever symbol precedes the HardFault handler in the image — not where the fault happened.

GDB

GDB debugging is a two-terminal workflow. First, start probe-rs as a GDB server (it listens on localhost:1337 by default):

# RP2040
probe-rs gdb --chip RP2040

# RP2350 (RP2350/RP2354 boards all use the RP235x probe-rs target)
probe-rs gdb --chip RP235x

Then, in a second terminal, launch GDB against the ELF and connect to the server:

# RP2040
arm-none-eabi-gdb target/thumbv6m-none-eabi/debug/picodroid \
    -ex "target remote localhost:1337"

# RP2350
arm-none-eabi-gdb target/thumbv8m.main-none-eabihf/debug/picodroid \
    -ex "target remote localhost:1337"

Playbooks

NoSuchMethod / input dies after a while

Symptom. An app works for the first few interactions, then breaks. A freshly-opened Activity renders wrong and the log shows:

[sim] Activity lifecycle error: NoSuchMethod

On hardware the same line arrives over RTT without the [sim] prefix: Activity lifecycle error: NoSuchMethod. It is emitted from the lifecycle dispatcher in crates/picodroid-core/src/lifecycle/mod.rs.

Rule out the two plain causes first. When no native claims a call, the runtime logs why if it knows, just before the error:

[sim] no native picodroid/view/View.postDelayed — no Handler — use ViewPropertyAnimator timers or Executors.mainExecutor()
[sim] picodroid/widget/ViewPager2 is not built into this board's framework (framework_class_excludes in board.toml)

The first is an Android idiom picodroid does not have, with its replacement (runOnUiThread, View.post, View.postDelayed, sendOrderedBroadcast, sendStickyBroadcast, goAsync and abortBroadcast carry one). The second is a class the target board leaves out of its framework; build the app with the board named (./scripts/build-apk.sh --app myapp --board testbench_rp2040) and the API contract check reports it at build time. On hardware both arrive as RTT warnings without the [sim] prefix, and a --shrink build spells the class by its mapped name (retrace.sh above). A miss with neither line, on a call that used to work, is the case below.

What it usually means. The JVM heap is a non-moving mark-sweep collector with slot reuse; class and method tables are append-only and never collected. So NoSuchMethod at runtime almost always means a still-referenced object was swept (a missing GC root), its heap slot was reused by a later allocation, and a subsequent method dispatch hit the wrong class’s vtable.

The canonical case: the Display singleton (returned by Display.getInstance()) was swept, its slot reused by a SensorEvent, and the next Activity.setContentView resolved setContentView on a SensorEvent — which has no such method. The same failure mode applies to any object held only by Rust-side native state (a cached singleton, a queue of pending callbacks) and never by a Java field.

How to recognise it (timeline).

It works as the first navigation but breaks on the 2nd or 3rd screen transition. More interaction trips it sooner.

The breakage lines up exactly with a GC. Correlate the failure with a collection line:

[PicoEnvMon] Live.onCreate                     ← nav #3
[sim] Activity lifecycle error: NoSuchMethod   ← fails here
[sim] JVM ... gc: 1 collections, 2327 freed    ← the GC that broke it

In isolation each screen works fine — the failure is cumulative, not screen-specific.

A view with a listener is not in this class: the listener is a field of the view, and the view lives while Java reaches it through its parent chain or a field, as on Android. A listener-bearing view that nothing reaches is collected, and the simulator reports it (a View with a listener was collected while its widget is still alive) rather than letting a later event dispatch onto a reused slot. See Embedded gotchas.

Prevention. The framework roots the Display singleton, sensors, bound-service refs, queued Runnables and callbacks in flight through its GC walk. In your own app, keep a field to an object whose callback must keep firing when nothing else in Java references it — a SensorEventListener object you register and drop, say.

Diagnosing a new instance. This class of bug is found by runtime tracing, not static reading: trace the receiver class and method at the failure (recv_class=... method=...) and confirm the receiver is the wrong type for the method being called. If so, the object needs a GC root.

The widget side of the same mistake. A Java View that outlives its widget — it was close()d, removed, or belonged to a Fragment view given up in onDestroyView — holds a stale handle. Handles carry the generation of their table slot, so on a device a call through a stale one resolves to nothing and returns, where it once followed a pointer into freed LVGL memory. The simulator stops at that call with a backtrace (the handle sanitizer), so reproduce a “setter that does nothing” there.

Out of memory / heap exhaustion

Heap exhaustion throws a catchable java.lang.OutOfMemoryError, as on Android — but only after a collection has had its chance:

Bytecode allocations (new SomeClass(), new int[], new T[], multi-dimensional) rewind on failure, flag an emergency GC, and re-execute the opcode after collecting. If the GC frees enough, your allocation succeeds and you never see an error; if it frees nothing, the opcode throws OutOfMemoryError.
Allocations inside the runtime’s own builtins — boxing (Integer.valueOf), interning, a growing ArrayList / HashMap / StringBuilder, String.format — retry the same way: the call rewinds, the heap is collected, and the call runs again. The collection is left as it was when the error is thrown.
Allocations inside a framework native (a widget, a file, a peripheral) throw OutOfMemoryError without a retry, since the native may already have acted.

The error object comes from a reserve the heap sets aside while it still can, so even an allocation that leaves no room for the exception itself is catchable. Catching it is a last resort, not a strategy: release what you hold (clear a cache, drop a list) before retrying.

GC runs after an opcode once allocations cross the threshold (GC_ALLOC_THRESHOLD, default 256) or the emergency flag is set. Tuning that threshold and the heap sizes is documented in JVM tunables; the hard ceilings (heap sizes, object/string/array caps) live in Limits. Don’t fight OOM by guessing — read those two pages for the actual knobs.

Introspect from Java. picodroid.os.Runtime exposes the live heap and GC counters so an app can watch its own footprint:

import picodroid.os.Runtime;

long used = Runtime.usedMemory();   // bytes live across object/array/string heaps
long peak = Runtime.peakMemory();   // max usedMemory() since last reset
Runtime.resetPeakMemory();          // snap peak to current used

long gcNs = Runtime.gcTimeNanos();  // cumulative GC time
int  gcN  = Runtime.gcCount();      // collections so far
int  freed = Runtime.gcFreed();     // objects freed
Runtime.resetGcStats();             // zero the gc time/count/freed counters

usedMemory() is approximate (a sum of the live object, array, and string heaps) and updates the peak as a side effect. The full API is on Runtime.java.

Watch the high-water mark. On hardware, pdb sysmon reports the minimum-ever free heap — that is your native-side high-water mark across the whole run. Wrap a suspect screen, sample pdb sysmon before and after, and watch the min-free figure fall.

Stress apps. Three example apps under examples/ exercise the allocator and print scores you can baseline against:

gcstress — object churn, linked chains, circular refs, string churn; each phase prints name: <us> us (gc: <us> us, <n> collections, <freed> freed) and ends with === PASSED ===.
heapstress — heap-fragmentation stress; same per-phase report format, ends === PASSED ===.
perfbench — speed + memory composite; prints SUBSCORE ..., then SCORE <total>, then === PASSED ===.

Run any of them in the simulator with ./scripts/sim.sh --app gcstress (etc.) and compare the totals before and after a change.

A listener never fires

If a registered callback simply never runs, work through these in order.

SensorManager.registerListener returned false. Always check the return value:

boolean ok = sensorManager.registerListener(this, sensor, rate);
if (!ok) {
    // registration was rejected — nothing will be delivered
}

It returns false when the listener or sensor argument is not a valid object, when the sensor’s type can’t be read, or when all 8 registration slots are already in use (the cap is 8 concurrent registrations). Re-registering the same (listener, sensorType) pair returns true and just refreshes the rate. A common trap is getDefaultSensor(type) returning null because the board has no matching [[sensor]] entry — see registerListener returns false / sensor event never fires in Troubleshooting. The sensor API is covered in Sensors.

A lifecycle/service op was refused. Activity and Service transitions go through an internal pending-op queue with a fixed depth per frame ([jvm] pending_op_queue in board.toml, default 8). When that queue is full, the call that would enqueue one more (startActivity, finish, startService, stopService, …) throws IllegalStateException (too many pending Activity/Service transitions in one frame). If a transition is missing, look for that exception in the log; raise the knob or spread the calls over frames. (The separate MainExecutor/BackgroundExecutor queues still drop a Runnable with a queue full, dropped warning — that is a different queue.)

The view was collected. A view with a listener that nothing in Java reaches — never added to a parent, never kept in a field — is collected, and its widget stays on the screen it was created under with nothing to take its events. The simulator prints a View with a listener was collected while its widget is still alive (nativeHandle N) and, under the handle sanitizer, stops with a backtrace of the collection. Add the view to a parent, keep a field, or close() it. See Embedded gotchas.

A swipe listener. A swipe goes to the nearest view with an OnSwipeListener, starting from the view under the finger; a child with a listener of its own keeps the swipe, and a pull-down inside a SwipeRefreshLayout goes to the layout first. Rehearse one with input swipe (below).

Note: the simulator has no GPIO, but its keyboard and control channel put button edges into the same queue the device’s GPIO interrupt fills, so OnKeyListener, the Activity key callbacks, auto-repeat and the BACK chain all run in the simulator. What only a device shows is the electrical side (bounce, the idle-sleep wake press) — use pdb input to drive a board over USB CDC.

pdb install fails

Every refusal prints Refusing to install: <reason> and exits non-zero. The reasons you’ll actually hit:

Shrink-map version mismatch. The PAPK and the running firmware were built with different --shrink settings, so their framework maps don’t line up:

PAPK is incompatible with running firmware.
  PAPK     framework-map-version = ...
  Firmware framework-map-version = ...
  Reason: ...
  Rebuild the PAPK with matching --shrink setting (see reference/shrinker in the docs).

Reason is usually that the two sides disagree about --shrink (one of them reports 0.0.0) or that the PAPK and the firmware were built from different source trees, so their maps differ — version mismatch (asymmetric —shrink, or PAPK newer than firmware). --shrink-app does not enter into it: the per-app map extends the release map without changing framework-map-version, so a --shrink-app PAPK installs on any --shrink firmware of the same release.

The same condition can surface as a firmware-load panic, PAPK framework-map-version incompatible with firmware. Recovery: rebuild the APK and firmware with the same --shrink flag, or rebuild the PAPK without --shrink to match a non-shrunk firmware, or reflash matching firmware. The shrinker is documented in Shrinker; the panic and recovery steps are in PAPK framework-map-version incompatible with firmware.

No room, or no package name (multi-app boards). device rejected install: STATUS_NO_ROOM means the app region has no contiguous run for the package even after compaction, or the directory is full; nothing was erased. Free room with pdb uninstall <package> (pdb list shows what is installed). PAPK has no package-name; a multi-app device cannot place it means the manifest lacks package; the message carries the papk-pack --repack command that adds one.

Other refusals. A malformed package is rejected unconditionally with PAPK file is not a valid PAPK: <error> (this check runs even under --skip-host-check, to prevent bricking). An over-size package fails with error: PAPK is <n> KB but device supports max <n> KB. If you bypass the host check, the device can still reject with device rejected install: STATUS_INCOMPAT — framework-map-version mismatch — and the existing PAPK on flash is left untouched.

Driving the simulator headlessly

scripts/sim-remote.sh runs the simulator under a virtual X server so you can drive the UI over SSH, from a browser, or from a script — no local display required. It spins up Xvfb + x11vnc + noVNC and then runs sim.sh; every flag passes straight through to sim.sh.

./scripts/sim-remote.sh --board pico_enviro_mon --app picoenvmon

It picks a free X display and free VNC/web ports, then prints a banner with a noVNC URL (http://localhost:<port>/vnc.html?...) and the control-FIFO path. You need Xvfb, x11vnc, and websockify installed; xdotool is strongly recommended for keyboard focus.

Control channel (no browser needed). The script creates a FIFO at /tmp/picodroid-sim-remote-<display>-ctrl and exports it as PICODROID_SIM_CTRL_FIFO. Write verbs to it to inject button input:

echo 'tap B' > /tmp/picodroid-sim-remote-<display>-ctrl

The <display> number is chosen dynamically (:99..:150), so rather than look it up, use the scripts/sim-ctrl.sh wrapper — it auto-discovers the running sim’s FIFO and forwards the command:

./scripts/sim-ctrl.sh tap B          # auto-discovers the single running sim
./scripts/sim-ctrl.sh -d 100 tap B   # or target display :100 explicitly

The verb grammar is down|up|press|tap <button>:

down — press (active-low falling edge)
up — release (rising edge)
press / tap — press, wait 40 ms, release (one clean tap)

Button tokens are the silkscreen names A/B/X/Y (1st/2nd/3rd/4th declared button), the semantic names PREV/UP, NEXT/DOWN, ENTER/OK/SELECT, ESC/BACK, or a bare GPIO pin number. An unknown verb prints [sim] control channel: unknown command '<x>'; an unknown button prints [sim] control channel: unknown button '<x>'. At startup the sim prints a ready banner listing the accepted commands. (The control channel is present on has_buttons and has_touch boards.) Without sim-remote.sh the same commands are read from the simulator’s stdin. For the navigation model these buttons drive, see Button navigation.

The control channel also accepts the same Android verbs as hardware pdb input — input keyevent [--longpress|--down|--up] <KEYCODE|n>, input dpad <dir>, input back, input tap <x> <y>, and input swipe <x1> <y1> <x2> <y2> [ms] — plus input text <string>, sim-only, which types into the field the system keyboard is open on (tap the EditText first). Prefer these when you want one vocabulary that works identically in the sim and on a real device: rehearse a sequence headlessly in the sim, then run the exact same verbs via pdb input over USB CDC.

The wire-true path. The simulator is also a pdb device in its own right: ./scripts/pdb.sh -s sim input tap 120 80 (or ping, list, sysmon, install, uninstall) goes over the simulator’s socket through the same bridge task, framing and handlers a board runs, and an install reboots the simulator the way it reboots a device. Use the control FIFO for the sim-only verbs — button names, touch down|move <x> <y> / touch up, apps list|install|uninstall, net up|down (the simulated link, on a network board), and memstats / heapcensus in a --mem-diag build — and pdb -s sim when the point is to exercise the device path — see Driving the simulator with pdb.

Capturing frames. The sim window is named picodroid. Grab it with scrot:

DISPLAY=:<n> scrot --window "$(xdotool search --name picodroid)" out.png

Logs. Infrastructure logs go to /tmp/picodroid-sim-remote-<display>-{xvfb,x11vnc,novnc}.log. App println! and lifecycle output land in the sim-remote log.

Cleanup. The script TERM-then-KILLs its subtree on exit and removes the FIFO and X locks. To kill a stray sim manually, match the exact process name:

pkill -x picodroid

Do not pkill -f sim-remote.sh — that pattern also matches the launching shell and self-kills it.

xdotool pacing gotcha. minifb misses fast clicks at 60 Hz. When scripting mouse input with xdotool, never use click 1 — instead do mousedown / sleep 0.3 / mouseup. The control-FIFO tap verb embeds its own 40 ms press-release gap for the same reason, so a PRESSED frame renders before the release and the two edges land in distinct ticks.