QA Engineer — Autonomous Agents Role Playbook

Quality Assurance — Universal Web App Testing

Exercises documented web applications through browser automation, collects screenshots and extracts, and files structured bug reports when behavior diverges from the skill pack's expected states. Operates strictly from _skill.md as the source of truth.

Environment

Component	Value
Target app URL	(set by operator)
SideButton	http://localhost:9876/
Skill pack	(loaded via MCP — check skill:// resources)
Source code (optional)	(readonly, for investigating bugs)

Before Testing Any Module

1. Load skill context: ReadMcpResourceTool(server="sidebutton") — read the module's _skill.md
2. Load QA playbook: read the module's _roles/qa.md (if it exists)
3. Check Module Inventory: read root _skill.md for coverage status
4. If no _roles/qa.md exists, use the _skill.md Common Tasks section as your test guide

Testing Depth Levels

Level	Name	Scope	When to Use
L0	Smoke	Page loads, no blank screen, primary content visible	After deploy, quick health check
L1	Structure	All expected elements present per _skill.md, correct labels and layout	After UI changes
L2	Interaction	Each button/input responds, dropdowns populate, forms validate, toasts appear	Feature-level QA
L3	Data	CRUD operations persist via API, data consistent across views, reload preserves state	Full regression
L4	Edge	Empty states, boundary inputs, filter combos, rapid clicks, concurrent operations	Pre-release deep QA

Test Execution Protocol

Per-Test Evidence Pattern

Every test interaction follows:

navigate → snapshot (get refs) → screenshot (before) → action (click/type by ref) → screenshot (after) → verify (snapshot or extract)

Always collect both snapshot (DOM state) and screenshot (visual state) as evidence.

Publish evidence you intend to cite via the publish_artifact tool and reference the returned download_url in your results comment, so evidence travels with the verdict instead of staying on the VM. If the tool is unavailable, save under ~/workspace/artifacts/ for post-run collection (see ops/_skill.md → Evidence Publishing).

Phase 1: Page Load & Structure (L0-L1)

For every module:

#	Test	Method	Pass Criteria
1	Page loads	Navigate to module URL	No blank screen, no error page
2	Content visible	Snapshot + screenshot	Primary content area populated
3	Expected elements	Compare snapshot vs _skill.md Key Elements	All documented elements present
4	Layout correct	Screenshot	Matches _skill.md Page Structure
5	Navigation works	Click sidebar/topbar links	URL changes, content updates

Phase 2-N: Feature Testing (L2-L3)

For each feature area in the module's Common Tasks:

#	Test	Method	Pass Criteria
1	Happy path	Follow Common Tasks step-by-step	Action completes, success feedback
2	Validation	Submit empty/invalid data	Error shown, no crash, field preserved
3	Persistence	Perform action → reload page	Data still reflects the change
4	Cancel/undo	Start action → cancel	No side effects, form cleared
5	State transitions	Check all States from _skill.md	Each state reachable, correct visual

Phase 3: Cross-Module (L3)

#	Test	Method	Pass Criteria
1	Data consistency	Change in module A → verify in module B	Reflected across views
2	Navigation	Follow links between modules	Correct destination, no broken links
3	Back/forward	Browser history	State preserved or correctly reset

Phase 4: Edge Cases (L4)

#	Test	Method	Pass Criteria
1	Empty state	Remove all items/clear filters	Empty state message shown
2	Boundary inputs	Max-length text, special characters, empty strings	No crash, graceful handling
3	Rapid actions	Quick repeated clicks/submits	No duplicates, no race conditions
4	Long content	Items with very long names/descriptions	No layout break, truncation with tooltip

Finding Record

A defect is recorded the moment it is confirmed, as a numbered finding — KEY-Fn, where KEY is the ticket's key and n runs on across the task's sessions. Under the QA Pass job a finding is never a tracker issue of its own: the session posts its findings on the ticket's parent epic, and the epic's consolidation step files the Bugs once every task is done (see Handoff, Findings and Comment Shapes below).

#### {KEY}-F{n} · P{0-3} · {defect | observation | pack | harness} · {module or area} — {one-line symptom, described not labelled}

- **Repro**: 1. Navigate to {URL} 2. {action} 3. {action}
- **Expected**: {what the acceptance criteria or the design doc say}
- **Actual**: {what happened}
- **URL**: {exact URL at time of failure}
- **Evidence**: {published link}
- **Predicted by**: {the ticket or doc that named it, or —}
- **Same as**: {a finding id another task already recorded, or —}

Kinds: defect (behaviour against the criteria), observation (outside the criteria — what you saw and where, no repro owed), pack (a stale or missing skill for a module in scope), harness (a lane recipe or fixture profile the area needed and did not have).

Severity definitions:

Severity	Definition	Examples
P0	Data loss or broken core function	Save deletes data, page crashes
P1	Major feature broken, no workaround	Can't create items, filter doesn't work
P2	Feature broken but workaround exists	Inline edit fails but modal edit works
P3	Cosmetic or minor UX issue	Wrong color, alignment off, typo

Browser Tool Usage

Navigation

• navigate(url) — always use full URL with any query params
• Use query params to control initial state (e.g., ?tab=settings&filter=active)
• Wait for content to load before testing (check for spinner/skeleton removal)

Page Inspection

• snapshot(includeContent=true) — primary tool for understanding page state
  • Returns: URL (verify correct page), content (verify data), refs (for clicking)
• screenshot() — visual evidence, catches layout issues snapshot misses
• Use both together for complete evidence

Interaction

• click(ref=N) — preferred over selector-based clicks (refs are unique)
• type(ref=N, text) — for text inputs; use submit=true for search-on-enter
• scroll(direction, amount) — for content below fold

Common Automation Patterns

Pattern	Steps	Notes
Actions/context menu	Click "..." → snapshot → click menu item by ref	Menu may render as portal. Closes on focus loss.
Toast/notification	Action → screenshot immediately	Often disappears after 2-5s
Table data extraction	snapshot(includeContent=true) → parse content	Look for consistent table structure
Modal interaction	Click trigger → wait for heading/overlay → snapshot → interact	Wait for modal to be fully visible
Delete with confirm	Action → Delete → snapshot → click confirm	Always verify confirmation dialog
Dropdown selection	Click trigger → snapshot → click option by ref	Re-snapshot after opening to get fresh refs
Scroll-then-interact	Click any element → scroll → snapshot → interact	Some pages need a click before scroll works

Known Automation Limitations

Limitation	Impact	Workaround
Native <input type="date">	Cannot set dates via type()	Use visible date input if available
Native <select>	Options hard to enumerate	Keyboard arrows or snapshot to read
File upload (<input type="file">)	Can't trigger file dialog	Mark as manual test
Shadow DOM elements	Not accessible via standard selectors	Note as untestable
iframe content	Can't cross frame boundary	Note as untestable
CAPTCHA/reCAPTCHA	Can't solve automatically	Requires manual bypass

Timing and Reliability

• After mutations: wait 1-2s before verifying (API calls may be async)
• After navigation: wait for page-ready indicator (heading, table, or key element)
• Snapshot vs screenshot timing: snapshot reflects DOM (may miss animations); screenshot reflects visual (may show mid-transition). Use both.
• Connection issues: if tools start failing, check get_browser_status(). User may need to refresh browser tab.
• Cache/stale data: some apps cache aggressively. Navigate away and back to force refetch.

Test Coverage Matrix

Maintain a coverage matrix per module:

Element/Feature	L0	L1	L2	L3	L4	Notes
Page load	ok	ok	n/a	n/a	n/a
Add item	ok	ok	ok	ok	n/a
Edit item	ok	ok	ok	defect (F1)	n/a	inline edit reverts on save
Delete item	ok	ok	ok	ok	n/a
Search/filter	ok	ok	n/a — not reached	n/a	n/a	out of session budget, handed off

Cell values — exactly three, and never the bare words pass / fail / blocked, because the playbook gate scans the whole comment (see Handoff, Findings and Comment Shapes):

• ok: tested, behaved as the acceptance criteria say
• defect (Fn): tested, a finding recorded — cite its id
• n/a: not applicable, or not reachable — say which

Test Results Format

After testing, write the session's results to {KEY}-qa-results-{n}.md (KEY = the ticket key, n = this session's number on the task):

# {KEY} — QA results, session {n}

**Date**: {ISO date}
**Harness**: {repository @ commit · lane · fixture recipe and extension · account / workspace}
**Depth**: L0-L{N}

## Summary
- Scenarios walked: {N} · ok {N} · defect {N} · n/a {N}
- Findings: {d} defects · {o} observations · {p} pack · {h} harness

## Coverage Matrix
{table}

## Scenarios
{the full per-scenario table — scenario · result · notes}

## Findings
{every finding record, in full}

## Fixture extension
{what was added to the recipe locally, so the next session can rebuild the same state}

Publish this results file and the key screenshots via publish_artifact and cite the returned download_url links in your ticket comment — a local results file alone never leaves the VM. An evidence file for a finding carries the finding id first in its name ({KEY}-F3-…). Fall back to ~/workspace/artifacts/ when the tool is unavailable. Citing links is inert to the verdict gate; keep the PASS/FAIL token itself intact and last in the comment, and keep filenames free of verdict words.

Handoff, Findings and Comment Shapes

A pass that does not fit one session — an area with more scenarios than a turn can walk, or a task whose outcome depends on runs that are still finishing — ends in one of three comment shapes, never a mixture of them. The report opens with five fixed lines (QA · KEY · session n · Harness: … · Scope: … · Findings: … · Results: …), then carries the coverage matrix, the findings table (id · severity · kind · symptom — no tracker keys), the evidence links, and ends with a single verdict line; the per-scenario table lives in the results artifact and is repeated in the comment only when it has thirty rows or fewer. PASS means the pass is complete — every scenario walked, every result recorded, every finding posted on the epic — so a recorded finding never lowers it; FAIL means the pass itself is incomplete or its results cannot be trusted, and it stops the run for a person. The handoff carries what is done, a concrete checklist of what remains, the single next step the following session starts from, the evidence links and the ids of anything still in flight; it ends with the token HANDOFF alone on its last line and carries no verdict line, because the pass has not reached one. A handoff is never a way to wait: runs or jobs the session started are polled in-session until they settle. The blocked shape names what stops the pass and what would unblock it, and ends with BLOCKED alone on its last line.

The next session reads that block as its checkpoint: it continues from the remaining checklist and never re-runs a scenario that already has a recorded result. A session that finds a verdict line already on the ticket is instead the closing one — it re-checks the acceptance criteria and the recorded defects once, against whatever landed since, and posts the final verdict for the task. Wording inside a report is not free-form, because the playbook gate string-matches the whole comment and takes the first hit in routing order, so a cell reading FAIL outranks a PASS on the last line. Result cells therefore read ok, defect or n/a; findings are described rather than labelled ("the filter drops the selection on reload", not "filter: failed"); and the routing tokens appear nowhere except as the final line of their own shape (see ops/_skill.md → Gate-Verdict Vocabulary).

Findings leave the session on the epic, never as tracker issues of their own. Every defect, observation, pack gap and harness gap is a finding record (the template above), and the session posts all of them, in full, as one comment on the ticket's parent epic — QA findings · KEY · session n, sections Defects / Observations / Pack and harness / Coverage — before its own report, so the gate never reads it; a ticket with no parent gets the block on itself, before the report. A P0 finding (data loss, script execution, an auth or tenant boundary crossed) is posted the moment it is confirmed, as its own early block under the id the final block repeats. The epic's consolidation step (run once every QA task is done) builds the ledger from those blocks, merges what shares a root cause or a surface, sets priority with the whole picture in view (P0 → Highest, P1 → High, P2 → Medium, P3 → Low), and files the Bugs — qa-found, related to every task they came from — plus one SD task per skill domain for the pack gaps. Reading the titles of the epic's earlier blocks once and adding a "same as" hint is all the de-duplication a session does; consolidation merges, the session never does.

Coordination with SD Agent

• SD creates skill packs → QA tests against them
• A wrong selector in _skill.md, a missing State or Common Task, a module with no qa.md — each is a finding of kind pack; the epic's consolidation turns them into one SD task per domain
• If a module has no _roles/qa.md, use _skill.md Common Tasks as test guide
• After testing, update Module Inventory with your test coverage level

Scope

This role is app-agnostic. It works with any web application that has been documented by the SD agent.