# Audit methodology

This document defines how raqueeb.dev is scored and what the public evidence at `/audit` means. The machine-readable version is `/audit/methodology.json`; the dashboard, the scoring code (`scripts/audit/score.mjs`) and this text must agree, and the unit tests in `scripts/audit/score.test.mjs` pin the arithmetic.

## 1. Two different things: evidence and scores

**Automated checks** produce facts: Lighthouse numbers, axe violation counts, overflow at a given width, a header present or absent, a test suite passing or failing. They run on every push through GitHub Actions and are published as JSON under `/audit/evidence/`.

**Category scores** are judgements against a written rubric. Some categories can only be judged by a person (design, UX, content, branding). Even the measurable ones are scored by rubric, not by pass rate: a crawl with zero axe violations is evidence for the accessibility score, not the score itself, because automated tools find only a fraction of accessibility problems.

The overall score is computed from category scores. It is never computed from test pass rates.

## 2. Categories and weights

| Category | Weight | Primary data sources |
| --- | ---: | --- |
| Design and visual quality | 10% | Screenshots in both themes, token inspection, manual review |
| UX and navigation | 10% | Manual walkthrough, keyboard suite |
| SEO | 10% | Full crawl, structured-data validator, sitemap, robots, live redirects |
| Performance | 10% | Lighthouse medians (lab), page weight; field data when it exists |
| Accessibility | 10% | axe in both themes, keyboard suite in three engines, target-size sweep |
| Mobile responsiveness | 10% | Viewport sweep at 320, 375, 390, 768, 1024, 1440 px in both themes |
| Content and credibility | 10% | Manual verification of every claim against repositories and live sites |
| Branding | 5% | Manual review across pages and social images |
| Technical implementation | 10% | Type check, lint, format, dependency audit, build gates, bundle size |
| Conversion and contact | 5% | Manual review, link check |
| Security | 5% | Live headers, CSP, HTTPS, redirects, security.txt, npm audit, CMS and OAuth test suites |
| Reliability and testing | 5% | Suite inventory and results, CI status, deployment gates |

Weights sum to 100. The rubric for each category (what a 10, 9, 7, 5 and 3 mean) is in `methodology.json` and shown on the dashboard.

## 3. The formula

```
overall = Σ (category score × weight) / 100
```

Scores are on a 0 to 10 scale with one decimal. The precise weighted value is stored; rounding to two decimals is for display only. If any of the twelve categories has no score, the result is "Not enough evidence to score", not a lower number.

Example (assessment of 3 November 2026):

```
0.1×(9.2+9.2+9.5+9.7+9.3+9.6+8.8+9.3) + 0.05×(9.3+9.0+9.4+9.5) = 9.32
```

## 4. Scoring rules

1. Every category score must cite evidence and a rubric level.
2. Lighthouse scores are displayed separately and never replace the quality score.
3. Accessibility violation counts are displayed independently of the accessibility rating.
4. Security checks are static and configuration checks. No penetration test has been performed; the dashboard says so.
5. Missing evidence lowers confidence or yields "not enough evidence"; it is never a pass.
6. Historical scores are never rewritten. A new assessment adds an entry to `history.json`.
7. Every score is reproducible from `methodology.json`, `assessments.json` and the linked evidence files.
8. The formula and the weights are public.
9. The pipeline does not target a number. If the arithmetic gives 8.7, the dashboard shows 8.7.

## 5. Confidence

| Level | Meaning |
| --- | --- |
| High | The assessment was made for the same commit as the latest automated run and every check passed |
| Medium | Same commit with warnings, or an older assessment with all current checks passing |
| Low | Failed, stale or untested checks, or no assessment at all |

A check older than 30 days is shown as **stale** whatever it said. An assessment made for an earlier commit is labelled **historical** and the dashboard shows how many commits behind it is.

## 6. Data sources, tools and environments

| Check | Tool | Where it runs |
| --- | --- | --- |
| Lighthouse | Lighthouse CLI, Chrome | GitHub Actions against the freshly built site (lab); owner runs against the live site with `scripts/qa/lh-batch.mjs` |
| Crawl (SEO, axe, overflow, console, links) | Playwright + axe-core | GitHub Actions and locally, against `astro preview` |
| Structured data | `scripts/qa/jsonld.mjs` | Build output |
| Viewport sweep | Playwright | GitHub Actions and locally |
| Keyboard and cross-browser | Playwright, Chromium, Firefox, WebKit | GitHub Actions and locally |
| Security headers and redirects | HTTPS requests to the live site | GitHub Actions (production as it is at run time) and locally |
| Type check, lint, format, unit tests | astro check, ESLint, Prettier, node:test | GitHub Actions and locally |
| CMS gate and OAuth Worker suites | node | Locally (they need throwaway credentials); results recorded in assessments |

**Lab versus field.** Lighthouse numbers are lab measurements under simulated throttling on a runner. They are reproducible but they are not what visitors experience. Real-user Core Web Vitals (including INP) come from the Chrome User Experience Report and need enough traffic; until that exists the dashboard says so and TBT is shown only as a lab diagnostic.

**Automated versus manual accessibility.** axe finds roughly a third of accessibility problems. The keyboard suite covers focus order and visibility. A screen-reader session has not been run; the rating says so.

**Automated versus subjective.** SEO checks are mechanical and complete for what they check. Design, UX, content and branding are judgements and are labelled as such with medium confidence unless reviewed independently.

## 7. Frequency and retention

- Automated checks run on every push to `main` and can be started by hand from the Actions tab.
- Each run writes `latest.json` and one evidence file per check, named by date and commit, under `/audit/evidence/`. Evidence files are never modified after they are written.
- `history.json` gains one entry per run and per assessment. Entries are never edited or removed.
- If a run fails, `latest.json` records the failure and the previous successful entries remain in `history.json`.

## 8. Privacy

Every evidence object passes through `scripts/audit/sanitize.mjs` before it is written: local filesystem paths, runner paths, credential-shaped strings, cookies and local hostnames are replaced. Raw logs, traces and screenshots are not published. The published report copies contain no client data beyond what the client sites already show publicly.

## 9. Limitations of this methodology

- Subjective categories are self-assessed; an independent review would raise confidence, not necessarily the score.
- Field performance data does not yet exist.
- Security is assessed by configuration and static checks only.
- Automated runs on a shared CI runner vary by a point or two in Lighthouse; medians and a two-point tolerance for regression detection absorb that.