Prepare identity, access, and test data

Create dedicated test identities

Create at least one account for each role you want tested, such as member, manager and administrator.

For each account:

• Use a non-production tenant or organization.
• Grant only the permissions the tests need.
• Turn on the feature flags the PR needs.
• Keep credentials in your secrets manager, and store the ones Pie uses in Credentials Manager.
• Disable CAPTCHA and interactive bot challenges for this identity or for Pie’s trusted network path.

Don’t use an employee’s account. Personal settings, stale sessions and manual edits make results hard to reproduce.

Write down what each account is for before testing:

Make login deterministic

Use a direct test login on the preview if you can. If the real sign-in uses one-time passwords, email links or single sign-on, give Pie a test-safe way through it. Pie can’t receive codes sent by SMS or read authenticator apps.

Two-factor authentication can stay on for the test account: store a fixed test code in the credential’s Password / OTP / magic link field. Describe any other sign-in steps in App Instructions.

Test-safe options:

• A sandbox inbox Pie can read through a stored credential
• A deterministic test OTP accepted only outside production
• A dedicated identity-provider test tenant
• A seeded session created through an audited test endpoint

Keep authorization real. A login shortcut mustn’t skip the role checks or feature flags the test needs to verify.

Seed named test data

Create small, named fixtures for the user paths you’re testing. A checkout test might need an active customer, an in-stock product, a saved address, and a sandbox payment method.

Each fixture should have:

• A stable identifier
• A documented owner
• A seed command, API, or setup workflow
• A reset procedure
• A clear expiry policy

Use unique prefixes for records created during a run, so tests on different PRs at the same time don’t edit the same data.

Ideally, one idempotent command seeds the data and returns a manifest:

./scripts/test-data seed \
  --profile order-ready-for-refund \
  --namespace pie-pr-123-run-456 \
  --output manifest.json

Run it twice before you trust it. The second run should return the same manifest, not a second set of records.

The manifest names the target records, so Pie doesn’t have to search the whole database:

{
  "profile": "order-ready-for-refund",
  "namespace": "pie-pr-123-run-456",
  "tenant": "pie-e2e",
  "requiredRoles": ["support-agent"],
  "requiredFlags": ["refund-workflow"],
  "records": {"order": "ORDER-REFUNDABLE-001"},
  "resetCommand": "./scripts/test-data reset --namespace pie-pr-123-run-456",
  "expiresAt": "<RUN_EXPIRY_ISO8601>"
}

Set expiresAt to a real timestamp far enough out to cover the run and its retries, and let the cleanup job delete the namespace once it passes. Use a fixed random seed when generated values vary, and store that seed in the manifest. Never seed real personal data, credentials, payment data or production tokens.

Reset between runs

Choose one reset strategy:

• Recreate the tenant from a fixture.
• Delete records with the run’s unique prefix.
• Restore a database snapshot in the preview namespace.
• Call idempotent cleanup APIs for the tested resources.

The reset must be safe to run more than once. It must never delete records outside the preview tenant or namespace.

Expose a readiness response with no secrets in it, so the deployment can hold Pie back before it tests missing data:

{
  "ready": true,
  "profile": "order-ready-for-refund",
  "checks": {
    "userExists": true,
    "roleAssigned": true,
    "flagEnabled": true,
    "targetRecordExists": true,
    "externalServicesAreTestMode": true
  }
}

Gate the preview on this response. If any check reports false, hold the run instead of letting Pie test a state that was never seeded.