Prepare Identity, Access, and Test Data

Pie needs a predictable starting state before it can test anything. Give it a dedicated identity, the same permissions your target user has, and data you can seed and reset without touching production. Pie then signs in on its own, lands on the changed feature with the right role, and begins every run from the same place. Application engineering usually owns the accounts and the fixtures.

Step 1: Create Dedicated Test Identities

Create at least one dedicated account for each role in scope, such as member, manager, and administrator.

For each account:

• Use a non-production tenant or organization.
• Grant only the permissions needed for the pilot.
• Enable the same feature flags required by the PR.
• Keep credentials in the agreed secret-management path, and store the ones Pie uses in Credential Manager.
• Disable CAPTCHA and interactive bot challenges for this identity or for Pie’s trusted network path.

Do not share an employee account. Personal settings, stale sessions, and manual edits make results hard to reproduce.

Record the contract before testing:

Step 2: Make Login Deterministic

Prefer a direct test login supported by the preview environment. If the real path includes one-time passwords, email links, or single sign-on, provide a test-safe way to complete that path. Pie does not intercept live SMS codes or authenticator apps, so two-factor authentication must be off for the test account.

Examples include:

• A sandbox inbox Pie can read through a stored credential
• A deterministic test OTP accepted only outside production
• A dedicated identity-provider test tenant
• A seeded session created through an audited test endpoint

Keep authorization real. Bypassing login must not also bypass role checks or feature flags that the test needs to verify.

Step 3: Seed Named Test Data

Create small, named fixtures for the user paths in scope. A checkout test might need an active customer, an in-stock product, a saved address, and a sandbox payment method.

Each fixture should have:

• A stable identifier
• A documented owner
• A seed command, API, or setup workflow
• A reset procedure
• A clear expiry policy

Use unique prefixes for records created during a run. This prevents concurrent PR tests from editing the same data.

Prefer one idempotent command that returns a manifest:

./scripts/test-data seed \
  --profile order-ready-for-refund \
  --namespace pie-pr-123-run-456 \
  --output manifest.json

Run it twice before you trust it. The second run should return the same manifest, not a second set of records.

The manifest should name the target records instead of asking Pie to search an unbounded database:

{
  "profile": "order-ready-for-refund",
  "namespace": "pie-pr-123-run-456",
  "tenant": "pie-e2e",
  "requiredRoles": ["support-agent"],
  "requiredFlags": ["refund-workflow"],
  "records": {"order": "ORDER-REFUNDABLE-001"},
  "resetCommand": "./scripts/test-data reset --namespace pie-pr-123-run-456",
  "expiresAt": "<RUN_EXPIRY_ISO8601>"
}

Set expiresAt to a real timestamp far enough out to cover the run and its retries, and let the cleanup job delete the namespace once it passes. Use a fixed random seed when generated values vary, and store that seed in the manifest. Never seed real personal data, credentials, payment data, or production tokens.

Step 4: Reset Between Runs

Choose one reset strategy:

• Recreate the tenant from a fixture.
• Delete records with the run’s unique prefix.
• Restore a database snapshot in the preview namespace.
• Call idempotent cleanup APIs for the tested resources.

The reset must be safe to run more than once. It must not delete records outside the preview tenant or namespace.

Expose a non-secret readiness response so the deployment can block Pie before a vacuous test starts:

{
  "ready": true,
  "profile": "order-ready-for-refund",
  "checks": {
    "userExists": true,
    "roleAssigned": true,
    "flagEnabled": true,
    "targetRecordExists": true,
    "externalServicesAreTestMode": true
  }
}

Gate the preview on this response. If any check reports false, hold the run rather than letting Pie test a state that was never seeded.

Check the Three Runs

Run the critical path three times with the same test identity. Confirm that:

1. Login completes without manual input.
2. The expected role and feature flags are visible.
3. Required records exist at the start.
4. Writes complete in the preview environment.
5. Reset returns the account to the same starting state.
6. A lower-privilege account cannot open the privileged route.
7. Seeding twice succeeds without duplicate or conflicting records.

Troubleshooting

• Pie signs in but cannot see the feature: Check organization membership, role propagation, feature flags, and cached sessions.
• The first run passes and later runs fail: Reset consumed or mutated data between runs.
• Tests collide: Give each PR or run its own tenant, namespace, or record prefix.
• A verification link expires: Use a sandbox inbox and generate the link during the run.

If sign-in works but the changed feature stays hidden, resolve the feature flag values for the test identity next.

Need Help?

• Support: support@pie.inc