# Pie Labs โ Complete Documentation
> Pie is an AI-native, autonomous QA platform. Upload your mobile (iOS/Android) or web app build and Pie's AI agents automatically discover features, generate test cases, execute tests, detect bugs, and produce detailed reports with visual replays.
>
> REST API: https://api.pie.inc (OpenAPI spec: https://api.pie.inc/docs/swagger)
> MCP Server: https://mcp.pie.inc
> Dashboard: https://app.pie.inc
---
---
## Welcome to Pie
## What is Pie?
Pie is an AI-native QA platform that fundamentally changes how teams approach testing. Tests are authored, executed, and maintained by AI agents, not humans writing scripts. You simply provide your application, and our agents explore it like real users would, discovering features, generating tests, and identifying bugs.
Think of Pie as your QA team's co-pilot. It handles the repetitive, time-consuming aspects of testing while your team focuses on building great products.
## What You'll Learn
New here? Start with the Quickstart. Want the mental model first? Read How Pie Transforms Your Workflow.
## Key Benefits
- **๐ Zero Setup**: No code changes or complex configurations required
- **โก Lightning Fast**: Comprehensive reports in 15-30 minutes
- **๐ค AI-Powered**: Autonomous test generation and execution
- **๐ Actionable Insights**: Clear, prioritized bug reports with visual replays
- **๐ [Self-Healing](/docs/how-it-works/contextual-understanding-self-healing/)**: Tests adapt to UI changes automatically
---
## How Pie Transforms Your Workflow
We've simplified the complexity of end-to-end (E2E) testing into a powerful, three-step framework: **Push, Probe, and Pass**. This is your new path from application to actionable insight.
## The Three-Step Framework
### Push: Give Us Your App, We'll Handle the Rest
Getting started is as simple as it gets. There's no need to alter your source code or deal with complex setups. Just "push" your application to us in whatever way works for you:
- **Web Apps**: Provide a URL.
- **Mobile Apps**: Drop your `.apk` (Android) or `.app` (iOS) build, zipped or not. See [Mobile Builds](/docs/integrations/mobile-builds/) for supported formats.
### Probe: Let Our AI Do the Heavy Lifting
Once you've pushed your app, our fleet of AI agents gets to work. This *probe* phase is a fully autonomous discovery and testing mission.
- **Deep Discovery**: Our agents crawl every screen and element to build a comprehensive map of your app, a deep, structural understanding of its features and user journeys.
- **Intelligent Test Generation**: Using this structural map, the agents generate hundreds of relevant E2E tests and run them in parallel across cloud devices (real and emulated devices in Pie's cloud infrastructure). This entire process, from discovery to execution, can take as little as 15-30 minutes.
### Pass: Get a Clear Verdict and Ship Faster
The final *pass* phase delivers a clear, data-driven verdict on your build's quality.
- **Instant Readiness Score**: An at-a-glance score tells you exactly how ready your build is for release.
- **Actionable Bug Reports**: Drill down into a prioritized list of deduplicated issues, complete with visual, step-by-step reproductions so your team can fix bugs fast.
- **Confident Go/No-Go Decisions**: Armed with clear data, you can make deployment decisions that accelerate your release velocity without compromising quality.
## Key Benefits
- No source code changes required
- AI-powered test generation and execution
- Faster, more confident releases
- Insightful, QA-validated results in minutes
---
## Quickstart Guide: Your First Test Run
Ready to run your first test? This guide walks you through the process. The onboarding is fast, intuitive, and gets you from setup to insights in minutes.
## Step 1: Create Your Account or Sign In
Your journey with Pie begins at the sign-in page. To get started:
- **Use your Google account**: Click "Continue with Google" for quick, password-free access.
- **Use your email**: Click "Continue with email" and enter your registered work email and password.
- **New to Pie?**: Click "Sign up" to create your account in moments.
Enterprise SSO is supported. Contact your account team for setup details. If your Google login prompts for 2FA, complete it as usual.
## Step 2: Provide Your Application
Once you're signed in, it's time to give Pie your application.
### For Mobile Apps
- You'll see a prompt: "Drop your app and let's do this."
- Drop your `.apk` (Android) or `.app` (iOS) build, or a zipped version of either, or use the "Browse Your Build" button.
- Pie runs [mobile app testing](https://pie.inc/blog/mobile-app-testing-guide/) across functional flows, visual states, and OS variants โ no test scripts required.
### For Web Apps
- Simply enter the URL of your web application when prompted.
## Step 3: Provide App Credentials (If Needed)
If your app requires a login to access its main features, Pie will prompt you to enter test credentials. We recommend using a dedicated test account rather than a real user account. Credentials are stored encrypted and used only during test execution. For more details, see the [Credential Manager](/docs/credentials/).
## Step 4: Let the Magic Happen
After submission, Pie begins mapping your app automatically. You can close the browser and we'll email you when your report is ready, or stay on the dashboard to watch in real time.
## Step 5: Watch It Live (Optional)
Open the active run from your dashboard to follow test execution in real time. Watch as our agents surface features and generate tests.
## Step 6: Review Your Report
Within **15-30 minutes**, your test report will be available on the dashboard. It includes:
- โ
**Readiness Score**
- ๐บ๏ธ **Summary of Key Features**
- ๐ **[Issues](/docs/core-concepts/issues-and-findings/) Found**
- ๐ **All Executed Test Cases**
The Readiness Score appears at the top of your run page. Issues and Test Cases are each available as separate tabs below.
## You're All Set!
That's it, your first test run is complete. In minutes, you've gone from signup to smart, actionable insights.
## Where to Go Next
---
## Connect Pie to Your AI Workspace
Access Pie's testing data directly from Claude, ChatGPT, or any MCP-compatible platform. Analyze test results, import test cases from CSV, find duplicates, manage issues, and generate reports - all through natural conversation.
## Setup Guide
### Step 1: Generate Your API Key
1. **Log in to app.pie.inc**
2. **Go to Settings โ API Keys**
3. **Click + Create API Key**
4. **Name your key** (e.g., "Claude Integration") and click **Create**
5. **Copy the key immediately** - you won't see it again
6. **Click Done** to acknowledge you've copied the key
### Step 2: Configure Your Platform
Choose your platform and follow the configuration steps:
#### Claude Desktop
1. **Open Claude Desktop โ Settings โ Developer โ Edit Config**
2. **Add Pie to your configuration file:**
```json
{
"mcpServers": {
"pie": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://mcp.pie.inc",
"--header",
"X-API-Key: YOUR_API_KEY_HERE"
]
}
}
}
```
> **Note:** Windows users should replace "npx" with "npx.cmd"
**For multiple Pie apps:**
```json
{
"mcpServers": {
"pie-staging": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://mcp.pie.inc",
"--header",
"X-API-Key: STAGING_API_KEY"
]
},
"pie-production": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://mcp.pie.inc",
"--header",
"X-API-Key: PRODUCTION_API_KEY"
]
}
}
}
```
3. **Save the file**
4. **Quit Claude completely** (not just close the window) and restart
#### Cursor
1. **Open Cursor settings and navigate to the MCP configuration file**
2. **Add Pie to your configuration:**
```json
{
"mcpServers": {
"pie": {
"url": "https://mcp.pie.inc",
"headers": {
"X-API-Key": "YOUR_API_KEY_HERE"
}
}
}
}
```
3. **Save the file and restart Cursor**
#### VS Code
1. **Open VS Code settings and locate the MCP servers configuration**
2. **Add Pie to your configuration:**
```json
{
"servers": {
"pie": {
"url": "https://mcp.pie.inc",
"headers": {
"X-API-Key": "YOUR_API_KEY_HERE"
}
}
}
}
```
> **Note:** Pass your API key in the `X-API-Key` header, the same way you would for Cursor or Claude.
3. **Save the file and reload VS Code**
#### Claude Code (CLI)
Run this command in your terminal:
```bash
claude mcp add --transport http pie https://mcp.pie.inc \
--header "X-API-Key: YOUR_API_KEY_HERE"
```
Replace `YOUR_API_KEY_HERE` with the API key you copied from Pie.
### Step 3: Test Your Connection
1. **Open your AI platform and type:**
```
Fetch Pie
```
2. **You should see an overview** of your app including test runs, coverage, and active issues.
## What You Can Do
### Key Workflows
**AI-Powered Bug Fixing:** Fetch issues with full context (repro steps, screenshots, expected/observed behavior) โ AI agent fixes code โ creates PRs. Single issues or batch operations.
**Test Case Management:** Create, run, update, archive test cases. Import from CSV, find duplicates, expand coverage.
**Issue Management:** Review, approve, reject, resolve issues. Cross-run analysis and duplicate detection.
**Reporting:** Weekly summaries, coverage analysis, trend analysis across runs.
For all 37 tools with natural language examples, see the [Complete Function Reference](available-functions).
## Need Help?
- **Documentation:** [docs.pie.inc](/docs)
- **Troubleshooting:** [MCP Troubleshooting](/docs/mcp/troubleshooting)
- **Support:** support@pie.inc
---
## Available Functions
Pie's MCP server exposes **37 tools** for regular users (plus 3 admin-only tools). Each tool can be invoked through natural conversation - just describe what you want and your AI assistant will call the right tool.
> **Looking for structured tool schemas?** See the [MCP Tool Reference](../tool-reference) for all tools with parameter types, required fields, and technical details.
## Issue Management
**Tools:** `get_issues`, `get_issue`, `approve_issues`, `reject_issues`, `resolve_issues`
### Review Issues
**Prompt:** `List the issues`
Fetches all active issues from your latest test run with severity, type, and descriptions.
**Prompt:** `Show me the details for issue ISS-123`
Gets full details including test case info, steps, assertions, and comments.
### Filter Issues
**Prompt:** `Show me all approved issues`
**Prompt:** `List issues that were first found in run R-456`
**Prompt:** `Show me all issues including resolved and rejected ones`
### Approve Issues
**Prompt:** `Approve issues ISS-101 and ISS-102 as valid bugs`
Confirms issues as valid after admin review. Changes triage state from "pending" to "approved".
**Prompt:** `Undo the approval on ISS-101`
Reverses a previous approval back to "pending".
### Reject Issues
**Prompt:** `Reject ISS-103 as a false positive because the expected behavior changed in the latest release`
Marks issues as false positives with a reason. Also rejects associated findings.
**Prompt:** `Undo the rejection on ISS-103`
Reverses a rejection, restoring the issue and unarchiving associated test cases.
### Resolve Issues
**Prompt:** `Resolve ISS-104 and ISS-105 - these have been fixed in the latest build`
Closes issues as fixed/resolved.
### Cross-Run Analysis
**Prompt:** `Check for issues approved in the latest run, and compare them to resolved issues from previous runs`
Compares current issues against historical runs to identify recurring issues and resolution trends.
### Find Duplicate Issues
**Prompt:** `List the duplicate issues, suggest which one to archive`
Identifies duplicates and recommends which to keep based on detail quality.
## Test Runs
**Tools:** `create_run`, `run_discovery`, `get_runs`
### View Run History
**Prompt:** `Show me all test runs`
Lists all runs with IDs, build info, status, and timestamps.
### Create a New Run
**Prompt:** `Create a new test run`
**Prompt:** `Create a run with build B-789 on iOS 18.2`
### Trigger Discovery
**Prompt:** `Run discovery on my app`
Starts an automated discovery process that explores your app and generates test cases. Only works when the app has no existing test cases.
## Key Features
**Tools:** `manage_key_features`, `get_available_icons`
Key features (also called "groups") organize test cases into logical categories like "Login", "Checkout", or "Profile Management".
### View All Features
**Prompt:** `List the key features and provide an overview`
Shows all features with test coverage summaries and test case counts.
### Create a Feature
**Prompt:** `Create a key feature called "Onboarding Flow" with description "New user registration and setup"`
### Update a Feature
**Prompt:** `Rename the "Login" feature to "Authentication" and update its description`
### Delete a Feature
**Prompt:** `Delete the "Legacy Checkout" feature`
### Get Feature Tests
**Prompt:** `List the test cases under [Key Feature Name]`
Shows all test cases associated with the specified feature.
## Scripts
**Tools:** `get_scripts`, `create_script`
Scripts are shell commands (typically curl commands) that Pie executes during test runs. They let tests interact with your backend - fetching test data, creating users, generating OTPs, or bypassing verification steps. Reference them in test steps using `#{script-name}` syntax, and Pie's AI agent executes them at the right moment during the test flow.
### List Scripts
**Prompt:** `Show me all available scripts`
Returns script IDs, names, and shell command instructions for each script.
### Create a Script
**Prompt:** `Create a script called "get-unique-phone" that runs: curl -X GET "https://api.example.com/test-phone" -H "Authorization: Bearer TOKEN"`
The script can then be referenced in any test case as `#{get-unique-phone}`.
### Use Scripts in Test Cases
**Prompt:** `Create a test case: Sign up with a new phone number from #{get-unique-phone}, complete the OTP flow using #{generate-otp}, and verify the user lands on the dashboard`
Scripts are referenced with `#{script-name}` in test prompts. Pie's AI agent executes each script when the test reaches the relevant step, captures the output, and uses the returned data (phone numbers, OTPs, user credentials, etc.) in the following test steps.
### Common Script Patterns
**Generating unique test data:**
**Prompt:** `Create a script called "create-test-user" with: curl -X POST "https://staging-api.example.com/test/users" -H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" -d '{"role": "premium"}'`
**Bypassing OTP/verification:**
**Prompt:** `Create a script called "generate-otp" with: curl -X POST "https://staging-api.example.com/test/otp" -H "Authorization: Bearer TOKEN" -d '{"phone": "{{use the phone number from signup}}"}'`
**Fetching environment-specific data:**
**Prompt:** `Create a script called "get-valid-card" with: curl -X GET "https://staging-api.example.com/test/payment-cards" -H "Authorization: Bearer TOKEN"`
Scripts support parameterization with `{{placeholder}}` syntax - values from earlier test steps or previous script outputs are automatically substituted. See [Script Parameterization](/docs/scripts/parameterization/) for details.
## App Information
**Tools:** `get_app`, `health_check`
### View App Config
**Prompt:** `Show me my app details`
Returns app ID, name, platform, configuration, and settings.
### Health Check
**Prompt:** `Check if Pie is connected`
Verifies connectivity and authentication with the Pie API.
---
## MCP Tool Reference
This page documents every tool exposed by the Pie MCP server (`https://mcp.pie.inc`) with structured parameter schemas. Use this reference when building integrations or when your AI coding assistant needs exact tool definitions.
For natural-language usage examples, see [Available Functions](../available-functions).
**MCP Server:** `https://mcp.pie.inc`
**Protocol Version:** 2024-11-05
**Transport:** Streamable HTTP
**Authentication:** `X-API-Key` header or `api_key` query parameter
## Test Case Management
### get_testcases
Fetch test cases with optional filtering. Returns simplified list by default (TestcaseID, Title, Description). By default returns only active test cases (excludes archived, in-creation, and issue-linked).
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| detailed | boolean | No | Set to `true` for full details (steps, assertions). Only use with `testCaseIds` to avoid large responses |
| testCaseIds | string | No | Comma-separated list of test case IDs (e.g., `1,2,3`) |
| archived | string | No | Filter: `true` for archived only, `false` for non-archived only |
| status | string | No | Comma-separated from `active`, `in-creation`, `archived`, `issue-linked` |
| testSuiteId | string | No | Filter by test suite ID |
| cursor | string | No | Pagination cursor |
### create_custom_testcase
Create and automatically queue a new test case from a natural language prompt. The test will be generated, queued, and executed automatically.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| prompt | string | Yes | Natural language description of the test case to create |
| runId | string | No | Run ID to execute against. Defaults to the latest default run |
| credentialId | string | No | Credential ID to use. Use `get_credentials` to find available IDs |
| bundleId | string | No | Override URL for web app tests |
### run_specific_testcases
Execute existing test cases by their IDs. If you have a description instead of IDs, call `get_testcases` first to find the matching IDs.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| testCaseIds | integer[] | Yes | Array of numeric test case IDs to execute (e.g., `[123, 456]`) |
| runId | string | No | Run ID to execute against. Defaults to the latest run |
### update_testcase
Update an existing test case's title, description, steps, assertions, and/or key feature assignment. Supports partial updates.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| testCaseId | integer | Yes | The ID of the test case to update |
| title | string | No | New title |
| description | string | No | New description |
| steps | string[] | No | New array of test steps |
| assertions | string[] | No | New array of assertions |
| groupId | string | No | Key feature ID to assign. Use `manage_key_features` with `action=get` to list available IDs |
| credentialId | string | No | Credential ID to assign |
| bundleId | string | No | Override URL for web app tests. Set empty string to clear |
### archive_testcases
Archive test cases by marking them as archived.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| testCaseIds | integer[] | Yes | Array of test case IDs to archive |
### unarchive_testcases
Unarchive previously archived test cases.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| testCaseIds | integer[] | Yes | Array of test case IDs to unarchive |
### add_testcase_to_suite
Add a custom test case to the main test suite (default suite). Removes the test case from `in-creation` status.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| testCaseId | integer | Yes | The ID of the test case to add to the main suite |
## Test Run Management
### create_run
Create a new test run.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| buildID | string | No | Build ID to associate with the run |
| osVersion | string | No | OS version for test execution (e.g., `18.2` for iOS, `33` for Android) |
### run_discovery
Create a new discovery run to automatically explore the app and generate test cases. Only works if the app has no existing test cases.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| buildID | string | No | Build ID to associate with the discovery run |
### get_runs
Fetch all test runs for the application. Returns run IDs, build info, status, and timestamps.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| cursor | string | No | Pagination cursor |
## Test Suites
### create_test_suite
Create a new test suite to organize test cases into logical groups.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| name | string | Yes | Suite name (e.g., "Smoke Tests", "Regression Suite") |
| testCaseIds | integer[] | No | Test case IDs to include |
| suiteInstructions | string | No | Custom instructions that override app-level instructions |
| suiteURL | string | No | Custom URL for browser platforms only |
| osVersion | string | No | Android OS version (e.g., `30`, `33`) |
### get_test_suites
List all test suites for the application. Returns suite names, IDs, and test case counts. No parameters.
### update_test_suite
Update an existing test suite. Supports partial updates.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| testSuiteId | string | Yes | ID of the test suite to update |
| name | string | No | New name |
| addTestCaseIds | integer[] | No | Test case IDs to add |
| removeTestCaseIds | integer[] | No | Test case IDs to remove |
| suiteInstructions | string | No | Updated custom instructions |
| suiteURL | string | No | Updated URL (browser platforms only) |
| osVersion | string | No | Updated Android OS version |
### delete_test_suite
Delete a test suite. Test cases are NOT deleted, just unlinked from the suite.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| testSuiteId | string | Yes | ID of the test suite to delete |
## Scripts
### get_scripts
Fetch all execution scripts for the current app. Scripts are shell commands referenced in test steps using `#{script-name}` syntax. No parameters.
### create_script
Create a new execution script.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| name | string | Yes | Script name used for `#{name}` syntax in test steps |
| instructions | string | Yes | Shell command(s) to execute when invoked |
## Advanced Testing
### trigger_rediscovery
Trigger rediscovery for an existing test case to detect product changes and auto-update the test.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| testCaseID | integer | Yes | The test case ID to trigger rediscovery for |
| rediscoveryInstructions | string | No | Context about what changed (e.g., "The navigation menu was redesigned") |
### get_step_doms
Fetch DOM (HTML) content for each step of a test case execution. DOMs can be large - use `stepIds` to limit scope.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| testCaseId | string | Yes | Test case ID to fetch DOMs for |
| runID | string | No | Run ID. Defaults to the latest run |
| stepIds | string | No | Comma-separated step IDs to filter (e.g., `1,3,5`) |
| cursor | string | No | Pagination cursor |
### generate_tests_on_local
Generate a terminal command template for running local test case exploration on localhost URLs.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| prompts | string[] | No | Array of test case descriptions/prompts |
| localhost_url | string | No | Local URL where the website is hosted |
### start_test_monitoring
Fetch test results and findings for a completed test case. Only call this after the test execution has finished.
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| testCaseId | string | Yes | Test case ID from completed test execution |
| runId | string | No | Run ID associated with the test execution |
## Utility
### health_check
Check the health of the Pie API service. Verifies connectivity and authentication. No parameters.
---
## MCP Troubleshooting
Having issues with your Pie MCP integration? This guide covers common problems and their solutions.
## Connection Issues
### Connection not working?
- **Restart your AI platform completely** after config changes
- **Verify API key has no extra spaces**
- **Check that the configuration syntax** matches your platform's format
- **For Claude Desktop on Windows**, ensure you're using "npx.cmd" instead of "npx"
### No data appearing?
- **Ensure you've run at least one test** in Pie
- **Verify you're using the correct app name** from your config
- **Try:** `Fetch Pie` and show me everything available
## API Key Issues
### API key invalid?
- **Regenerate the key** in Pie portal
- **Update config file** with new key
- **Restart your AI platform**
## Configuration Errors
### Configuration file errors?
- **Validate JSON syntax** using a JSON validator
- **Check for missing commas or brackets**
- **Ensure API key is wrapped in quotes**
- **Verify the configuration format** matches your platform (Claude Desktop, Cursor, VS Code, or Claude Code)
## Platform-Specific Issues
### Claude Desktop
- Ensure you're using the correct command format
- Check that npx is installed globally
- Verify the configuration file location
### Cursor
- Check MCP server configuration in settings
- Ensure the URL format is correct
- Verify headers are properly formatted
### VS Code
- Confirm the MCP extension is installed
- Check server configuration syntax
- Verify API key is in URL parameter format
### Claude Code CLI
- Ensure Claude CLI is properly installed
- Check command syntax and parameters
- Verify network connectivity
## Still Need Help?
- **Documentation:** [docs.pie.inc](/docs)
- **Support:** support@pie.inc
- **Community:** Join our Discord for peer support
---
## Core Concepts
This section covers the fundamental components of the Pie platform. Mastering these concepts will help you interpret reports effectively, manage your testing suite, and leverage the full power of autonomous QA.
## Platform Fundamentals
## Key Concepts at a Glance
- **๐ฏ Intelligent Curation**: One issue per bug, not hundreds of duplicates.
- **๐น Visual Replays**: See exactly what went wrong with step-by-step screenshots.
- **๐ง Self-Healing Tests**: Tests automatically adapt to UI changes, eliminating maintenance.
- **โ๏ธ Natural Language**: Create complex, custom tests using plain English prompts.
- **๐ The Readiness Score**: Get a clear, data-driven go/no-go signal for every build.
---
## The Dashboard
Your dashboard is more than just a report; it's an interactive command center for monitoring your app's health. It gives you a high-level overview and lets you dive deep into the details when you need to. Here's how to interpret each element to make informed decisions.
## The Readiness Score
Think of this as your build's health summary.
- **90-100%**: Build is stable and likely ready for the next stage.
- **70-89%**: Caution is needed; there may be minor issues to review.
- **Below 70%**: Signals critical problems that are likely release blockers.
Use this score to quickly decide if a build is worth a deeper look or needs to be sent back to development immediately.
## Test Run Summary
This section provides the raw numbers behind the Readiness Score.
- **Pass/Fail Ratio**: A sudden increase in failed tests from one run to the next is a strong indicator of a significant regression.
- **Hours Saved**: A great metric to demonstrate the ROI of autonomous testing to your stakeholders.
## Issue Distribution Chart
This chart helps you spot trends.
- A spike in **"Usability"** issues could indicate a recent UI change introducing friction.
- A consistent number of **"Functional"** bugs might highlight a fragile area of your codebase.
Use this visual to understand the nature of the problems, not just the quantity.
## Key Features & Issue Report
Move from overview to action.
- **Key Features**: Helps contextualize where problems are happening. For example, if the *Checkout Flow* has 5 critical issues, you know where your most urgent problem lies.
- **Issue Report**: Your prioritized to-do list. Start at the top with *Critical* issues and work your way down.
---
## Managing Test Cases
In Pie, a test case is a simple, human-readable sequence of actions and checks. We've made it incredibly easy to generate and manage them.
## Clear Structure
Every test case is made of:
- **Steps**: Actions the AI takes.
- **Assertions**: Expected outcomes the AI verifies.
All written in plain English.
## Autonomous Generation
You don't have to write tests anymore. Our AI agents autonomously generate hundreds of test cases based on their deep exploration of your app, covering everything from critical user flows to tricky edge cases.
## Create Custom Tests with Plain English
Need a specific test? Just tell the Pie Test Assistant what you want in natural language.
For example:
**"Sign up with a new user, complete the profile, and log out."**
The assistant will:
- Explore the flow
- Draft the test case
- Validate it
- Add it to your suite for all future runs
## Effortless Management
All your tests live in a central repository where you can:
- Review visual replays
- Edit steps
- Move tests between features
- Archive outdated tests to keep your suite clean and relevant
---
## Test Suites
Test suites are logical groupings of test cases that help you organize, manage, and execute tests efficiently. Pie automatically generates suites during app discovery, and you can create custom suites to match your testing workflow.
## What Are Test Suites?
A test suite is a collection of related test cases grouped together for organized execution. When Pie explores your application during discovery, it identifies distinct features and user flows, then organizes related test cases into suites automatically.
Suites give you control over what gets tested and when:
- **Targeted testing:** Run only the tests relevant to your current work
- **Faster feedback:** Execute a focused smoke suite instead of the full regression
- **Resource efficiency:** Avoid running unnecessary tests that slow down your workflow
- **Clear organization:** Group tests by feature, priority, or test type
## Auto-Generated Suites
When Pie completes discovery on your application, it automatically creates suites based on detected features. For example, an e-commerce application might generate:
| Suite | Coverage | Typical Test Count |
|-------|----------|-------------------|
| Product Catalog | Search, filtering, product details | 20-30 |
| Checkout Flow | Cart, payment, order confirmation | 20-25 |
| User Authentication | Login, registration, password reset | 10-15 |
| Smoke Tests | Critical paths across all features | 40-60 |
The exact suites depend on your application's functionality. Pie discovers and organizes them based on the features it finds during exploration.
## Accessing Suites Manager
From your Pie dashboard, locate the **Suites Manager** button in the header navigation (top right, next to "New Run"). Click it to open the Suites Manager modal.
The Suites Manager displays:
- **Suite name:** The logical grouping name
- **Test case count:** Number of tests in each suite
- **Last run status:** When the suite was last executed
- **Run Tests button:** One-click execution for each suite
## Creating Custom Suites
Custom suites let you group tests for specific purposes beyond the auto-generated organization.
### Step 1: Open Suites Manager
Click the **Suites Manager** button in the dashboard header.
### Step 2: Create New Suite
1. Click **Add Test Suite**
2. Enter a descriptive name (e.g., "Pre-Release Validation" or "Sprint 23 Features")
3. Click **Create Suite**
Your new suite appears in the list, ready for test cases.
### Step 3: Add Tests to Your Suite
1. Navigate to **Test Cases** in the sidebar
2. Select the test cases you want to add (check the boxes)
3. Click **Add to Suite** from the actions menu
4. Choose your target suite from the dropdown
5. Confirm the selection
> **Note:** Tests can belong to multiple suites simultaneously. Adding a test to a custom suite doesn't remove it from existing suites.
## Common Suite Patterns
### Release Validation Suite
Group your P0 critical tests for pre-release checks. Keep this suite lean (30-50 tests) for fast feedback before deployments.
### Feature-Specific Suite
When working on a specific feature, create a suite containing only tests for that area. This speeds up development cycles.
### Regression Buckets: P0, P1, P2
Create tiered suites by priority:
- **P0:** Must-pass tests (run daily)
- **P1:** Important tests (run 2-3x per week)
- **P2:** Full coverage (run weekly)
## Running Test Suites
Click the **Run Tests** button next to any suite in the Suites Manager to execute all tests in that suite immediately.
### Recommended Execution Patterns
Based on common customer workflows:
| Pattern | Suite | Frequency | Purpose |
|---------|-------|-----------|---------|
| Daily smoke test | Smoke Tests | Every day | Catch critical regressions quickly |
| Weekly regression | Full Regression | Weekly | Comprehensive coverage |
| Pre-release validation | Release Suite | Before deploys | Ensure release readiness |
| Sprint testing | Feature Suite | During sprint | Focused feature validation |
### Suite-Level Configuration
Each suite can have its own configuration overrides:
- **Credentials:** Assign specific [credentials](/docs/credentials) to a suite for testing different user roles. Your Admin Features suite might use admin credentials, while Customer Flows uses standard customer login.
- **App Instructions:** Provide suite-specific [app instructions](/docs/app-instructions) to guide test execution. A Checkout suite might include instructions about test payment methods.
## Managing Suite Size
Pie supports up to 250 test cases per suite for optimal execution performance. If you need more coverage:
- Split large suites into focused sub-suites
- Create a parent suite for full regression, child suites for daily runs
- Archive tests that are no longer relevant
### Merging Test Suites
To combine two suites:
1. Create a new target suite with a combined name
2. Add all test cases from both source suites
3. Verify the merged suite runs successfully
4. Archive or delete the original suites if no longer needed
## Best Practices
### Start With Auto-Generated Suites
Pie's discovery creates sensible groupings based on your app structure. Use these as your foundation before creating custom suites.
### Keep Smoke Suites Lean
Your daily smoke suite should contain 40-60 critical path tests. This balances coverage with execution speed for rapid feedback.
### Use Suites for Cost Control
Running comprehensive regression suites costs more than targeted smoke tests. Structure your suites to match your budget and testing cadence:
- **Daily:** Smoke suite (~50 tests)
- **Mid-week:** Core regression (~200 tests)
- **Weekly:** Full regression (all tests)
### Document Suite Purpose
Name suites descriptively so team members understand their purpose:
- โ
"Checkout Critical Path" instead of "Suite A"
- โ
"Admin Panel Regression" instead of "Admin Tests"
## Troubleshooting
### Tests Not Appearing in Suite
If tests you added aren't showing up:
- Refresh the Suites Manager page
- Verify the tests weren't archived
- Check if filter settings are hiding certain test types
### Suite Execution Failures
If an entire suite fails to execute:
- Verify app credentials are configured correctly
- Check if the app URL is accessible
- Review [app instructions](/docs/app-instructions) for conflicting settings
### Duplicate Tests Across Suites
Tests appearing in multiple suites is expected behavior. To consolidate:
- Identify which suite should be the "source of truth"
- Remove duplicates from other suites
- Use the primary suite as the basis for custom suites
## Related Features
Test suites work alongside other Pie features:
- **[Test Cases](/docs/core-concepts/test-cases-reimagined):** The individual tests that make up your suites
- **[Credentials](/docs/credentials):** Configure authentication for suite-level testing
- **[App Instructions](/docs/app-instructions):** Provide suite-specific context for test execution
- **[The Dashboard](/docs/core-concepts/the-dashboard):** Monitor suite execution results and trends
- **[MCP Integration](/docs/mcp):** Automate suite runs through CI/CD
## Need Help?
Contact your Pie support team if you have questions about suite management or test organization strategies.
---
## Creating Custom Tests
While Pie's autonomous discovery provides broad coverage, the Pie Test Assistant empowers you to target specific scenarios with surgical precision.
## Why Create a Custom Test?
- **New Features**: Test a brand-new feature before the AI has had a chance to fully explore it in multiple runs.
- **Complex Business Logic**: Verify specific, multi-step workflows that are critical to your business (e.g., "Apply a specific discount code, use a gift card, and pay the rest with a credit card").
- **High-Priority Edge Cases**: Ensure the app gracefully handles known edge cases that you want to check in every single regression run.
- **Specific User Roles**: Test a flow from the perspective of a particular user type, like an "Admin" or "Moderator," which requires a specific setup.
## How the Assistant Works
You don't write scripts - you write instructions. When you provide a prompt:
- The AI agent performs a live exploration of your app to understand the flow.
- It translates that exploration into a formal test case with clear, natural-language steps and assertions.
- It validates the test to ensure it works before saving it.
## How-To: Build Your First Custom Test
Let's create a custom test in 5 simple steps.
**Example**: Testing a user profile update flow.
1. **Navigate to the Test Cases Tab**
Go to the **"Test Cases"** section in your project.
2. **Open the Pie Test Assistant**
Click the **"Add Test Case"** or **"Ask Pie"** button.
3. **Write Your Prompt**
In the chat interface, type a step-by-step instruction in plain English.
**Example Prompt**:
`"Log in with user 'test@pie.inc' and password 'password123'. Navigate to the user profile page, click the 'Edit Profile' button, change the first name to 'Alex', and save the changes. Verify that the name on the profile page is updated to 'Alex'."`
4. **Let the AI Work**
The assistant will perform the actions in your app, identify elements, and draft the test steps.
5. **Review and Save**
Review the steps and assertions. If they look correct, click **"Save Test Case."**
It will now be added to your suite and run automatically in all future test runs for this project.
---
## Best Practices for Prompting Pie
Ready to create custom test cases that just work? Great prompts lead to great tests.
Pie understands natural language, so you can describe what you want to test in plain English. For complex flows or when you need precise control over a specific user path, these practices help you get exactly the test case you're looking for.
## Spell Out Every Action
You'll get more thorough test cases when you walk through the entire workflow in your prompt. Where should Pie navigate? What should it click? What data should it enter?
Think of your prompt as walking a teammate through the process. Mention which screen to go to, what button to click, which fields to fill, and how to save.
**Example:**
> "Go to **Settings** > **User Management**. Click **Add User**, enter a unique email address, select the Admin role from the dropdown, and click **Save**."
You can include as much detail as you need. Every action you mention becomes a step in your test case.
## Make Values Unique
Want fresh data for every test run? You can tell Pie to generate unique values by adding a note at the end of your prompt:
```
Note: To make the values unique, use the current date-timestamp{DDMMhhmmss}
```
When you ask for a "unique name" or "unique email" in your prompt, Pie appends a timestamp automatically. Each test run creates independent data with no collisions.
## Use Clear Navigation Paths
You'll help Pie reach the right screen faster when you use the `>` symbol to show the menu path:
- `Settings > Account > Security`
- `Dashboard > Reports > Monthly`
- `Admin > Users > Roles`
- `Shop > Categories > Electronics`
You can also mention where the navigation starts. Does it begin from the sidebar? A specific menu? The more specific you are, the more reliable your test.
**Example:**
> "From the sidebar, navigate to **Admin** > **Users** > **Roles**. Select the Editor role, enable 'Can Publish' permission, and save."
## Search Before You Act
Creating a record and need to find it later? You can include a search step to make sure Pie interacts with the exact record it created.
**Example:**
> "Go to **Products** > **Catalog**. Click **Add Product**, enter a unique product name, set price to 25.00, and save. Search for the newly created product, open it, change the price to 30.00, and save."
The search step connects the "create" portion of your test with the "interact" portion, especially helpful in environments with lots of existing data.
## See It In Action
Here's how these practices come together for different scenarios:
### Signup and Login Flow
> "Navigate to the Register page. Enter a unique email address, create a password with 'Test123!', accept the terms, and click **Sign Up**. Log out. Return to the Login page, enter the same email and password, and sign in. Verify the dashboard loads."
### Content Management
> "Go to **Admin** > **Content** > **Pages**. Click **Create Page**, enter a unique page title, add body text 'Test content for validation', set status to Draft, and save. Search for the newly created page, open it, change status to Published, and save. Verify the page appears in the Published list."
### E-commerce Checkout
> "Navigate to **Shop** > **Products**. Add the first available item to cart. Go to **Cart**, increase quantity to 2, verify the total updates. Click **Checkout**, enter a unique email and shipping address, select Express Shipping, and complete purchase with test card 4111111111111111."
### Settings Validation
> "Navigate to **Settings** > **Notifications**. Disable all email notifications and save. Refresh the page. Verify all email notifications remain disabled. Re-enable 'Order Updates' only and save."
### Record Deletion
> "Navigate to **Contacts** > **All Contacts**. Click **Add Contact**, enter a unique name and email, and save. Search for the newly created contact, select it, and click **Delete**. Confirm the deletion. Search for the same contact again and verify no results are found."
โน๏ธ **Tip:** For any test case that creates records, add the unique timestamp note at the end of your prompt to avoid data conflicts.
## Quick Checklist
Before you submit your prompt:
- [ ] Every action spelled out in sequence
- [ ] Navigation paths use `>` format
- [ ] Unique identifier note included when creating records
- [ ] Search steps before interacting with created records
- [ ] Clear endpoint for the test
---
## Understanding Issues, Findings, & Reports
One of the biggest headaches with automated testing is noise. A single bug can cause dozens of test failures, flooding you with redundant alerts. Pie solves this by intelligently curating bug reports.
## Intelligent Curation
Our platform distinguishes between:
- **Finding**: A single instance of a potential bug.
- **Issue**: A curated, deduplicated report representing a unique root cause.
After a run, our AI analyzes all findings and clusters similar ones together. You get one clear issue per bug, not a flood of alerts.
## The Actionable Issue Report
Each issue report includes:
- A clear title
- Comparison of expected vs. observed behavior
- Step-by-step reproduction instructions
- Video replay of the issue
When a new issue is created, the system also auto-generates a **"repro" test** to confirm it's legitimate and reproducible.
## Teach the AI with Your Feedback
You have full control over issue management:
- Change an issueโs severity
- Dismiss false positives
This feedback trains our AI to improve future runs, adapting its analysis to your app's unique context.
Dismissed something by mistake? Easily restore it from the **"Resolved"** tab.
---
## How-To: Read Your Test & Issue Reports
The value of testing lies in the clarity of its results. Hereโs how to interpret the reports Pie generates.
## Reading the Test Case List
After a run, the **"Test Cases"** tab will show you a list of every test that was executed.
- **Passed Test**: The AI agent completed every step and verified every assertion successfully.
- **Failed Test**: The agent encountered a blocker.
Click on a failed test to see:
- Which step failed and why
- Whether an element wasnโt found or an assertion failed
- A **visual replay** showing the exact moment of failure
## Anatomy of an Issue Report
This is the most critical report for your development team. Every issue is a self-contained, actionable bug report.
### Title & Severity
- Descriptive title generated by the AI (e.g., *"Application Crashes When Navigating to Settings"*)
- Severity (Critical, High, Medium, Low) is auto-assigned based on impact
- You can manually adjust severity as needed
### Expected vs. Observed Behavior
- A clear summary of what was expected vs. what actually happened
### Visual Replay
- Video or step-by-step screenshots showing the bug in real-time
- Eliminates "could not reproduce" issues
### Reproduction Steps
- Precise, numbered instructions to manually reproduce the bug
### Manage the Issue
- Use **"Dismiss"** if itโs not a real bug (this trains the AI)
- Push valid bugs directly into your development backlog using the **Jira integration**
---
## How Pie Works
To trust an autonomous system, you need to know how it works. Here's a look under the hood at the architecture that gives Pie its speed, scale, and intelligence.
## Architecture Deep Dive
## Why This Matters
- **๐ฌ Transparent Technology**: Understand exactly how decisions are made.
- **โก Parallel Processing**: Multiple AI agents work simultaneously for speed.
- **๐ง Contextual Intelligence**: AI understands your app's purpose, not just its structure.
- **๐ Self-Improving**: The system learns and gets smarter from every test run.
- **๐ก๏ธ Enterprise-Grade**: Built for scale, security, and reliability.
---
## Platform Architecture
The heart of Pie is not a single, generic AI but a fleet of specialized, task-specific AI modules. We build each module for a distinct purpose - such as discovery, execution, or bug analysis - and put it through rigorous analysis against a battery of module-specific metrics before it goes into production.
## Modular Backend Architecture
- Built with a **Go-based backend** designed for massive parallelization
- Jobs are distributed across **independent workers**, ensuring high **resilience** and **scalability**
## Hybrid AI Approach
- Combines **generative AI** with **classical machine learning** concepts
- ML systems act as a **control system** for generative components
- Ensures:
- **Intelligent exploration**
- **Reliable and deterministic outcomes**
---
## The End-to-End Test Flow
The entire process, from build to report, is a highly orchestrated symphony of parallel and asynchronous jobs. This is how we deliver comprehensive reports so quickly.
## Job Spawning
- When you provide your app, our backend immediately queues up a series of jobs.
## Parallel Discovery & Generation
- Multiple workers explore your app simultaneously
- Test cases are generated as they go
## Parallel Execution
- As soon as a test case is ready, itโs picked up and executed by the next available worker
- **Discovery**, **generation**, and **execution** happen concurrently
- For [mobile app testing](https://pie.inc/blog/mobile-app-testing-guide/), this covers OS versions, screen densities, and manufacturer skins concurrently
## Asynchronous Bug Analysis
- Screenshots are streamed to a separate **"bug judge"** service (powered by Gemini)
- Visual issues are analyzed **without blocking** test execution
## Asynchronous Issue Clustering
- When a bug is found, a cloud function groups it with similar findings
- Results in a **single, deduplicated issue**
## Final Report
- All parallel results are aggregated into the **unified dashboard report**
- Provides a complete picture of your buildโs quality
---
## How Self-Healing Tests Work
Pieโs ability to create relevant tests and adapt to application changes stems from its deep, structural understanding of the product.
## Building a Contextual Model
During the initial **Discovery** phase:
- AI agents perform an exhaustive crawl of the application
- They navigate to every screen, identify every interactive element, and map all possible user paths
- This information is synthesized into a **comprehensive contextual model**
A structured representation of:
- The application's UI
- Components
- Workflows
This model forms the foundation for all test generation and execution.
## No More Brittle Tests
A major pain point of traditional automated testing is test maintenance. Pie solves this with **Self-Healing Tests**.
- Tests are based on:
- The contextual model
- Natural-language intent (e.g., *"click the login button"*)
- Not dependent on brittle selectors (e.g., `click("button[id='submit-123']")`)
If a buttonโs ID or text changes:
- The AI can still identify the correct element using:
- Context
- Position
- Function
This intelligent adaptation allows tests to **"heal" themselves**, virtually eliminating the need for manual test maintenance.
---
## The Readiness Score
The **Readiness Score** is the ultimate output of a test run, designed to provide a clear, data-driven signal for deployment decisions.
- The score starts at **100%**
- It is **reduced** based on the **number and severity** of approved **critical issues** found
### Example
- Each verified critical issue might reduce the score by **8 points**
This system ensures the score accurately reflects **release-blocking bugs**, helping you make **confident go/no-go decisions**.
---
## Testing Capabilities
Pie offers a comprehensive suite of automated tests designed to cover your application from end to end. With over 25 distinct test types, our AI can autonomously achieve up to 80% E2E test coverage for most apps, right out of the box.
Explore the full range of our automated testing capabilities in the table below.
## Table: Pie's Automated Test Types
| Category | Test Type | Description |
|---------|-----------|-------------|
| **Discovery & Exploration** | Automatic App Crawling | Systematically discovers all screens, features, and hidden functionalities to build a comprehensive map of the application. |
| | Dynamic Element Detection | Identifies all interactive UI elements, including buttons, forms, dropdowns, modals, and tooltips. |
| | User Path Mapping | Documents all possible user journeys and navigation flows to understand how a user interacts with the app from start to finish. |
| | Feature Prioritization | Automatically ranks application features based on their inferred criticality and usage patterns to focus testing on high-impact areas. |
| **Functional Testing** | Form Validation & Submission | Verifies multi-step forms, input validation (e.g., email format, required fields), and correct error state handling. |
| | Authentication & Authorization | Tests login/logout flows, session validity, password resets, OTPs, and role-based access controls to ensure secure user access. |
| | Data CRUD Operations | Validates Create, Read, Update, and Delete functionality to ensure data is handled correctly throughout the application. |
| | File Upload/Download | Confirms that users can successfully upload and download files and that the application handles different file types and sizes correctly. |
| | Payment & Checkout Flows | Tests the entire transaction process, including adding items to a cart, applying discounts, processing payments, and confirming orders. |
| **UI/UX & Visual Testing** | Visual Regression Testing | Compares screenshots of the UI against a baseline to detect unintended visual changes in layout, color, or element alignment. |
| | Cross-Browser Compatibility | Checks for rendering differences and functional consistency across major web browsers like Chrome, Firefox, and Safari. |
| | Mobile Responsiveness | Verifies that the application's layout and functionality adapt correctly to different screen sizes, orientations, and viewports. |
| | Loading Performance | Monitors the application for appropriate loading indicators (e.g., spinners, skeleton screens) and handles timeout conditions gracefully. |
| | Error Message Clarity | Ensures that error messages are user-friendly, informative, and provide clear instructions for recovery. |
| | Interactive Element Feedback | Validates that interactive elements provide clear visual feedback to the user, such as hover states, click effects, and loading indicators. |
| **Advanced Capabilities** | Third-Party Integrations | Tests integrations with external services, including OAuth login flows, API callbacks, and embedded widgets. |
| | Complex Business Logic | Validates sophisticated conditional workflows, rule engines, and complex calculations to ensure they produce the correct outcomes. |
| | Dashboard & Analytics | Verifies the accuracy of data visualizations, charts, and report generation features. |
| | Scheduling & Calendar | Tests features involving booking, appointments, availability checks, and time-zone handling. |
| | AI-Powered Features | Validates the functionality of integrated AI features, such as chatbots, recommendation engines, and other AI-driven workflows. |
| | Custom File Handling | Allows you to upload specific files (e.g., test data) for the AI agent to use during test execution. |
| **Custom & Strategic Tests** | Edge Case Testing | Probes the application with boundary values, special characters, and extreme data inputs to uncover hidden bugs. |
| | Negative Testing | Intentionally provides invalid inputs or performs unauthorized actions to verify that the system handles errors gracefully and securely. |
| | Automated Regression Suites | Automatically runs a comprehensive suite of tests on every new build to ensure that recent changes have not broken existing functionality. |
| | Smoke Testing | Executes a quick set of tests on the most critical application paths after a deployment to verify that the build is stable. |
| | A/B Testing Validation | Checks the behavior of different application variants under feature flags to ensure consistency and proper functionality for all user segments. |
---
## Bots
### **How Pieโs Agents Work**
Pieโs automation agents are fully autonomous, visual-first systems designed to test applications without human intervention. Unlike traditional DOM-based tools, our agents rely entirely on visual data to understand and interact with the interface.
## Functional Model: Visual Cognition
The agent operates using a pure computer-vision approach, independent of the underlying code structure.
* **Screenshot-Based Intelligence:** The agent does not read the DOM tree. Instead, it captures high-fidelity screenshots of the interface and analyzes pixel data to identify buttons, forms, and navigation paths.
* **Visual Inference:** By processing these screenshots, the agent infers the state of the application and determines the next logical action based on visual cues, just as a human user would.
## Operating Modes
The agent cycles through distinct phases to ensure robust testing:
| Mode | Description |
| :--- | :--- |
| **Observe** | Captures visual snapshots of the current state to understand the UI layout and context. |
| **Act** | Performs UI interactions (clicks, typing, gestures) based on the visual analysis. |
| **Analyze** | Visually verifies the outcome of actions, detecting regressions or errors based on visual changes rather than code exceptions. |
| **Handoff** | Uploads results and visual artifacts to Pie for review. |
## Key Behaviors
### End-to-End Autonomous Control
The system is designed for fully autonomous execution.
* **No Human in the Loop:** Once a session begins, the agent takes full control of the environment. It manages the browser context independently, ensuring that the test proceeds from start to finish without any manual input or supervision.
### Agentic & Probabilistic Execution
Unlike rigid, script-based automation, Pieโs agents are **probabilistic and agentic**.
* **Dynamic Paths:** Because the agent makes decisions in real-time based on visual input, two consecutive runs may not be identical. The agent adapts to slight variances in the UI or timing, finding the best path forward dynamically rather than following a hard-coded sequence.
* **Resilience:** This non-deterministic approach allows the agent to navigate through unexpected pop-ups or layout shifts that would typically break standard "deterministic" scripts.
---
## Scripts
Scripts are shell commands that Pie executes during test runs. They let your tests interact with backend systems - fetching test data, creating users, generating OTPs, or calling any API your tests need. You reference scripts in test steps using the `#{script-name}` syntax, and Pie's AI agent decides when to execute them during the test flow.
## How Scripts Work
### The Execution Flow
1. **You create a script** with a name and a shell command (typically a curl command)
2. **You reference it in a test case** using `#{script-name}` in your test steps or prompt
3. **During test execution**, Pie's AI agent detects the script reference and runs the command
4. **The output is captured** (stdout) and made available to subsequent test steps
5. **The AI agent uses the output** - for example, entering a generated phone number into a form field
Scripts have a **5-minute timeout**. If a script doesn't complete within 5 minutes, execution continues and the failure is logged.
### What Makes This Powerful
Pie's AI agent doesn't just blindly run scripts at a fixed point. It understands the test context and decides _when_ to execute each script based on what the test needs. If your test says "sign up with a unique phone number from #{get-unique-phone}", the agent will:
1. Execute the script when it reaches the phone number input field
2. Read the script's output (the generated phone number)
3. Enter that value into the field
4. Continue with the rest of the test
This means scripts integrate naturally into test flows without rigid step ordering.
## Creating Scripts
### From the Dashboard
1. Log in to [app.pie.inc](https://app.pie.inc)
2. Navigate to **Settings > Scripts**
3. Click **+ Add**
4. Enter a **Script Name** (this becomes the `#{name}` reference)
5. Enter the **curl command** or shell command
6. Click **Save**
### From Your AI Workspace (MCP)
```
Create a script called "get-unique-phone" that runs:
curl -X GET "https://api.example.com/test-phone" -H "Authorization: Bearer TOKEN"
```
### Script Command Format
Scripts are shell commands executed on Pie's infrastructure. The most common format is a curl command:
```bash
curl -X POST "https://your-staging-api.com/api/create-test-user" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_STAGING_TOKEN" \
-d '{"role": "premium", "plan": "enterprise"}'
```
Your command should:
- Target your **staging or test environment**, never production
- Include proper **authentication headers**
- Return data in a **consistent format** (JSON recommended)
- Be **idempotent** when possible - safe to run multiple times
## Referencing Scripts in Test Cases
Use the `#{script-name}` syntax anywhere in your test case prompt or steps.
### Single Script Reference
**Test prompt:**
```
Sign up for a new account. Use #{get-unique-phone} to generate
a phone number for registration. Complete the signup flow and
verify the user lands on the dashboard.
```
Pie will execute the `get-unique-phone` script when it needs the phone number, capture the output, and use it in the signup form.
### Multiple Script References
A single test case can reference multiple scripts. They execute in the order the AI agent needs them:
**Test prompt:**
```
Create a new banking user by calling #{create-new-banking-user}.
Use the returned credentials to log in. Then call #{get-valid-debit-card}
to fetch payment details and complete the card linking flow.
```
### Chaining Script Outputs
Script outputs carry forward through the test. If your first script returns a user ID, your second script can use that value:
**Test prompt:**
```
Call #{create-test-user} to create a new user. Then call
#{generate-otp} using the phone number from the previous step.
Enter the OTP to complete verification.
```
Pie's AI agent tracks all executed script outputs and passes them to subsequent steps, so values flow naturally through your test.
## Common Use Cases
### Generating Unique Test Data
The most common use case. Tests that create accounts need unique identifiers every run:
| Script Name | Command | Purpose |
|---|---|---|
| `get-unique-phone` | `curl https://api.example.com/test-phone` | Generate a unique phone number for signup |
| `get-unique-email` | `curl https://api.example.com/test-email` | Generate a unique email address |
| `create-test-user` | `curl -X POST https://api.example.com/users -d '{"type":"test"}'` | Create a user and return credentials |
### Bypassing Verification Steps
OTP, KYC, CAPTCHA, and other verification steps block automated testing. Scripts let you call your backend directly:
| Script Name | Command | Purpose |
|---|---|---|
| `generate-otp` | `curl https://api.example.com/test-otp?phone=...` | Get the OTP from your backend instead of SMS |
| `bypass-kyc` | `curl -X POST https://api.example.com/verify-kyc -d '{"userId":"...","status":"approved"}'` | Mark a user as KYC-verified |
| `get-captcha-solution` | `curl https://api.example.com/test-captcha` | Get a valid CAPTCHA response for test environments |
### Fetching Environment-Specific Data
Tests need data that matches your staging environment:
| Script Name | Command | Purpose |
|---|---|---|
| `get-valid-debit-card` | `curl https://api.example.com/test-cards` | Fetch test payment card details |
| `get-store-locations` | `curl https://api.example.com/test-locations` | Get valid store/location data |
| `get-prequalified-user` | `curl https://api.example.com/test-users?credit_score=750` | Fetch a user profile matching specific criteria |
### Resetting Test State
Clean up between test runs to ensure consistent starting conditions:
| Script Name | Command | Purpose |
|---|---|---|
| `reset-cart` | `curl -X DELETE https://api.example.com/cart?user=test` | Clear the shopping cart |
| `reset-notifications` | `curl -X POST https://api.example.com/clear-notifications` | Clear notification state |
| `seed-products` | `curl -X POST https://api.example.com/seed-test-data` | Populate product catalog for testing |
## Real-World Example: Fintech Loan Application
Here's a complete example showing how scripts solve a real testing challenge.
**The problem:** Testing a loan approval flow requires a user with a specific credit score, valid SSN, and pre-verified identity. You can't create this manually every test run.
**Step 1: Create the scripts**
Script: `create-prequalified-user`
```bash
curl -X POST "https://staging-api.yourbank.com/test/create-user" \
-H "Authorization: Bearer STAGING_TOKEN" \
-H "Content-Type: application/json" \
-d '{"credit_score": 750, "income": 85000, "employment": "verified"}'
```
Script: `get-test-ssn`
```bash
curl -X GET "https://staging-api.yourbank.com/test/generate-ssn" \
-H "Authorization: Bearer STAGING_TOKEN"
```
Script: `generate-otp`
```bash
curl -X POST "https://staging-api.yourbank.com/test/generate-otp" \
-H "Authorization: Bearer STAGING_TOKEN" \
-H "Content-Type: application/json" \
-d '{"phone": "{{use the phone number from create-prequalified-user}}"}'
```
**Step 2: Create the test case**
```
Test the loan application flow end-to-end:
1. Call #{create-prequalified-user} to create a test user with good credit
2. Log in with the returned credentials
3. Navigate to "Apply for Loan"
4. Enter the SSN from #{get-test-ssn}
5. When prompted for OTP, call #{generate-otp} and enter the code
6. Complete the loan application form
7. Verify the loan is pre-approved with an offer displayed
```
**What happens during execution:**
- Pie calls `create-prequalified-user`, gets back `{"phone": "5551234567", "password": "TestPass1"}`
- Pie logs in with those credentials
- When it hits the SSN field, it calls `get-test-ssn`, gets `{"ssn": "123-45-6789"}`
- When OTP is needed, it calls `generate-otp` with the phone number from step 1
- The test completes the flow and verifies the loan offer
## Naming Conventions
Use clear, verb-first names that describe what the script does:
| Good | Avoid |
|---|---|
| `create-banking-user` | `script1` |
| `fetch-unique-phone` | `phone` |
| `get-valid-debit-card` | `card-data` |
| `generate-otp` | `otp` |
| `reset-cart` | `cleanup` |
Use lowercase letters and hyphens. The name becomes the `#{name}` reference, so make it readable in context: "Call #{create-banking-user}" reads better than "Call #{script1}".
## Best Practices
- **Target staging, never production.** Scripts run on every test execution. Make sure your commands hit test environments only.
- **Include authentication.** Most APIs require tokens or keys. Include them in the curl command.
- **Return JSON.** Pie's AI agent parses script output best when it's structured JSON.
- **Keep scripts focused.** One script should do one thing. Chain multiple scripts rather than building a single complex one.
- **Test manually first.** Run your curl command in a terminal to verify it works before adding it to Pie.
- **Use descriptive references.** `#{create-prequalified-loan-user}` is clearer than `#{user-script}` when reading test steps.
- **Handle errors in your API.** Return meaningful error messages so Pie can report what went wrong.
## Troubleshooting
### Script Not Executing
- Verify the script name in your test matches exactly (case-sensitive)
- Confirm the curl command is properly formatted (test it manually)
- Check that your server endpoint is accessible from Pie's infrastructure
### Authentication Errors
- Verify your API token is current and not expired
- Check the Authorization header format
- Ensure your test server accepts requests from external IPs
### Script Output Not Used
- Confirm your API returns data (not an empty response)
- Check that the response format is parseable (JSON recommended)
- If chaining scripts, verify the earlier script completed successfully
### Timeout Issues
Scripts time out after 5 minutes. If your script needs more time:
- Optimize the underlying API call
- Break long operations into smaller scripts
- Consider making the API endpoint asynchronous with a polling pattern
## Related Features
Scripts work alongside other Pie features for comprehensive testing:
* **[Credentials Manager](/docs/credentials):** Store login credentials for different user types that scripts can reference during test execution.
* **[App Instructions](/docs/app-instructions):** Define global behaviors and context for your application. Scripts can work with app instructions to handle complex workflows.
* **API Keys:** Authenticates Pie's access to your application
* **Custom Test Cases:** Uses scripts to test specific workflows
For information on these features, see their respective documentation sections.
## Need Help?
Contact your Pie support team with details about your use case and testing requirements.
---
## Script Parameterization
Want your scripts to use dynamic values instead of hardcoded data? You can reference values generated earlier in your test and pass them directly to your API calls.
## How It Works
During test execution, Pie generates unique values like phone numbers, emails, and names. You can reference these values in your scripts using curly-brace syntax:
```
{{use the phone number generated}}
```
When Pie executes your script, it automatically replaces the placeholder with the actual generated value. If Pie generated a phone number at step two, your script at step five will use that exact number.
## The Syntax
Use double curly braces with a descriptive reference inside:
```json
{
"phone_number": "{{use the unique phone number generated at the beginning of the test}}",
"password": "{{use the password created during signup}}"
}
```
**Naming Rules:**
Keep your parameter names clear and consistent:
- Use **lowercase** letters only
- Use **underscores** between words
- Keep names descriptive
| โ
Do This | โ Not This |
|------------|-------------|
| `phone_number` | `PhoneNumber` |
| `access_token` | `accessToken` |
| `user_password` | `PASSWORD` |
## Examples
### Example 1: Using Generated Test Data
Your test generates unique values during signup. Later, you need those same values in an API call:
**In your script's curl command:**
```
"phone_number": "{{use the unique phone number generated at the beginning of the test}}"
```
Pie substitutes the actual generated value when the script runs.
### Example 2: Using Password Values
Similar to other test data, passwords created during test flows can be referenced:
```
"password": "{{use the password created during signup}}"
```
You can reference any value that was generated or entered earlier in the test execution.
### Example 3: Chaining API Responses
Scripts can pass values between each other. You'll find this essential for multi-step authentication or setup flows:
1. **First script** makes an API call and receives a response (like a challenge request ID)
2. **Second script** uses that response value and returns something new (like an access token)
3. **Third script** uses the token from the second script as a Bearer token in its Authorization header
You describe the sequence in your test steps, and Pie handles passing the values automatically.
### Example 4: Using Tokens as Bearer Authentication
When your API requires authentication from a previous step:
```bash
curl -X POST "https://api.yourapp.com/endpoint" \
-H "Authorization: Bearer {{use access_token from login_script}}" \
-d ''
```
The access token from your earlier login script gets inserted automatically.
## Best Practices
- **Reference values clearly.** Use descriptive text inside the curly braces so it's obvious what value you need.
- **Chain scripts logically.** Make sure the script generating a value runs before the script that uses it.
- **Test with hardcoded values first.** Confirm your API call works with static data, then add parameterization.
- **Keep naming consistent.** Stick to lowercase and underscores throughout your scripts.
## Troubleshooting
**Parameter not being replaced?**
- Verify the value was generated in an earlier step
- Make sure the step that generates the value completed successfully before your script runs
- Check that your reference text matches what Pie expects
**Script failing with parameterization?**
- Test your curl command with a hardcoded value first to confirm the API endpoint works
- Add parameterization only after the basic script works
- Check the response format from previous scripts if you're chaining values
**Values not passing between scripts?**
- Confirm the earlier script ran successfully and returned the expected response
- Verify you're referencing the correct field name from the response
---
## Credential Manager
Credential Manager stores login credentials for different user types so Pie can test authenticated areas of your application. Store credentials once, and Pie handles the rest: testing across multiple user roles without manual login intervention.
## Understanding Credential Manager
### What Is Credential Manager?
Credential Manager is a secure storage system for test account login details. Store credentials for different user types (admins, standard users, premium members) so Pie can test how your application behaves for each role.
### Why Use Multiple Credentials?
Most applications have different user types with varying permissions. Testing with multiple credentials ensures:
- **Role-Based Access Validation:** Verify admins see management panels while standard users don't
- **Permission Testing:** Confirm restricted features are properly locked for unauthorized users
- **User Experience Coverage:** Test how different account types experience your application
- **Subscription Tier Testing:** Validate feature availability across free, premium, and enterprise accounts
### Common Use Cases
#### SaaS Platforms
For B2B software with role-based access, credentials enable testing across Admin, Manager, and Standard User roles to verify each sees appropriate dashboards and controls.
#### E-commerce Applications
Test customer accounts with different loyalty tiers, saved payment methods, or order histories to validate personalized experiences and checkout flows.
#### Healthcare & Financial Services
For applications with strict access controls, credentials allow testing that sensitive data is properly restricted based on user authorization levels.
## Adding Credentials
### Accessing Credential Manager
1. Log in to your Pie dashboard
2. Navigate to **Settings** from the left sidebar
3. Select **Credentials Manager**
### Adding a New Credential
1. Click the **+ Add** button
2. Provide a descriptive **Name** (nickname) that indicates the user type or role
3. Enter the **Test Login** (email, username, or phone number)
4. Enter the **Password / OTP / Magic Link**
5. Click **Save**
Each credential name must be unique. You can store unlimited credentials.
### Managing Existing Credentials
To edit or delete a credential, click the three-dot menu (โฎ) on the credential card and select the appropriate action.
The default credential (set during onboarding) cannot be deleted, but you can edit it. If you delete a credential that's assigned to test cases, those tests will automatically revert to using the default credential.
## Using Credentials in Test Cases
Once you've added credentials, you can select which credential to use when running custom test cases or editing existing tests. Each test case maintains its own credential assignment. Changing a credential for one test doesn't affect other tests in your suite.
### Running a Custom Test Case
1. Open **Pie Canvas** from your project
2. Enter your test case description in the prompt field
3. After submitting, Pie displays the **Confirm credentials** dialog
4. Select a credential from the dropdown (Default, or any saved credential)
5. Click **Continue** to run the test with your selected credential
You can switch credentials between prompts. Each prompt submission can use a different credential if needed.
### Editing an Existing Test Case
You can change credentials when editing or re-running an existing test case:
1. Open the test case you want to edit from your list
2. Click the **Edit with Pie** button to open the test in the assistant view
3. In the left panel, locate the **Choose Credentials** dropdown
4. Select the credentials you want to use for this test
5. The test case details will update to show **Logged in as:** displaying your selected credential
When you change a credential for a test case, all subsequent runs of that test will use the new credential until you change it again.
### Credential Options
The credential dropdown includes:
- **Default:** The credential provided during your first Discovery run
- **Saved credentials:** Any additional credentials you've added via Credential Manager
- **No credential:** Run the test without logging in (for testing public pages)
- **+ Add new:** Quickly add a new credential without leaving the test flow
## Best Practices
### Naming Conventions
Use clear, descriptive names that indicate the user type, role, or access level. This makes selecting the right credential from the dropdown straightforward.
Examples:
- `admin-full-access`
- `standard-user`
- `premium-customer`
- `guest-readonly`
### Account Configuration
Configure test accounts appropriately:
- **Disable 2FA:** Two-factor authentication blocks AI agents from completing login flows
- **Use dedicated test accounts:** Never use production accounts or real user credentials
- **Maintain active status:** Ensure accounts aren't locked, expired, or pending verification
- **Update passwords promptly:** When credentials change, update them immediately to prevent test failures
### Coverage Strategy
Store credentials for each distinct user role in your application. This enables comprehensive testing of permission boundaries and role-specific features.
For applications where the same flow behaves differently based on user type (like a dashboard that shows different data per role), create separate test cases for each credential to validate all variations.
## Troubleshooting
### Login Failures
If Pie can't log in with stored credentials:
- Verify the username and password are correct
- Confirm the account is active and not locked
- Check that 2FA is disabled for the test account
- Ensure the password hasn't expired or been changed
### Wrong Credential Used
If Pie uses an unexpected credential:
- Check that you selected the correct credential from the dropdown before running the test
- Verify the **Logged in as** indicator shows the expected credential in the test case view
- If the credential was recently deleted, the test may have reverted to the default credential
### Access Denied by Application
If your application blocks Pie's login attempts:
- Check if bot detection or rate limiting is enabled
- Verify Pie's IP addresses are whitelisted if required
- Contact your Pie account manager for infrastructure configuration
## Related Features
Credential Manager works alongside other Pie features for comprehensive testing:
- **[Scripts](/docs/scripts):** Fetch test data or create user profiles during test execution
- **API Keys:** Authenticates Pie's access to your application
- **Custom Test Cases:** Uses credentials to test specific workflows
## Need Help?
Contact your Pie support team with details about your use case and testing requirements.
---
## Integrations & Automation
Pie is built to be a collaborative tool that slots directly into your team's existing development lifecycle. We integrate with the tools you already use for CI/CD, issue tracking, and communication.
## Core Integration Features
## Integration Benefits
- **๐ Seamless Workflow**: Connect with your existing tools and processes.
- **๐ค Automated Triggers**: Run tests automatically on code changes.
- **๐ Centralized Reporting**: Get results where your team already works.
- **โก Real-time Notifications**: Stay informed with instant alerts.
- **๐ Continuous Quality**: Maintain high standards across every release.
---
## Available Integrations
The platform offers a growing list of integrations to support workflow automation. The following describes the integrations that are currently available and provides context on those planned for future releases. This transparency is intended to help teams plan their implementation effectively, distinguishing between what is fully available today versus what is on the near-term roadmap.
## Pull Request Testing: Pie Bot
**Pie Bot** is the GitHub App that powers [Pie Loop](/docs/pie-loop/). Once installed on a repository, it runs on every pull request automatically:
- Reads the PR diff and tests the flows your change affects
- Validates each finding against your source code, so no false positives reach you
- Posts test results and status checks on the pull request
- Opens a PR fix for any confirmed regression, with before/after screenshots
- Folds every run into your weekly [Coverage Stories](/docs/pie-loop/coverage-stories/)
See the [GitHub integration overview](/docs/integrations/github/), or jump straight to [Connect Pie Bot](/docs/pie-loop/connect-pie-bot/).
## Notifications: Slack
The primary integration for real-time notifications is **Slack**. Teams can connect Pie to a designated Slack channel to receive automated alerts. These notifications include:
- Confirmation that a build has been received and a test run has started
- Notification when a test run is complete and the report is ready for review
- Weekly summaries of testing activity and key quality metrics
## Issue Tracking: Jira
While a fully automated, two-way synchronization with **Jira** is on the roadmap, the platform currently supports a **manual push** feature.
- From any Issue Report within Pie, users can click a button to create a ticket in their Jira project
- Pie automatically pre-fills the Jira ticket with:
- Issue description
- Severity
- Steps to reproduce
- Link back to the visual replay in Pie
This streamlines the bug-reporting process and eliminates the need for manual copy-pasting.
## CI/CD: GitHub Actions & API
For full automation, Pie can be integrated directly into a **CI/CD pipeline**.
- **GitHub Action**: Recommended method for triggering new Pie test runs when:
- A new build is created
- A pull request is merged
- **REST API** ([Full API Reference โ](https://api.pie.inc)):
For teams using platforms like Jenkins or CircleCI, or needing custom logic:
- Offers programmatic control to start tests
- Enables retrieval of test results
- Supports flexible integration into any pipeline
- [Interactive API docs with OpenAPI 3.0 spec](https://api.pie.inc) - 18 endpoints covering apps, builds, test cases, issues, results, and execution
---
## Table: Integration Summary
The following table provides a clear summary of the current status of Pie's key integrations. This helps manage expectations by distinguishing between currently available features and those planned for the future.
| Category | Tool | Status | Description |
|----------|------|--------|-------------|
| **Pull Request Testing** | Pie Bot | **Available** | Automatically test every pull request, post results and status checks, and open PR fixes for regressions. Powered by [Pie Loop](/docs/pie-loop/). |
| **Notifications** | Slack | **Available** | Receive real-time alerts on test run status in your designated Slack channel. |
| | Email | On Roadmap | Automated email alerts for test run completion and reports are planned for a future release. |
| **Issue Tracking** | Jira | **Manual Push** | Manually create a Jira ticket from any issue in Pie with pre-filled details and a link back to the report. |
| | Linear, Asana | On Roadmap | Direct, one-click integration with Linear and Asana is planned for future releases. |
| **CI/CD** | GitHub Actions | **Available** | Use the official Pie GitHub Action to trigger test runs automatically from your GitHub workflow. |
| | Jenkins, CircleCI | Via API | Integrate with Jenkins, CircleCI, or other platforms using the Pie API for custom automation. |
---
## Automating Build Uploads (CI/CD)
Automate your testing workflow by integrating Pie with your CI/CD pipeline using our GitHub Action. This eliminates manual build uploads and ensures your latest builds are automatically tested whenever you create a release, merge a pull request, or on a scheduled basis.
## Overview
The [Pie Build Uploader GitHub Action](https://github.com/pielabsai/upload-build-action) supports:
- **iOS builds** (.app files)
- **Android builds** (.apk files)
- **Automatic zipping** of build directories
- **Secure API authentication**
- **Flexible trigger strategies** (releases, schedules, manual, etc.)
## Prerequisites
Before setting up the GitHub Action, you need:
1. **A Pie account** - Sign up at [app.pie.inc](https://app.pie.inc)
2. **A Pie API key** - Generate one from your app's settings in the Pie dashboard
3. **A GitHub repository** with your mobile app project
## Setup Instructions
### 1. Store Your API Key
Add your Pie API key as a GitHub secret:
1. Go to your repository's **Settings** โ **Secrets and variables** โ **Actions**
2. Click **New repository secret**
3. Name it `PIE_API_KEY`
4. Paste your Pie API key as the value
### 2. Add the Action to Your Workflow
Create or update a workflow file in `.github/workflows/` (e.g., `upload-to-pie.yml`):
```yaml
name: Upload Build to Pie
on:
release:
types: [published]
jobs:
upload:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
# Your build steps here
- name: Upload to Pie
uses: pielabsai/upload-build-action@v1.2
with:
pie_api_key: ${{ secrets.PIE_API_KEY }}
build_path: path/to/your/build.app # or .apk
```
## Common Trigger Strategies
### On Release or Tag
Upload builds automatically when you publish a release:
```yaml
on:
release:
types: [published]
```
Or when you push a version tag:
```yaml
on:
push:
tags:
- 'v*' # Matches v1.0, v2.1.0, etc.
```
### Manual Trigger
Allow manual uploads from the GitHub Actions UI:
```yaml
on:
workflow_dispatch:
```
### Scheduled Uploads
Run daily at midnight UTC:
```yaml
on:
schedule:
- cron: '0 0 * * *'
```
### On Pull Request Merge
Upload builds when PRs are merged to main:
```yaml
on:
pull_request:
branches: [main]
types: [closed]
jobs:
upload:
if: github.event.pull_request.merged == true
# ... rest of job configuration
```
## Complete Example
Here's a complete workflow that builds and uploads an Android APK:
```yaml
name: Build and Upload to Pie
on:
push:
tags:
- 'v*'
jobs:
build-and-upload:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Set up JDK
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
- name: Build APK
run: ./gradlew assembleRelease
- name: Upload to Pie
uses: pielabsai/upload-build-action@v1.2
with:
pie_api_key: ${{ secrets.PIE_API_KEY }}
build_path: app/build/outputs/apk/release/app-release.apk
```
## Configuration Options
| Input | Description | Required |
|-------|-------------|----------|
| `pie_api_key` | Your Pie API key (use secrets) | Yes |
| `build_path` | Path to your build file or directory | Yes |
## Next Steps
Once your workflow is set up:
1. The action will automatically upload your builds to Pie
2. Pie will begin testing your app based on your configured test settings
3. View test results in your [Pie dashboard](https://app.pie.inc)
## Additional Resources
- [Pie Build Uploader GitHub Action Repository](https://github.com/pielabsai/upload-build-action)
- [GitHub Actions Documentation](https://docs.github.com/en/actions)
## Troubleshooting
**Build upload fails:**
- Verify your API key is correctly stored in GitHub secrets
- Check that the build path is correct and the file exists
- Ensure the build file format is supported (.app, .apk, or .zip)
**Need help?** Contact support or open an issue on the [GitHub Action repository](https://github.com/pielabsai/upload-build-action).
---
## Mobile Build Preparation
## Overview
Before Pie's AI agents can test your mobile application, you need to prepare and upload a test-ready build. This section provides comprehensive guides for different mobile platforms and frameworks.
## Choose Your Platform
Select the guide that matches your app's platform and framework:
## Prerequisites for All Mobile Builds
Before preparing any mobile build for Pie testing, ensure you have:
### Environment Configuration
- **Sandbox/Staging Environment**: Configure your app to connect to a test or staging environment, never production
- **Test Endpoints**: Verify your app can connect to endpoints accessible from the external internet
- **Test Accounts**: Create dedicated test accounts that Pie agents can use without disrupting real users
### Build Requirements
- **Development Tools**: Have the appropriate IDE and SDK installed (Xcode for iOS, Android Studio for Android)
- **Latest OS Versions**: Configure simulators/emulators with recent OS versions โ [Android device fragmentation](https://pie.inc/blog/mobile-app-testing-guide/#challenges) spans 25,000+ variants, making OS version coverage the baseline for meaningful results
- **Custom Test Files**: Prepare any custom test files your app requires (these can be uploaded separately to Pie)
## What Happens After Upload?
Once you upload your build to [Pie's upload portal](https://app.pie.inc/upload):
1. **Automatic Discovery**: Pie's AI agents begin exploring your app within 30 minutes
2. **Test Generation**: AI automatically generates comprehensive test coverage
3. **Issue Detection**: Any problems found are reported in actionable issue reports
## Build Process Overview
Each platform has specific requirements:
- **iOS Native**: Requires Xcode and iOS Simulator build
- **React Native iOS**: Requires JavaScript bundling and Xcode configuration
- **Android**: Requires Android Studio and APK generation
## Need Help?
For platform-specific instructions, select your platform from the guides above. If you encounter any issues during the build preparation process, contact the Pie Labs support team for assistance.
---
## iOS Native Apps
## Overview
This guide covers building iOS native applications (Swift or Objective-C) for Pie testing. The build must be a simulator build that can run independently without connecting to external services during startup.
## What You'll Need
- **Xcode**: Latest version installed on macOS
- **iOS Project**: Your native iOS app project
- **iOS Simulator**: Configured with the latest iOS version
- **Build Access**: Permissions to access project build settings
## Step-by-Step Instructions
### 1. Launch Your Project in Xcode
1. Open your iOS app project in Xcode
2. Select your app as the build target from the scheme selector
3. Choose a simulator with the latest iOS version available (not a physical device)
### 2. Clean Your Build Environment
1. Navigate to **Product** menu โ **Clean Build Folder** (โงโK)
2. This ensures a fresh build without cached artifacts that might cause issues
### 3. Build Your Application
1. Go to **Product** menu โ **Build** (โB)
2. Wait for the build process to complete successfully
3. Verify there are no build errors in the Issue Navigator
### 4. Locate Your Build
1. Select **Product** menu โ **Show Build Folder in Finder**
2. Navigate to: `Build/Products/Debug-iphonesimulator/` (default path)
3. Find your `.app` file (it will appear as type "Application")
**Note**: If you have customized your build settings or product name, the location may differ. Look for a folder ending in `-iphonesimulator`.
### 5. Package for Upload
1. Right-click the `.app` file in Finder
2. Select **"Compress"** to create a `.zip` archive
3. Your build is now ready for upload to [Pie's upload portal](https://app.pie.inc/upload)
## Important Considerations
### Build Type
- The build **must be a simulator build**, not a device build
- Device builds use different architectures and will not work with Pie's testing infrastructure
### Environment Configuration
- Ensure all environment variables point to your test or staging servers
- Verify API endpoints are accessible from external networks
- Remove any development-only features or debug panels that should not be tested
### Testing Your Build
Before uploading, verify your build works correctly:
1. Open iOS Simulator
2. Drag your `.app` file into the simulator to install it
3. Launch the app and verify it starts successfully
4. Test basic navigation and core functionality
5. Confirm it connects to your test environment, not production
## Troubleshooting
### Build Location Not Found
**Problem**: Cannot find the build in the default location.
**Solutions**:
- Check if you have custom build paths in your project settings
- Search for `.app` files in the entire `Build/Products/` directory
- Ensure the build completed successfully without errors
### Simulator vs Device Build Error
**Problem**: Pie rejects the uploaded build.
**Solutions**:
- Verify you selected an iOS Simulator (not a device) as the build destination
- Check the build path ends with `-iphonesimulator`, not `-iphoneos`
- Rebuild after confirming simulator is selected
### App Crashes on Launch
**Problem**: App crashes immediately when opened in simulator.
**Solutions**:
- Verify all required frameworks and dependencies are included
- Check console logs in Xcode for specific error messages
- Ensure app is configured for test environment, not production
- Confirm all environment variables are properly set
### Network Connection Issues
**Problem**: App cannot connect to backend services.
**Solutions**:
- Verify test endpoints are accessible from external networks
- Check firewall rules allow incoming connections
- Confirm API keys and credentials are for test environment
- Test endpoint connectivity from a different network
## Next Steps
After preparing your build:
1. Upload it to [Pie's upload portal](https://app.pie.inc/upload)
2. Review the [Post-Upload Requirements](/docs/integrations/mobile-builds/post-upload-requirements) for information you'll need to provide
3. Pie's AI agents will begin testing within 30 minutes
## Need Help?
If you encounter issues not covered in this guide, contact the Pie Labs support team with:
- A description of the problem
- Build error messages (if any)
- Your Xcode version and macOS version
- Screenshots of any relevant error dialogs
---
## React Native iOS Apps
## Overview
This guide covers building React Native iOS applications for Pie testing. The build must be self-contained with all JavaScript code bundled, so it can run without connecting to Metro bundler.
**Important**: These instructions apply to React Native CLI projects. Expo projects require a different build process.
## What You'll Need
- **React Native CLI**: Version 0.60 or higher
- **macOS Computer**: iOS builds cannot be created on Windows or Linux
- **Xcode**: Latest version installed with command line tools
- **Node.js**: With npm or yarn installed
- **CocoaPods**: If your project uses native dependencies
## Choose Your Build Method
React Native projects can be built using two methods:
1. **[Manual Bundling](#manual-bundling-method)**: Full control over the bundling process (recommended for first-time builds)
2. **[Automated Bundling](#automated-bundling-method)**: Faster process if your project is already configured
**Which method should I use?** Check your project's Build Phases in Xcode. If you see "Bundle React Native code and images", use the automated method. Otherwise, use manual bundling.
## Manual Bundling Method
### Step 1: Install Dependencies
Open terminal in your project root:
```bash
# Install JavaScript dependencies
npm install
# or
yarn install
# Install CocoaPods dependencies (if applicable)
cd ios && pod install && cd ..
```
### Step 2: Stop Metro Bundler
Ensure no Metro bundler is running to avoid conflicts:
```bash
# Kill any running Metro instances
killall -9 node
```
Alternatively, close any terminal windows running Metro bundler.
### Step 3: Bundle JavaScript Code
Run the bundle command from your project root:
```bash
npx react-native bundle \
--entry-file index.js \
--platform ios \
--dev false \
--bundle-output ios/main.jsbundle \
--assets-dest ios
```
**Entry file variations**: Your project might use:
- `index.js` - Standard JavaScript entry
- `index.tsx` - TypeScript entry
- `index.ios.js` - Platform-specific entry
- Custom entry defined in `package.json` or `app.json`
**This command creates**:
- Production JavaScript bundle at `ios/main.jsbundle`
- Asset files (images, fonts) in the `ios` folder
- Hermes bytecode files (`.hbc`) if using Hermes engine
- An assets folder (only if your project has images or fonts)
### Step 4: Add Bundle to Xcode
1. Open your iOS project:
```bash
open ios/YourProjectName.xcworkspace
```
(or `.xcodeproj` if not using CocoaPods)
2. In **Project Navigator** (left panel), right-click your project name
3. Select **"Add Files to '[YourProjectName]'"**
4. Navigate to the `ios` directory
5. Select `main.jsbundle` and the assets folder (if it exists)
6. **Important checkboxes**:
- โ
**"Copy items if needed"** must be checked
- โ
Your app target must be selected under **"Add to targets"**
7. Click **"Add"**
**Verify**: Click `main.jsbundle` in Project Navigator and check the File Inspector (right panel) to confirm your target is checked under "Target Membership".
### Step 5: Configure JavaScript Loading
Update your AppDelegate to load the bundled JavaScript instead of Metro.
#### For Objective-C (`AppDelegate.m`)
Find the `jsCodeLocation` configuration in the `application:didFinishLaunchingWithOptions:` method.
**Replace the existing code with**:
```objc
jsCodeLocation = [[NSBundle mainBundle]
URLForResource:@"main"
withExtension:@"jsbundle"];
```
If you see conditional DEBUG code like this, the Release build already uses the bundle:
```objc
#if DEBUG
jsCodeLocation = [[RCTBundleURLProvider sharedSettings]
jsBundleURLForBundleRoot:@"index"];
#else
jsCodeLocation = [[NSBundle mainBundle]
URLForResource:@"main"
withExtension:@"jsbundle"];
#endif
```
#### For Swift (`AppDelegate.swift`)
**Replace the existing code with**:
```swift
let jsCodeLocation = Bundle.main.url(forResource: "main",
withExtension: "jsbundle")
```
If conditional DEBUG code is present, the Release build already uses the bundle:
```swift
#if DEBUG
let jsCodeLocation = RCTBundleURLProvider.sharedSettings()
.jsBundleURL(forBundleRoot: "index")
#else
let jsCodeLocation = Bundle.main.url(forResource: "main",
withExtension: "jsbundle")
#endif
```
### Step 6: Build the App
1. Select your target simulator in Xcode (latest iOS version recommended)
2. Choose your build scheme:
- **Debug** - Larger file size, includes diagnostics
- **Release** - Optimized build (recommended for testing)
3. Navigate to **Product** โ **Clean Build Folder** (โงโK)
4. Navigate to **Product** โ **Build** (โB)
5. Wait for the build to complete successfully
### Step 7: Test Your Build Offline
**This is a critical verification step:**
1. **Disable network connectivity** (turn off WiFi)
2. **Close all terminal windows** (ensure Metro isn't running)
3. **Open iOS Simulator**
4. Navigate to **Product** โ **Show Build Folder in Finder**
5. Go to:
- Debug: `Build/Products/Debug-iphonesimulator/`
- Release: `Build/Products/Release-iphonesimulator/`
6. **Drag the `.app` file** into the simulator to install it
7. **Launch the app** and verify it starts without errors
8. **Test core functionality** to confirm everything works
9. **Re-enable network**
If the app works offline, your bundle is correctly configured and ready for upload!
### Step 8: Package for Upload
1. Right-click the `.app` file in Finder
2. Select **"Compress"** to create a `.zip` archive
3. Upload to [Pie's upload portal](https://app.pie.inc/upload)
### Step 9: Restore Development Settings (Optional)
If you modified `AppDelegate.m` or `AppDelegate.swift`, restore the original code to continue local development with Metro bundler.
## Automated Bundling Method
If your project has the "Bundle React Native code and images" build phase configured:
### Quick Steps
1. Open your project:
```bash
open ios/YourProjectName.xcworkspace
```
2. Select your app target โ **Build Settings**
3. Search for **"Build Configuration"** and set it to **"Release"**
4. Navigate to **Product** โ **Clean Build Folder** (โงโK)
5. Navigate to **Product** โ **Build** (โB)
6. The JavaScript bundle is created automatically during build
7. Locate your `.app` file in **Product** โ **Show Build Folder in Finder**
8. [Test offline](#step-7-test-your-build-offline), compress to `.zip`, and upload
**Advantage**: No manual bundling or code modifications needed. The Release scheme handles everything automatically.
## Troubleshooting
### Blank or White Screen
**Possible Causes**:
- Bundle not found in app bundle
- Incorrect entry file specified
- Hermes compatibility issues
**Solutions**:
- Verify `main.jsbundle` has Target Membership checked in Xcode
- Confirm entry file name (check `package.json` for custom entry)
- If using Hermes, ensure all dependencies are compatible
### App Tries to Connect to Metro (localhost:8081)
**Possible Causes**:
- AppDelegate still in development mode
- Debug scheme selected instead of Release
- Cached build artifacts
**Solutions**:
- Verify AppDelegate changes were saved
- Switch to Release scheme or update AppDelegate code
- Clean Build Folder and rebuild: **Product** โ **Clean Build Folder**
- Search codebase for hardcoded `localhost:8081` references
### Assets Not Loading (Missing Images/Fonts)
**Possible Causes**:
- Assets folder not added to Xcode
- Assets not included in target
- Incorrect asset paths in code
**Solutions**:
- Verify assets folder was added with "Copy items if needed"
- Check Target Membership in File Inspector
- Confirm code uses correct relative paths for assets
### Native Modules Failing
**Possible Causes**:
- CocoaPods not installed
- Static linking issues
- New architecture compatibility (RN 0.68+)
**Solutions**:
- Run `cd ios && pod install && cd ..`
- Check module documentation for static build requirements
- Verify module compatibility with React Native architecture
### Build Errors in Xcode
**Possible Causes**:
- Hermes configuration issues
- Missing bundle file
- Duplicate symbols
**Solutions**:
- Verify `hermes_enabled` in `ios/Podfile`
- Confirm `ios/main.jsbundle` exists
- Delete DerivedData: `rm -rf ~/Library/Developer/Xcode/DerivedData`
## Hermes Engine Considerations
If your project uses Hermes (default in React Native 0.70+):
- โ
Bundling process remains the same
- โ
Generates more efficient bundles with better performance
- โ
Smaller file sizes
- โ
Check `ios/Podfile` for `hermes_enabled = true`
- โ ๏ธ Ensure all dependencies are Hermes-compatible
## Build Scheme Comparison
| Feature | Release | Debug |
|---------|---------|-------|
| **Bundle Size** | Smaller | Larger |
| **Performance** | Optimized | Unoptimized |
| **Debug Symbols** | No | Yes |
| **Recommended For** | Pie Testing | Issue Diagnosis |
**Recommendation**: Use **Release** builds for Pie testing as they represent production-like behavior.
## Additional Resources
- [React Native Publishing Guide](https://reactnative.dev/docs/publishing-to-app-store)
- React Native version-specific release notes
## Next Steps
1. Upload your build to [Pie's upload portal](https://app.pie.inc/upload)
2. Review [Post-Upload Requirements](/docs/integrations/mobile-builds/post-upload-requirements)
3. Pie's AI agents will begin testing within 30 minutes
## Need Help?
Contact Pie Labs support with:
- React Native version
- Build errors or console logs
- Screenshots of relevant issues
- Description of what you've tried
---
## Android Apps
## Overview
This guide covers building Android applications (native or React Native) for Pie testing. You'll create an APK file that can be installed on Android emulators.
## What You'll Need
- **Android Studio**: Latest version with Android SDK
- **Android Virtual Device (AVD)**: Configured emulator
- **Your Android Project**: Native or React Native Android app
- **Java Development Kit (JDK)**: Properly configured (usually bundled with Android Studio)
## Initial Setup
### Create an Android Virtual Device
If you don't already have an AVD configured:
1. Open **Android Studio**
2. Navigate to **Tools** โ **Device Manager**
3. Click **"Create Device"**
4. Select **Phone** category โ Choose latest **Pixel** model
5. Select the **latest Android OS version** available
6. Click **"Finish"** to create the AVD
### Configure Project for Testing
Before building, ensure your app is configured correctly:
1. Open `build.gradle` (app level)
2. Verify all API endpoints point to **staging/test environments**
3. Confirm API keys and configurations are for **test environment only**
4. Remove any production-specific security measures that would block testing
## Step-by-Step Build Instructions
### 1. Open Your Project
1. Launch **Android Studio**
2. Open your Android app project
3. Wait for **Gradle sync** to complete
- You'll see a progress indicator in the bottom status bar
- Resolve any sync errors before proceeding
### 2. Generate the APK
1. Navigate to **Build** menu โ **Build Bundle(s) / APK(s)** โ **Build APK(s)**
2. Android Studio will compile your app
- This may take several minutes for the first build
- Watch the Build Output panel for progress
3. Once complete, a notification will appear with a link to the APK location
### 3. Locate Your APK
**Option A**: Click **"locate"** in the build completion notification
**Option B**: Navigate manually to the default path:
```
app/build/outputs/apk/debug/app-debug.apk
```
**Note**: The exact path may vary based on:
- Build configuration (debug/release)
- Product flavors
- Custom output directories
### 4. Verify Your Build
Before uploading, test the APK on your emulator:
1. **Start your Android Virtual Device** from Device Manager
2. **Install the APK**: Drag and drop the APK file onto the emulator window
3. **Launch the app** from the app drawer
4. **Verify test environment**: Confirm it connects to your staging/test servers
5. **Test basic functionality**: Navigate through key features to ensure stability
### 5. Upload to Pie
1. Your APK is ready for direct upload (**no compression needed**)
2. Go to [Pie's upload portal](https://app.pie.inc/upload)
3. Upload the `app-debug.apk` file
Android APK files can be uploaded directly without zipping!
## Build Configuration Best Practices
### Use Debug or Staging Variants
- **Debug builds** are ideal for testing (include helpful diagnostics)
- **Avoid release builds** unless specifically needed (they may have obfuscation)
- If using build flavors, use staging or QA flavors, not production
### Environment Configuration
Verify your `build.gradle` is properly configured:
```gradle
android {
buildTypes {
debug {
applicationIdSuffix ".debug"
debuggable true
// Test environment configuration
}
}
}
```
### Permissions
Ensure `AndroidManifest.xml` includes all necessary permissions:
```xml
```
### Security Features
For testing purposes:
- **Disable anti-debugging features** that might interfere with testing
- **Remove certificate pinning** for test environments
- **Disable ProGuard/R8 obfuscation** in debug builds
- **Allow cleartext traffic** for test endpoints (if needed)
Example network security configuration for testing:
```xml
your-test-server.com
```
## React Native Android Considerations
If you're building a React Native Android app:
### JavaScript Bundle
React Native automatically bundles JavaScript code in debug builds. No additional bundling steps are required for debug APKs.
### Metro Bundler
The APK will be self-contained and won't require Metro bundler to run. If you want to verify this:
1. Close all terminal windows running Metro
2. Install and launch the APK on the emulator
3. App should run without connecting to Metro
### Assets
All JavaScript assets are automatically included in the APK during the build process.
## Troubleshooting
### Gradle Sync Failed
**Problem**: Project won't sync with Gradle.
**Solutions**:
- Check your internet connection (Gradle may need to download dependencies)
- Update Android Studio to the latest version
- Navigate to **File** โ **Invalidate Caches** โ Restart
- Check `build.gradle` for syntax errors
- Verify JDK path is correctly configured
### Build Failed with Errors
**Problem**: Build process fails with compilation errors.
**Solutions**:
- Read the error message in the Build Output panel
- Check for missing dependencies in `build.gradle`
- Verify all imported libraries are compatible with your target SDK
- Clean and rebuild: **Build** โ **Clean Project**, then **Build** โ **Rebuild Project**
### APK Not Found
**Problem**: Cannot locate the generated APK.
**Solutions**:
- Use the "locate" link in the build notification
- Search for `*.apk` files in your project directory
- Check if you have custom output directories defined in `build.gradle`
- Verify the build actually completed successfully (check Build Output)
### App Crashes on Launch
**Problem**: APK installs but crashes immediately.
**Solutions**:
- Check Logcat in Android Studio for crash logs
- Verify all required permissions are in `AndroidManifest.xml`
- Confirm minimum SDK version is compatible
- Check for missing native dependencies
- Ensure proper ProGuard rules if using code obfuscation
### Network Connection Issues
**Problem**: App cannot connect to backend services.
**Solutions**:
- Verify `INTERNET` permission in `AndroidManifest.xml`
- Check that API endpoints are accessible from external networks
- Confirm network security configuration allows connections to test servers
- Test endpoint connectivity from a web browser
### Wrong Environment
**Problem**: App connects to production instead of test environment.
**Solutions**:
- Double-check `build.gradle` configuration
- Verify build variant is debug or staging (not release)
- Check for hardcoded production URLs in code
- Confirm environment variables are properly set
## Build Variants and Flavors
If your project uses product flavors:
```gradle
android {
flavorDimensions "environment"
productFlavors {
staging {
dimension "environment"
applicationIdSuffix ".staging"
// Use this variant for Pie testing
}
production {
dimension "environment"
// Don't use for testing
}
}
}
```
To build a specific flavor:
1. Open **Build Variants** panel (usually on the left side)
2. Select the desired variant (e.g., `stagingDebug`)
3. Build the APK as described above
## API Level Compatibility
Ensure your app is compatible with Pie's testing infrastructure:
- **Minimum SDK**: API 21 (Android 5.0) or higher recommended
- **Target SDK**: Use recent versions for best results
- **Test on**: Latest available Android versions in AVD
## Next Steps
After preparing your Android build:
1. Upload it to [Pie's upload portal](https://app.pie.inc/upload)
2. Review the [Post-Upload Requirements](/docs/integrations/mobile-builds/post-upload-requirements)
3. Pie's AI agents will begin testing within 30 minutes
## Need Help?
Contact Pie Labs support with:
- Android Studio version
- Gradle version
- Build errors or Logcat output
- Screenshot of any error dialogs
- Description of the issue you're experiencing
---
## Post-Upload Requirements
## Overview
After uploading your build to Pie, you'll need to provide additional information to help Pie's AI agents test your application effectively. This page outlines what information to send and why it's important.
## How to Submit Information
Send the required information via email to the Pie Labs team. You'll receive upload confirmation with contact details when you upload your build.
## Required Information
### 1. Authentication Details
Provide test credentials so Pie's AI agents can access your application.
#### Test Account Credentials
- **Username/Email**: Test account login
- **Password**: Account password
- **Additional Factors**: 2FA codes, security questions, or PIN codes (if applicable)
**Important**: Only provide credentials for dedicated test accounts, never production accounts or accounts with access to real user data.
#### Special Login Flows
Document any non-standard authentication requirements:
- Social login (Google, Facebook, Apple) test accounts
- SSO or enterprise login procedures
- Biometric authentication alternatives
- Phone verification bypass codes
- Magic links or email verification
**Example**:
> "Use test account: testuser@example.com / TestPass123. The app will send a verification code to this email. Use code 123456 to bypass email verification in test environment."
### 2. Testing Boundaries
Help Pie's AI understand what to test and what to avoid.
#### Features to Focus On
List specific features, workflows, or screens that are critical:
- Key user journeys (e.g., "complete a purchase", "create a new post")
- Recently updated features that need validation
- Complex workflows requiring thorough testing
- High-priority user flows
**Example**:
> "Focus on: checkout flow, user profile creation, search functionality, and push notification settings."
#### Areas to Avoid
Specify features or actions the AI should not test:
- Work-in-progress features
- Known broken functionality
- Admin-only sections
- Features requiring special permissions
- Actions that might trigger alerts or notifications
**Example**:
> "Avoid: admin panel (requires special access), payment processing with real cards, social sharing features (under development)."
#### Critical User Journeys
Describe end-to-end workflows that must be tested:
**Example**:
> "Critical journey: New user signs up โ completes profile โ browses products โ adds item to cart โ proceeds to checkout โ completes order. This flow must work seamlessly."
### 3. Custom Test Files (If Applicable)
Some applications require specific files for testing.
#### When to Provide Files
Upload custom files if your app needs:
- **Documents**: PDFs, Word docs, spreadsheets that users would upload
- **Images**: Photos, avatars, or other images for testing uploads
- **Media**: Videos or audio files
- **Data Files**: CSV, JSON, or XML files for import features
- **Test Data**: Sample datasets or configuration files
#### How to Upload Files
- Mention required files in your email to Pie Labs
- Pie team will provide a secure upload link
- Include descriptions of how each file should be used
**Example**:
> "App requires test images for profile photos (JPG, max 5MB) and PDF documents for the document scanner feature. Will upload via provided link."
## Optional Guidelines
While not required, this information helps Pie's AI test more effectively.
### Expected Behavior
Describe how critical features should work:
**Example**:
> "When user adds item to cart, a badge with item count should appear on cart icon. Checkout flow should take 3-4 steps: cart review โ shipping info โ payment โ confirmation."
### Known Issues
List any known bugs or limitations:
**Example**:
> "Known issue: Dark mode toggle sometimes requires app restart. Search results may be slower during peak hours. These are expected and don't need reporting."
### Specific Test Scenarios
Suggest particular scenarios to test:
**Example**:
> "Test scenarios: 1) User with empty cart tries to checkout 2) User uploads very large image 3) User rapidly switches between tabs 4) User loses network during form submission."
### Performance Benchmarks
Provide performance expectations if relevant:
**Example**:
> "Home screen should load within 2 seconds. Search results should appear within 3 seconds. Video playback should start within 1 second."
### Edge Cases
Highlight unusual situations to test:
**Example**:
> "Edge cases: User with 100+ items in favorites, extremely long usernames (20+ chars), multiple rapid purchases, offline mode usage."
## Information Template
Use this template to structure your submission:
```
Subject: Test Configuration for [Your App Name]
AUTHENTICATION:
- Username: [test account]
- Password: [password]
- Special Instructions: [any special login steps]
TESTING FOCUS:
- Key Features: [list critical features]
- Avoid: [list areas to skip]
- Critical Journeys: [describe main workflows]
CUSTOM FILES NEEDED:
- [Describe any required test files]
OPTIONAL CONTEXT:
- Expected Behavior: [key behaviors]
- Known Issues: [documented bugs]
- Test Scenarios: [specific scenarios]
- Performance Notes: [benchmarks if applicable]
CONTACT:
- Name: [your name]
- Email: [your email]
- Best time to reach: [timezone/hours]
```
## Best Practices
### Security
- โ
**Use dedicated test accounts** created specifically for automated testing
- โ
**Ensure test accounts have no access** to production data or real user information
- โ
**Use test environment credentials** only, never production
- โ **Never share** credentials with access to real user data
- โ **Never provide** production API keys or secrets
### Clarity
- โ
**Be specific**: "Test the checkout flow" is better than "test payments"
- โ
**Use examples**: Show what you mean rather than just describing
- โ
**List steps**: Break down complex workflows into numbered steps
- โ
**Highlight priorities**: Indicate what's most important to test
### Communication
- โ
**Respond promptly** to any follow-up questions from Pie Labs
- โ
**Update information** if test accounts or environments change
- โ
**Notify about changes** when you upload new builds with different requirements
- โ
**Provide context** about your app's purpose and target users
## Account Management
### Test Account Setup
Create test accounts that:
- **Have appropriate permissions** for the features you want tested
- **Won't trigger rate limits** or security alerts
- **Represent different user roles** if your app has role-based access
- **Have realistic test data** (profile info, settings, history)
### Multiple User Roles
If your app has different user types, provide accounts for each:
**Example**:
> "Two test accounts provided:
> 1) Regular user: user@test.com / UserPass123
> 2) Premium user: premium@test.com / PremPass123
> Premium account has access to advanced features and ad-free experience."
### Account Lifecycle
Inform Pie Labs about:
- Account expiration dates (if any)
- Password rotation schedules
- When test environment resets occur
- Maintenance windows that affect test accounts
## Environment Details
### Test Environment Information
Provide context about your test environment:
**Example**:
> "Test environment: https://staging.example.com
> - Database resets nightly at 2 AM UTC
> - Payment processing uses Stripe test mode
> - Email notifications disabled
> - Push notifications functional"
### Data Persistence
Clarify what happens to test data:
**Example**:
> "Test data persists for 7 days, then automatically purges. Feel free to create, modify, or delete any data. Environment resets every Sunday at midnight UTC."
## What Happens Next?
After submitting this information:
1. **Confirmation**: Pie Labs team confirms receipt (usually within 24 hours)
2. **Setup**: Team configures testing parameters based on your requirements
3. **Testing Begins**: AI agents start testing within 30 minutes of setup
4. **Issue Reports**: You'll receive reports of any issues discovered
5. **Ongoing Communication**: Team may reach out with questions or updates
## Need to Update Information?
If anything changes after your initial submission:
- **Email the Pie Labs team** with updates
- **Use the same subject line** as your original email (for threading)
- **Clearly indicate** what has changed
- **Provide new credentials** if test accounts change
## Example Submission
Here's a complete example:
```
Subject: Test Configuration for ShopEasy Mobile App
AUTHENTICATION:
- Username: test.user@shopeasy.com
- Password: TestShop2024!
- Note: App will show a verification code on login screen in test mode.
Any 6-digit code works for test accounts.
TESTING FOCUS:
- Key Features: Product browsing, search, cart management, checkout flow,
order history, wishlist, user profile editing
- Avoid: Admin dashboard (requires special access), refer-a-friend feature
(not yet functional), seller portal
- Critical Journey: Browse products โ Add 3 items to cart โ Apply promo code
"TEST10" for 10% off โ Complete checkout โ View order confirmation
CUSTOM FILES NEEDED:
- No custom files required. All product images and data pre-loaded in test environment.
OPTIONAL CONTEXT:
- Expected Behavior: Cart icon badge shows item count. Checkout requires
shipping address before payment. Order confirmation includes order number.
- Known Issues: Dark mode theme slightly inconsistent on product detail pages.
This is expected and already tracked.
- Test Scenarios: Try searching for "shoes", filtering by price, sorting results,
adding items to wishlist, updating cart quantities
- Performance: Home screen loads in under 2s, search results in under 1s
CONTACT:
- Name: Jane Doe
- Email: jane.doe@shopeasy.com
- Best time to reach: 9 AM - 5 PM EST, Monday-Friday
```
## Questions?
If you're unsure what information to provide or need clarification:
- **Email the Pie Labs team** with your questions
- **Reference this guide** and mention which section you need help with
- **Provide context** about your app and specific concerns
The team is here to help you get set up successfully!
---
## Appendix
Essential reference materials to help you get the most out of Pie. From supported technologies to security compliance, find everything you need to understand and implement autonomous QA testing.
## Reference Materials
## What's Included
- **๐ง Technology Stack**: Complete list of supported platforms and frameworks
- **๐ก๏ธ Security Standards**: SOC 2 compliance and GDPR readiness details
- **๐ Terminology**: Clear definitions of QA and testing concepts
- **๐ Technical Details**: Deep dive into Pie's internal architecture
- **๐ Best Practices**: Guidelines for optimal implementation
---
## Authentication
Pie supports three authentication methods. The API checks them in priority order: **Scheduler Secret > API Key > Firebase Bearer Token**. Most integrations use API Key auth.
## Quick Reference
| Method | Header | Use Case |
|--------|--------|----------|
| **API Key** | `X-API-Key: ` | Programmatic access, CI/CD, MCP integrations |
| **Firebase Bearer Token** | `Authorization: Bearer ` | Browser sessions, user-specific operations |
| **Scheduler Secret** | `X-API-Key: ` + `X-App-ID: ` | Internal scheduled jobs |
## Firebase Bearer Token Authentication
Used for browser-based sessions and user-specific operations like app setup.
### Headers
| Header | Required | Description |
|--------|----------|-------------|
| `Authorization` | Yes | `Bearer ` |
| `X-App-ID` | Yes* | Your app identifier. *Not required for `/v1/apps` and `/v1/setup-app` |
### Example
```bash
curl -X GET "https://api.pie.inc/v1/apps" \
-H "Authorization: Bearer YOUR_FIREBASE_ID_TOKEN"
```
```bash
curl -X GET "https://api.pie.inc/v1/test-cases" \
-H "Authorization: Bearer YOUR_FIREBASE_ID_TOKEN" \
-H "X-App-ID: YOUR_APP_ID"
```
### How It Works
- The Firebase ID token is verified using Firebase Admin SDK
- The user's UID is extracted from the token
- Access is validated against the `userAppPermissions` collection
- Admin/superadmin users (via Firebase custom claims) bypass app access checks
### Error Responses
| Status | Message | Cause |
|--------|---------|-------|
| 401 | `Authorization header is required` | No auth header provided |
| 401 | `Invalid authorization header format` | Header doesn't follow `Bearer ` format |
| 401 | `X-App-ID header is required` | Missing `X-App-ID` on endpoints that require it |
| 401 | `Invalid Firebase ID token` | Token is expired, malformed, or revoked |
| 401 | `User does not have access to this app` | User not in the app's permission list |
## Common Headers
These headers are used alongside authentication on specific endpoints:
| Header | Description | Used By |
|--------|-------------|---------|
| `Content-Type: application/json` | Required for POST/PUT/PATCH request bodies | All write endpoints |
| `Accept: text/csv` | Request CSV format response | GET endpoints (test cases, issues, results, runs) |
| `X-Export-Format: csv` | Alternative CSV export trigger | GET endpoints |
## MCP Server Authentication
The MCP server at `https://mcp.pie.inc` uses the same API Key authentication:
| Method | Format |
|--------|--------|
| Header | `X-API-Key: YOUR_API_KEY` |
| Query parameter | `?api_key=YOUR_API_KEY` |
For setup instructions per platform (Claude Desktop, Cursor, VS Code, Claude Code), see [Connect Pie to Your AI Workspace](/docs/mcp/).
---
## Error Codes
All Pie API error responses follow a standard format:
```json
{
"success": false,
"status": 401,
"message": "Human-readable error description",
"data": null
}
```
## MCP Server Errors
When using the MCP server (`https://mcp.pie.inc`), tool call errors are returned as MCP protocol error responses:
| Error | Cause | Resolution |
|-------|-------|------------|
| `Invalid API key` | MCP request sent without valid API key | Check your MCP config includes the correct `X-API-Key` header or `api_key` query parameter |
| `App not found` | API key not associated with any app | Verify the API key was created for the correct app |
| `Tool not found` | Called a tool name that doesn't exist | Check the [MCP Tool Reference](/docs/mcp/tool-reference/) for valid tool names |
---
## Supported Technologies
Pie is platform- and framework-agnostic. We test at the UI layer, just like a real user, so we're compatible with your entire tech stack.
## Platforms
- Web (including PWAs)
- Native iOS
- Native Android
## Frontend Frameworks
- React
- Angular
- Vue.js
- Svelte
- And more
## Mobile Frameworks
- Native iOS (Swift/Objective-C)
- Native Android (Kotlin/Java)
- React Native
- Flutter
- And more
## Backend Technologies
Since we test the UI, we work with any backend, including:
- Node.js
- Python
- Java
- Go
- Ruby
- Etc.
## Pie's Internal Tech Stack
For transparency, our own platform is built with:
- **Go** (Backend)
- **TypeScript/Remix** (Frontend)
- **Firebase** (Database)
- **Python** (for specific cloud functions)
---
## Security & Compliance
We are built with enterprise-grade security to ensure your applications and data are handled safely and responsibly. The platform is designed to be **SOC 2 compliant** and **GDPR ready**, and includes **Role-Based Access Control (RBAC)** to manage team permissions.
## Data Encryption
- All data, including application builds and test credentials, is encrypted:
- **In transit** using TLS 1.2+
- **At rest** using AES-256
## Secure Credential Handling
- Test credentials and API keys are stored in a **secure, encrypted vault**
- Accessed only by AI agents in **isolated, ephemeral environments** during test execution
## Isolated Test Environments
- Each test run executes in a **completely isolated, sandboxed environment**
- Environments are **created on-demand** and **destroyed immediately** after the run
- Ensures no cross-contamination of data between tests or customers
## SOC 2 Compliance
- Designed to meet **SOC 2 compliance standards**
- Infrastructure, software, people, and procedures are **regularly audited** to ensure:
- Security
- Availability
- Confidentiality
## GDPR Readiness
- Fully **GDPR ready**
- Ensures user data from the **European Union** is handled according to privacy and user rights regulations
## Role-Based Access Control (RBAC)
- Robust **RBAC** system to manage user permissions
- Ensures team members access only the features and data relevant to their roles
---
## Glossary
To help you get the most out of Pie and understand the world of software quality, hereโs a list of key terms.
- **AI Agent**: The autonomous software that explores your app, executes tests, and analyzes results.
- **Assertion**: A check within a test case to verify that a specific condition is true.
*Example*: After clicking "Login," an assertion would be *"Verify the user is on the dashboard screen."*
- **Bug (or Defect)**: An error or flaw in the application that causes it to produce an incorrect or unexpected result.
- **CI/CD (Continuous Integration/Continuous Deployment)**: The practice of frequently merging code changes into a central repository (CI) and then automatically releasing them (CD). Pie integrates into this pipeline to automate testing for every change.
- **Contextual Model**: The AI's internal, structured representation of an application. It serves as a comprehensive map of the app's screens, UI elements, features, and their relationships, enabling contextual understanding and self-healing tests.
- **Discovery**: The initial automated crawl where an AI Agent builds the Contextual Model.
- **End-to-End (E2E) Testing**: A testing method that validates an application's workflow from beginning to end, mimicking a real user journey. Pie specializes in autonomous E2E testing.
- **Finding**: A single, raw instance of a potential bug detected by an AI Agent.
- **Functional Testing**: Testing that verifies the application's features work according to their specified requirements. It answers the question: *"Does this feature do what it's supposed to do?"*
- **Issue**: A curated, deduplicated report representing a unique, underlying bug, created by grouping related findings.
- **Readiness Score**: An overall quality metric (0-100%) summarizing a build's health for a go/no-go decision.
- **Regression Testing**: The process of re-running tests to ensure that recently added code changes have not broken any existing functionality. Pie automates this entirely.
- **Self-Healing Test**: A test that automatically adapts to UI changes, eliminating test maintenance.
- **Severity vs. Priority**:
- **Severity**: Measures the technical impact of a bug on the application (e.g., a *"Critical"* severity bug might be a crash).
- **Priority**: Measures the urgency of fixing the bug from a business perspective (e.g., a typo on the homepage might be *"Low"* severity but *"High"* priority).
- **Smoke Testing**: A preliminary set of tests run on a new build to ensure its most critical functionalities are working. A failed smoke test usually means the build is rejected immediately.
- **Usability Testing**: Testing focused on how easy and intuitive the application is for a real user. Pie's AI can identify usability issues like confusing navigation or poorly designed layouts.
---
## Frequently Asked Questions (FAQs)
Here are answers to some common questions that will help new and experienced users leverage the platform better.
## How do I get started with Pie?
Getting started with Pie is straightforward. Schedule a personalized demo with our team to see how Pie's autonomous testing platform works with your specific applications and QA challenges. During the demo, we'll demonstrate our AI-native testing capabilities and show you exactly how Pie can transform your testing workflow from manual to autonomous.
## What do I need for my very first test run?
- For a **web app**: Just the URL
- For a **mobile app**: A zipped `.app` (iOS Simulator) or `.apk` (Android) file
- If login is required: Have a set of test credentials ready
## How long does it take to see results?
For most applications, your first comprehensive report - including a **Readiness Score** and a list of issues - will be ready within **15-30 minutes** of starting a run.
## Do I have to write test scripts?
No. Pie's AI agents autonomously generate hundreds of test cases by exploring your app just like a real user would.
For specific or complex scenarios, use the **Pie Test Assistant** to create a new test using a **plain English prompt** - no code required.
## How do I test a specific user flow that the AI might miss?
Use the **Pie Test Assistant**.
Write a prompt in plain English (e.g., *"Log in, go to settings, and change the notification preferences"*).
The AI will generate and validate the test case for you.
## How do I get the bug reports into my team's workflow?
Use our **one-click Jira integration** to push any issue directly into your Jira project.
This creates a **pre-filled ticket** with all necessary details.
You can also integrate Pie into your **CI/CD pipeline** to trigger runs automatically.
## What happens if I dismiss an issue by mistake?
Dismissed issues aren't gone forever.
You can find them in the **"Resolved"** tab within the issues section.
Click **"Restore"** to move the issue back to your active list.
## What's the difference between a "Finding" and an "Issue"?
- **Finding**: A single, raw instance of a potential bug detected in a specific test
- **Issue**: A curated, deduplicated report created by grouping related findings into one actionable root cause
This means you get **one report per unique bug**, not hundreds of duplicate alerts.
## How do "Self-Healing Tests" actually work?
Instead of relying on brittle code selectors (like a specific button ID), Pie's tests are based on the AIโs **contextual understanding** of your app, stored in an **application model**.
If a buttonโs text or ID changes, the AI still identifies it by **position**, **function**, and **context**.
This allows tests to **heal themselves** and run without manual updates.
## How is the Readiness Score calculated?
The score starts at **100%** for every test run.
It is reduced by a set amount for each **approved critical issue** found.
This makes the score a true reflection of **release-blocking bugs**, giving you a clear and reliable signal for **go/no-go** decisions.