Introduction

This repository hosts integration tests and associated CI infrastructure for the Zcash ecosystem. The following tests are provided:

• Functional tests in Python of zebrad, zainod, and zallet, using regtest mode and primarily their JSON-RPC interfaces.

The functional tests and CI workflows were originally part of the zcashd codebase, with the Python test framework (and some of the tests) inherited from Bitcoin Core.

User Documentation

This section contains user documentation specific to Zcash integration testing.

Running Tests

Prerequisites

Binaries

All tests require the zebrad binary; most tests require the zallet binary; some tests require the zainod binary.

By default, binaries must exist in the ./src/ folder under the repository root. Alternatively, you can set the binary paths with environment variables:

export ZEBRAD=/path/to/zebrad
export ZAINOD=/path/to/zainod
export ZALLET=/path/to/zallet

Python dependencies

The zmq, toml, and base58 Python libraries are required.

With uv (recommended)

uv sync

Without uv

On Ubuntu or Debian-based distributions:

sudo apt-get install python3-zmq python3-base58 python3-toml

On macOS or other platforms:

python3 -m venv venv
. venv/bin/activate
pip3 install pyzmq base58 toml

Running the full test suite

With uv:

uv run ./qa/zcash/full_test_suite.py

Without uv:

./qa/zcash/full_test_suite.py

Running individual tests

Run a single test:

./qa/pull-tester/rpc-tests.py <testname>

Run multiple specific tests:

./qa/pull-tester/rpc-tests.py <testname1> <testname2> <testname3>

Run all regression tests:

./qa/pull-tester/rpc-tests.py

Parallel execution

By default, tests run in parallel with 4 jobs. To change the number of jobs:

./qa/pull-tester/rpc-tests.py --jobs=n

Test runner options

-h, --help            show this help message and exit
--nocleanup           Leave test datadirs on exit or error
--noshutdown          Don't stop nodes after the test execution
--srcdir=SRCDIR       Source directory containing binaries (default: ../../src)
--tmpdir=TMPDIR       Root directory for datadirs
--tracerpc            Print out all RPC calls as they are made
--coveragedir=COVERAGEDIR
                      Write tested RPC commands into this directory

Debugging

Set PYTHON_DEBUG=1 for debug output:

PYTHON_DEBUG=1 qa/pull-tester/rpc-tests.py wallet

For real-time output, run a test directly with python3:

python3 qa/rpc-tests/wallet.py

Cache management

A 200-block regtest blockchain and wallets for four nodes are created the first time a regression test is run and stored in the cache/ directory. Each node has the miner subsidy from 25 mature blocks (25*10=250 ZEC) in its wallet.

After the first run, the cached blockchain and wallets are copied into a temporary directory and used as the initial test state.

If you get into a bad state, you can recover with:

rm -rf cache
killall zebrad
killall zainod
killall zallet

Writing Tests

Adding a new test

1. Add a new file NEW_TEST.py to the qa/rpc-tests/ folder.
2. Update qa/pull-tester/rpc-tests.py, adding a new entry 'NEW_TEST.py', to the NEW_SCRIPTS array (either at the end of the array, or in the appropriate position based on how long the test takes to run).
3. Write your test (either from scratch, or by modifying an existing test as appropriate).
4. Open a pull request with your changes.

Test framework

The test framework lives in qa/rpc-tests/test_framework/. Key modules:

P2P test design

Mininode

mininode.py contains all the definitions for objects that pass over the network (CBlock, CTransaction, etc., along with the network-level wrappers msg_block, msg_tx, etc.).

P2P tests have two threads: one handles all network communication with the nodes being tested (using Python’s asyncore package); the other implements the test logic.

NodeConn is the class used to connect to a node. Implement a callback class that derives from NodeConnCB and pass it to the NodeConn object to receive callbacks when events of interest arrive. Be sure to call self.create_callback_map() in your derived class’s __init__ function.

Call NetworkThread.start() after all NodeConn objects are created to start the networking thread, then continue with the test logic in your existing thread.

RPC calls are also available in P2P tests.

Comptool

Comptool is a testing framework for writing tests that compare the block/tx acceptance behavior of a node against one or more other node instances, or against known outcomes, or both.

Set the num_nodes variable (defined in ComparisonTestFramework) to start up one or more nodes. Implement a generator function called get_tests() which yields TestInstances. Each TestInstance consists of:

• A list of [object, outcome, hash] entries, where:
  • object is a CBlock, CTransaction, or CBlockHeader.
  • outcome is True, False, or None.
  • hash (optional) is the block hash of the expected tip.
• sync_every_block: if True, each block is tested in sequence and synced; if False, all blocks are inv’d together and only the last block is tested.
• sync_every_transaction: analogous to sync_every_block for transactions.

For examples, see invalidblockrequest.py and p2p-fullblocktest.py.

Platform Support

Support for testing different platforms (build “targets” and operating systems) is governed by the Platform Policy.

Tested platforms can be thought of as “guaranteed to work”. Automated testing ensures that each tested platform builds and passes tests after each change to the upstream binaries.

The following table contains the set of testing platform configurations that binary providers can make use of as part of their own workflows.

“End of Support” dates are the latest currently-known date after which the platform will be removed. These dates are subject to change.

Developer Documentation

This section contains documentation aimed at contributors to the Zcash integration tests.

Regtest Mode

Regtest (regression test) mode runs a local Zcash network where blocks are generated on demand. This gives tests full control over block timing and chain state.

The draft ZIP for regtest mode provides the formal specification.

Key constants

Network upgrade activation

Zebra activates Overwinter through Canopy at block height 1 by default. Tests that need NU5 or later upgrades require explicit -nuparams configuration.

Interacting with nodes during a test

To inspect node state mid-test, you can pause execution (e.g. with a debugger breakpoint or time.sleep()) and issue RPC calls against the node’s RPC port. Each node’s RPC port is logged at startup.

Platform Policy

General

A testing platform configuration (or “tested platform”) is a combination of a Rust target and an operating environment (such as a specific operating system version).

The continuous integration checks in this repository ensure that tested platforms will always build and pass tests. These platforms place work on Zcash developers as a whole, to avoid breaking the platform. The broader Zcash community may also feel more inclined to support tested platforms in their downstream uses of zebrad, zainod, or zallet (though they are not obligated to do so). Thus, tested platforms require commensurate and ongoing efforts from the maintainers of the platform, to demonstrate value and to minimize any disruptions to ongoing Zcash development.

This policy defines the requirements for accepting a proposed platform into the set of tested platforms.

While these criteria attempt to document the policy, that policy still involves human judgment. Platforms must fulfill the spirit of the requirements as well, as determined by the judgment of the approving reviewers. Reviewers and team members evaluating platforms and platform-specific patches should always use their own best judgment regarding the quality of work, and the suitability of a platform for the Zcash project. Neither this policy nor any decisions made regarding platforms shall create any binding agreement or estoppel by any party.

For a list of all supported platforms, see Platform Support.

The availability of a platform in releases of Zcash ecosystem software is not a hard stability guarantee about the future availability of that platform. Tested platforms are a commitment to the support of a platform, and we will take that commitment and potential disruptions into account when evaluating the potential removal of a platform that has been part of a stable release of Zcash ecosytem software. The addition or removal of a platform will not generally affect existing stable releases, only current development and future releases.

In this policy, the words “MUST” and “MUST NOT” specify absolute requirements that a platform must meet to qualify. The words “SHOULD” and “SHOULD NOT” specify requirements that apply in almost all cases, but for which the approving teams may grant an exception for good reason. The word “MAY” indicates something entirely optional, and does not indicate guidance or recommendations. This language is based on IETF RFC 2119.

Tested platform policy

The Zcash developers guarantee that a tested platform builds and passes all tests, and will reject patches that fail to build or pass the test suite on a platform. Thus, we place requirements that ensure the platform will not block forward progress of the Zcash project.

A proposed new platform MUST be reviewed and approved by the Zcash core team based on these requirements.

In addition, the Zcash infrastructure team MUST approve the integration of the platform into Continuous Integration (CI), and the CI-related requirements. This review and approval MAY take place in a PR adding the platform to CI, or simply by an infrastructure team member reporting the outcome of a team discussion.

• The platform MUST provide documentation for the Zcash community explaining how to build for the platform, using cross-compilation if possible. If the platform supports running binaries, or running tests (even if they do not pass), the documentation MUST explain how to run such binaries or tests for the platform, using emulation if possible or dedicated hardware if necessary.
• The platform MUST have a designated team of developers (the “platform maintainers”) supporting it, without the need for a paid support contract.
• The platform MUST NOT place undue burden on Zcash developers not specifically concerned with that platform. Zcash developers are expected to not gratuitously break a tested platform, but are not expected to become experts in every tested platform.
• The platform MUST have substantial, widespread interest within the Zcash community, and MUST serve the ongoing needs of multiple production users of Zcash across multiple organizations or projects. These requirements are subjective. A tested platform MAY be removed if it becomes obsolete or no longer meets this requirement.
• The platform MUST build and pass tests reliably in CI, for all components that this repo’s CI marks as mandatory.
• Building the platform and running the test suite for the platform MUST NOT take substantially longer than other platforms, and SHOULD NOT substantially raise the maintenance burden of the CI infrastructure.
• The platform MUST NOT have a hard requirement for signed, verified, or otherwise “approved” binaries. Developers MUST be able to build, run, and test binaries for the platform on systems they control, or provide such binaries for others to run. (Doing so MAY require enabling some appropriate “developer mode” on such systems, but MUST NOT require the payment of any additional fee or other consideration, or agreement to any onerous legal agreements.)

A platform MAY be removed if it no longer meets these requirements.

CI Infrastructure

This repository’s CI pipeline builds and tests the Z3 stack components (zebrad, zainod, and zallet) together as an integrated system.

What CI does

On every pull request and push to main, CI:

1. Builds the zebrad, zainod, and zallet binaries from their latest released versions (or from a specific commit when triggered by a cross-repo dispatch).
2. Runs the RPC test suite against the built binaries on required platforms.
3. Reports results back to the triggering repository when invoked via cross-repo dispatch.

Platform matrix

Required platforms must pass for CI to succeed. RPC tests are only run on required platforms to manage CI costs. When CI is triggered via cross-repo dispatch, callers may specify which platforms to run; all explicitly requested platforms are treated as required.

Cross-repository integration

External repositories can trigger integration tests from their PRs and receive results back as status checks. See Cross-Repository CI for the mechanism and setup instructions.

Cross-Repository CI Integration

This repository supports triggering integration tests from external repositories via GitHub’s repository_dispatch mechanism. When CI runs on a PR in an integrated repository, that repository dispatches an event here, which runs the full integration test suite using the PR’s commit. Results are reported back as status checks on the originating PR.

Integration testing is currently set up for the zallet, zebrad, and [zaino] repositories.

How it works

The integration has two sides: the requesting repository (e.g. ZcashFoundation/zebra) and this integration-tests repository (zcash/integration-tests). Two GitHub Apps are used, one for each direction of communication:

• Dispatch app: This is an app owned by the requesting org that is manually installed on zcash/integration-tests by repository administrators and is used by the requesting repository’s CI to trigger workflows here. Each requesting organization needs to create its own dispatch app.
• Status-reporting app (z3-integration-status-reporting, owned by zcash, public): Installed on the requesting repository. Used by integration-tests CI to write status checks back to the requesting repository’s PR.

This two-app model follows the principle of least privilege: each app only has the permissions and credentials for the direction it serves, limiting the blast radius if either app’s credentials are compromised.

Requesting repository setup

The requesting repository’s CI workflow includes a job that dispatches to this repository:

trigger-integration:
  name: Trigger integration tests
  runs-on: ubuntu-latest
  steps:
    - name: Generate app token
      id: app-token
      uses: actions/create-github-app-token@v2
      with:
        app-id: ${{ secrets.DISPATCH_APP_ID }}
        private-key: ${{ secrets.DISPATCH_APP_PRIVATE_KEY }}
        owner: zcash
        repositories: integration-tests
    - name: Trigger integration tests
      env:
        GH_TOKEN: ${{ steps.app-token.outputs.token }}
      run: >
        gh api repos/zcash/integration-tests/dispatches
        --field event_type="<project>-interop-request"
        --field client_payload[sha]="$GITHUB_SHA"

This uses the dispatch app’s credentials to generate a token scoped to zcash/integration-tests, then sends a repository_dispatch event. The owner field must be zcash because the token needs write access to the zcash/integration-tests repository.

The client_payload must include:

• sha: The commit SHA to build and test from the requesting repository.

It may also include these optional fields:

• platforms: A JSON array of platform names (e.g. ["ubuntu-22.04", "mingw32"]) to run only a subset of platforms. Requested platforms are treated as required (must pass). The valid list of platform names is maintained here. Unrecognized platform names are reported as error statuses on the requesting PR. When omitted, all platforms run.
• test_sha: A commit SHA or ref in the zcash/integration-tests repository itself. When provided, the test suite is checked out at this ref instead of main. This is useful for testing integration-tests changes alongside project changes.

Integration-tests repository setup

In the integration-tests repository CI, three things are configured:

1. ci.yml trigger: The workflow’s on.repository_dispatch.types array includes the event type (zebra-interop-request, zallet-interop-request, and zaino-interop-request are currently supported).

2. Build job: A build job checks out the requesting repository at the dispatched commit SHA:

   - name: Use specified commit
     if: github.event.action == '<project>-interop-request'
     env:
       SHA: ${{ github.event.client_payload.sha }}
     run: echo "PROJECT_REF=${SHA}" >> $GITHUB_ENV
   
   - name: Use current main
     if: github.event.action != '<project>-interop-request'
     run: echo "PROJECT_REF=refs/heads/main" >> $GITHUB_ENV
   

3. Status reporting: Four composite actions in .github/actions/ handle communication back to the requesting repository:

   • interop-repo-ids maps the event type (e.g. zebra-interop-request) to the requesting repository’s owner and name. This mapping is maintained as a case expression so that only known event types resolve to valid repositories.
   • start-interop generates a token from the z3-integration-status-reporting app scoped to the requesting repository and creates a pending status check on the dispatched commit.
   • finish-interop (run with if: always()) updates that status to the job’s final result (success, failure, or error).
   • notify-interop-error reports error statuses for any requested platforms that are not recognized by the CI matrix (see platforms in client_payload).

   Each job calls interop-repo-ids first, then passes its outputs to start-interop at the beginning and finish-interop at the end.

Security model

Who can trigger integration test workflows? Two independent gates control this:

1. App installation on zcash/integration-tests: The dispatch app must be installed on zcash/integration-tests, which requires approval from a zcash org admin. For zcash-internal repos, the org-private [z3-integration-dispatch] app is used, so no external organization can use it. For external repos, the requesting organization creates its own dispatch app, which a zcash admin must explicitly approve for installation.
2. Event type allowlist in ci.yml: The workflow only responds to event types explicitly listed in on.repository_dispatch.types. Even if an app could dispatch an event, it would be ignored unless its type is listed.

Credential separation: Each app’s private key is stored only where it is needed — the dispatch app’s key in the requesting repository, the z3-integration-status-reporting key (a single key pair, since it is one app) in zcash/integration-tests. If a dispatch app’s credentials are compromised, an attacker could trigger integration test runs but could not write arbitrary statuses. If the status-reporting credentials are compromised, an attacker could write status checks to repositories where the app is installed but could not trigger workflow runs.

App permissions:

Setting up integration for a new repository

To add cross-repository integration for a new project, follow these steps. The instructions below use Zebra (ZcashFoundation/zebra) as a running example.

1. Set up the dispatch app (requesting org side)

The requesting repository needs a dispatch app installed on zcash/integration-tests.

The requesting organization creates a GitHub App with:

• Repository permissions: Contents: Read and write

Then have a zcash org admin install it on zcash/integration-tests. Store the app’s ID and private key as repository secrets in the requesting repository (as DISPATCH_APP_ID and DISPATCH_APP_PRIVATE_KEY) so that they can be used for CI configuration as described in the next step.

2. Update the requesting repository’s CI

Add a trigger-integration job to the requesting repository’s CI workflow (see the example above). Use the event type <project>-interop-request appropriate to the requesting repository where <project> is one of zallet, zebra, or zaino.

3. Install z3-integration-status-reporting on the requesting repository

The z3-integration-status-reporting app must be installed on the requesting repository so that integration-tests can write status updates back to the PR or workflow run that triggered the test.

An admin of the requesting organization can install the app via https://github.com/apps/z3-integration-status-reporting/installations/new

During installation, select only the specific repository that needs integration (e.g. zebra).

4. Verify

Open a test PR in the requesting repository and confirm that:

• The trigger-integration job dispatches successfully.
• Integration tests run in this repository.
• Status checks appear on the requesting repository’s PR commit.
• Both success and failure states propagate correctly.