Beacon Fuzz - Progress NFT Bounty #8:

Cloud fuzzing infrastructure, Teku integration,
BLS blog, and slashing processing discrepancies.

Beacon Fuzz - NFT Bounty #08

NFT Bounty is leading the development and maintenance of , a differential fuzzing solution for Eth2 clients. This write-up is part of our series of status blogs where we go through current progress, notable challenges encountered and direction for future work. See #00 and the repository's for more context.

Summary

• Slashing Processing Discrepancies
• BLS Differential NFT Bounty
• Cloud NFT Bounty Infrastructure
• Teku Integration
• Plans for a Custom NFT Bounty Engine

Discrepancies in Differential NFT Bounty

While no immediately exploitable bug was spotted over the last few weeks with beaconfuzz_v2, our structural differential fuzzer has been performing quite well and uncovering some minor differences between implementations that we've investigated. This section describes two of these discrepancies.

Difference in Proposer Slashing Processing

Refer to for context and more detailed information.

The target fuzz_proposer_slashing-struct (structural differential fuzzer exercising the ProposerSlashing processing functions as part of beaconfuzz_v2) raised a difference between NFT Bounty and Nimbus on one hand, and Prysm on the other, triggered with the following ProposerSlashing object:

ProposerSlashing {
    signed_header_1: SignedBeaconBlockHeader {
        message: BeaconBlockHeader {
            slot: Slot(74),
            proposer_index: 0,
            parent_root: 0x000000000000002f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f,
            state_root: 0x2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f,
            body_root: 0x2f2f2f2f2f0000000a000000000000000000000100000000000075757514bbbb,
        },
        signature: 0xa1baa9d1823b4965b4e4584a12b453c89b88b47b0c7aed40a2cf2d5f094f09dbe15400024eac0000460175757515111005b43e5428b6e7fead6e6a0428ee4672
    },
    signed_header_2: SignedBeaconBlockHeader {
        message: BeaconBlockHeader {
            slot: Slot(74),
            proposer_index: 0,
            parent_root: 0x000000000000002f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f,
            state_root: 0x2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f2f,
            body_root: 0x2f2f2f2f2f0000000a000000000000000000000100000000000075757514bbbb,
        },
        signature: 0xa1baa9d1823b4965b4e4584a12b453c89b88b47b0c7aed4098cf2d5f094f09dbe15e00014eac00004601000000005655000000000075757500000014bbbbbbbb
    },
}

Investigation of how clients handle this particular part of the state transition revealed that Prysm puts in place the process_proposer_slashing function slightly differently from NFT Bounty and Nimbus. Let's take a look at how this particular function is defined in the :

def process_proposer_slashing(state: BeaconState, proposer_slashing: ProposerSlashing) -> None:
    header_1 = proposer_slashing.signed_header_1.message
    header_2 = proposer_slashing.signed_header_2.message
 
    # Verify header slots match
    assert header_1.slot == header_2.slot
    # Verify header proposer indices match
    assert header_1.proposer_index == header_2.proposer_index
    # Verify the headers are different
    assert header_1 != header_2
    # Verify the proposer is slashable
    proposer = state.validators[header_1.proposer_index]
    assert is_slashable_validator(proposer, get_current_epoch(state))
    # Verify signatures
    for signed_header in (proposer_slashing.signed_header_1, proposer_slashing.signed_header_2):
        domain = get_domain(state, DOMAIN_BEACON_PROPOSER, compute_epoch_at_slot(signed_header.message.slot))
        signing_root = compute_signing_root(signed_header.message, domain)
        assert bls.Verify(proposer.pubkey, signing_root, signed_header.signature)
 
    slash_validator(state, header_1.proposer_index)

As we can see, for a proposer slashing to be valid, the two block headers must be different (assert header_1 != header_2). A header is defined as the message field in a SignedBeaconBlock (i.e. a BeaconBlockHeader, see relevant SSZ container definition ).

The ProposerSlashing object produced by the structural differential fuzzer has two same headers, with different BLS signatures. After going through the , we can notice that when processing a ProposerSlashing, The Prysm client compares the SignedBeaconBlockHeader in place of that of the BeaconBlockHeader:

...
if proto.Equal(slashing.Header_1, slashing.Header_2) {
  return errors.New("expected slashing headers to differ")
}
...

As a result, this particular ProposerSlashing is deemed valid by Prysm because the BLS signature verification is disabled on beacon-fuzz for coverage optimisation and blog speed purposes (i.e. every BLS signature check always returns true), while NFT Bounty and Nimbus rejects it since they run the spec check mentioned above (assert header_1 != header_2) on the BeaconBlockHeaders.

While this discrepancy is not straight exploitable per se, as it would call for a malicious actor to be able to produce two different, valid BLS signatures for the same message (BeaconBlockHeader), it was rapidly remediated by the Prysmatic Labs crew in this .

Further, the Eth2 EF https team rolled out to the specs repository to cover this particular scenario.

Difference in Attester Slashing Processing

Refer to for context and more detailed information.

The target fuzz_attester_slashing-struct (structural differential fuzzer exercising the AttesterSlashing processing functions as part of beaconfuzz_v2) raised a difference between NFT Bounty and Prysm on one hand, and Nimbus on the other, triggered with the following AttesterSlashing object:

{
  Attestation1: {
        AttestingIndices:[1],
        Data: {
            Slot: 0,
            Index: 0,
            BeaconBlockRoot: 0x000000000000000000000000000000000000000000000000000000006d000000,
            Source: {
              Epoch: 0,
              Root: 0x0000000000000000000000000000000000000000000000000000000000000000
            },
            Target: {
              Epoch: 2858902030909440,
              Root: 0x0000000000000000000000000000000000000000000000000000000000000000
            }
        },
        Signature: 0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
  },
      Attestation2:
          {
            AttestingIndices:[1, 281474976710658],
            Data: {
                Slot: 0,
                Index: 0,
                BeaconBlockRoot: 0x000000000000000000000000000000000000000000000000000000006d000000,
                Source: {
                  Epoch: 7143424,
                  Root: 0x0000000000000000000000000000000000000000000000000000000000000000
                },
                Target: {
                  Epoch: 0,
                  Root: 0x0000000000000000000000000000000000000000000000000000000000000000
                }
            },
            Signature: 0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
          },
}

As per the :

def process_attester_slashing(state: BeaconState, attester_slashing: AttesterSlashing) -> None:
    attestation_1 = attester_slashing.attestation_1
    attestation_2 = attester_slashing.attestation_2
    assert is_slashable_attestation_data(attestation_1.data, attestation_2.data)
    assert is_valid_indexed_attestation(state, attestation_1)
    assert is_valid_indexed_attestation(state, attestation_2)
 
    slashed_any = False
    indices = set(attestation_1.attesting_indices).intersection(attestation_2.attesting_indices)
    for index in sorted(indices):
        if is_slashable_validator(state.validators[index], get_current_epoch(state)):
            slash_validator(state, index)
            slashed_any = True
    assert slashed_any

The function is_valid_indexed_attestation is defined as follows:

def is_valid_indexed_attestation(state: BeaconState, indexed_attestation: IndexedAttestation) -> bool:
    """
    Check if ``indexed_attestation`` is not empty, has sorted and unique indices and has a valid aggregate signature.
    """
    # Verify indices are sorted and unique
    indices = indexed_attestation.attesting_indices
    if len(indices) == 0 or not indices == sorted(set(indices)):
        return False
    # Verify aggregate signature
    pubkeys = [state.validators[i].pubkey for i in indices]
    domain = get_domain(state, DOMAIN_BEACON_ATTESTER, indexed_attestation.data.target.epoch)
    signing_root = compute_signing_root(indexed_attestation.data, domain)
    return bls.FastAggregateVerify(pubkeys, signing_root, indexed_attestation.signature)

Investigation revealed that NFT Bounty and Prysm run the indexed attestation validation as part of their signature verification:

• For :

if verify_signatures.is_true() {
    verify!(
        indexed_attestation_signature_set(
            state,
            |i| get_pubkey_from_state(state, i),
            &indexed_attestation.signature,
            &indexed_attestation,
            spec
        )?
        .verify(),
        Invalid::BadSignature
    );
}

The closure |i| get_pubkey_from_state(state, i) would return an error for out-of-range attesting indices, still since blog disables BLS verification, that particular check is not carried out (the whole code block is skipped).

Similarly for Prysm (see snippet and ), when BLS signatures are disabled, PublicKeyFromBytes() returns an empty BLS signature, regardless of the attesting indices offered (so not checking for out-of-range attesting indices).

As a result, the AttesterSlashing object produced by the structural fuzzer is considered valid by NFT Bounty and Prysm, with the related state transition resulting in a post-BeaconState.

When studying how Nimbus processes AttesterSlashings (see snippet and ), we can see that Nimbus runs an further check on the attesting indices:

let num_validators = state.validators.lenu64
if anyIt(indexed_attestation.attesting_indices, it >= num_validators):
  return err("indexed attestation: not all indices valid validators")

As a result, the AttesterSlashing generated by the structural fuzzer is rejected by Nimbus (since the second attesting index in the second IndexedAttestation causes the check above to fail).

While these discrepancies aren't immediately exploitable, they demonstrate the efficiency of our structural differential fuzzer, highlighting subtle differences between implementations.

BLS Differential NFT Bounty

We've also written 10 dedicated fuzzers for the following BLS-12-381 libraries (refer to this :

• (library applied in ZCash)

These fuzzers live in the bls-fuzz directory and exercise the serialization of public keys and signatures, along with elliptic curve additions and multiplications.

As a result, we've flagged the following bugs/discrepancies affecting the BLST library:

These high-severity vulnerabilities would have allowed malicious actors to carry out signature malleability attacks (i.e. producing two different signatures for the same message), with disastrous consequences for Eth2 (think RANDAO manipulation!).

Also, another non-critical discrepancy has been raised, please refer to for further details.

Cloud NFT Bounty Infrastructure

Over the past few weeks, we've been deploying and running our fuzzers on a dedicated cloud infrastructure (AWS). We're at present with 16 EC2 instances running more than 30 fuzzers, among them fuzzers targeting the several eth2 networking stacks.

At the moment, we're building, deploying and monitoring these fuzzers manually, with the help of some plain build scripts. We'll be working on an automated tooling over the next few weeks, with the help of DevOps experts.

Teku Integration into beaconfuzz_v2

We've been working away at incorporating Teku into the beaconfuzz_v2 differential fuzzing. As this is the first client to be integrated with beaconfuzz_v2 that is written in an interpreted language, this involves an further technical step of running the Java Virtual Machine (JVM) to execute the Teku bytecode.

The Java Native Interface (JNI) is employed to invoke and interact with the JVM. As this interface is designed for use with C or C++, it was not feasible to immediately invoke the JNI from Rust. In place of that, we incorporated some small C "glue code" to act as an intermediary.

While much of this "glue code" could be reused from the earlier Teku integration as part of the C++ based beaconfuzz_v1, there were some nuances with regards to converting the C++ code to C and passing the result data immediately to byte buffers managed by Rust (to avoid excessive allocations and memory use within C).

Refer to for the Rust & C side of the Teku integration. Though some more polishing and testing is needed before the PR can be merged, we can successfully execute Teku attestation processing as part of the current differential fuzzer.

This integration has also involved some Java-based harness code (to expose relevant Teku state-transition operations), along with modifications to Teku to allow disabling of BLS signature verification (for performance improvements and consistency with present fuzz harnesses). This was completed in Teku PRs and respectively.

By keeping all Java code in the Teku repository, we can take advantage of the Teku CI to detect breaking changes that call for marks to the fuzz harnesses, and it also allows the harnesses to be more without much friction reused for other projects.

Much thanks to the Teku team for their assistance with these modifications and in getting this functionality merged (of particular note was their help implementing unit tests for the harnesses, structuring the fuzz harness project, and integrating it into the build system.)

Custom NFT Bounty Engine

We've also started exploring the possibility of building our own custom fuzzing engine (as opposed to relying on present ones such as AFL, libFuzzer and Honggfuzz).

A custom fuzzing engine would have the following advantages:

• Coverage: Right now, our fuzz targets use one BeaconState per run (selected randomly when these fuzzers initialize). This is due to a constraint in the way libFuzzer and other mutation-based fuzzing engines handle fuzz inputs: each state transition function takes both a BeaconState and a consensus-object (e.g. Attestation, Deposit, and other SSZ containers). Traditional fuzzing engines only allow to mutate one input (in our case the consensus object as a SSZ container), forcing us to rely on the same BeaconState for all blog cycles. By creating our own customized fuzzing engine, we can swap BeaconStates periodically (for example, every 1,000 blog cycle)
• Speed: A custom fuzzing engine will also allow us to materially raise the blog speed by allowing us to load the BeaconStates in memory once per blog initialisation (as opposed to loading them from disk for each client implementation exercised). Also, better multithreading management will also raise blog speed.
• Custom Mutations: We're planning on with as a base for our mutation algorithms, allowing us to alternate between structural and mutation-based fuzzing within the same targets.

Next Steps

Over the next few weeks, the Beacon Fuzz team will be looking into:

• NFT Bounty the new endpoints developed by the client teams as part of the weak subjectivity checkpoint syncing support;
• Enhancing the DevOps experience on our cloud fuzzing infrastructure;
• Potentially performing more BLS differential fuzzing, targeting different bindings.