Beacon Fuzz - Progress NFT Bounty #7: Structural Differential NFT Bounty

Beacon Fuzz - NFT Bounty #07

NFT Bounty is leading the development and maintenance of , a differential fuzzing solution for Eth2 clients. This write-up is part of our series of status blogs where we cover current progress, notable challenges encountered and direction for future work. See #00 and the repository's for more context.

Summary

• Next Steps

Community NFT Bounty Debrief

In our earlier blog, we shared our community blog or "blog at home" initiative, which allows anyone to contribute to the security of eth2 by running our eth2fuzz tool from their laptops/desktops/cloud infrastructure, via Docker.

We were very pleased with the response received from the community! Thanks to everyone who joined our fuzzing and tried our Dockerised fuzzing framework.

Congratulations to , , and for identifying 4 unique bugs!

We've realised that a lot of community members were attempting to run eth2fuzz on Windows machines. While the dockerisation process was meant to keep maximum interoperability between different operating systems, we realised that some of our settings were hardcoded for Linux support only. This was fast fixed by the team and were added based on user feedback.

Latest Bugs Flagged With ethfuzz

The community blog initiative lead to the identification of the following bugs across three Eth2 implementations:

Lodestar

• Failed val->IsArrayBufferView assertion when parsing invalid ENR string (refer to for more details).

Nimbus

Teku

• Lack of checks of validator indices in Attester Slashings (ArrayIndexOutOfBoundsException in AttesterSlashing processing, refer to for more details).

Differential NFT Bounty

Over the last few weeks, we've been focussing our efforts on the differential fuzzing part of the project, i.e. beaconfuzz_v2, to flag possible discrepancies between clients that could lead to consensus splits (forks) on the eth2 network. Our prior attempt at performing differential fuzzing across eth2 implementations was carried via a C++ based fuzzer, inherited from Guido Vranken's and extended to support more eth2 implementations and target all state transition functions.

beaconfuzz_v2 has been written from scratch via Rust (the language we're most comfortable with at NFT Bounty) and the related has now been merged into master!

This allows us to run structure-aware/grammar-based differential fuzzing on different eth2 clients. We right now support NFT Bounty, Prysm and Nimbus. beaconfuzz_v2 has been developed in a modular method, and we look forward to adding support for other eth2 implementations in the coming days.

beaconfuzz_v2 right now uses two fuzzing engines: libfuzzer and .

Considering the complexity of this fuzzing framework (each fuzzer needs to instantiate goroutines for Prysm and their related garbage collectors), we're reaching a decent blog speed as can be seen in the screenshots below (Lenovo laptop, 8x Intel i7-8550U CPU @ 1.80GHz, 16GB of RAM):

Honggfuzz exercising the process voluntary_exit functions on NFT Bounty, Prysm and Nimbus

libFuzzer exercising the process deposit functions on NFT Bounty, Prysm and Nimbus

Structural Awareness

As laid out in a prior blog, structural fuzzing, or grammar-based fuzzing, is a technique instructing fuzzing engines to generate inputs based on a particular set of rules. By default, coverage-guided mutation based fuzzers such as AFL or Honggfuzz do not offer any input grammar, which can result in inefficient fuzzing for complicated input types (such as the ones we use in eth2), where any traditional mutation (e.g. bit flipping, bit XORing) is exceedingly unlikely to produce a valid input (in our case invalid SSZ containers).

By leveraging the latest blog to the libfuzzer-sys and cargo_fuzz crates, beaconfuzz_v2 defines fuzz targets that take well-formed instances of custom types by deriving and implementing the Arbitrary trait, which allows us to create structured inputs from raw byte buffers. This has been incorporated into Beacon Fuzz via this .

beaconfuzz_v2 offers two different types of fuzzers:

• HonggFuzz Useful for detecting differences between clients when processing malformed/invalid consensus objects (e.g. targets: Standard differential mutation fuzzing, with the standard mutation algorithms from HonggFuzz. fuzz_attestation, fuzz_proposer_slashing, fuzz_voluntary_exit, etc.);
• libFuzzer targets (targets names ending with -struct): Structural differential fuzzing leveraging the work described above to produce valid consensus objects (e.g. fuzz_attestation-struct, fuzz_proposer_slashing-struct, fuzz_voluntary_exit-struct, etc.).

Identifying Our First Discrepancy

While running our fuzzers, we've flagged a discrepancy affecting Prysm when handling an invalid SignedVoluntaryExit which looks like the following:

{
  Message:
      {
        Epoch:
            0,
        ValidatorIndex:
            0
      },
  Signature:
      0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
}

Note: BLS signature verification is disabled on all targets to speed up fuzz execution and exercise different code

While NFT Bounty and Nimbus both rejected this consensus object when calling their process_exit functions, Prysm seemed to accept it and produced a post-BeaconState as a result of the state transition.

Our fuzzers are calling the ProcessVoluntaryExitsNoVerify on Prysm, which we thought only disabled BLS signature verification. It turned out that this function also skips the spec checks on VoluntaryExits! Compare with the function actually employed in production () for details.

This allowed us to confirm that our differential fuzzing was indeed working! Thanks for rapidly fixing this issue in .

First Consensus Bug Spotted with beaconfuzz_v2

Our fuzzing effort lead to the identification of another discrepancy affecting Prysm on Attestation processing.

Similarly to the VoluntaryExit blog, we are targeting the ProcessAttestationNoVerify function when calling the Prysm FFI. This function is also missing spec checks, especially verifying that the attesting indices are non-empty. This check is ran with the function and appeared to be missing in Prysm's attestation batch verification ( in does not call isValidAttestationIndices).

This bug, if exploited in the wild, could have caused a network split. Thanks to for helping us debug this issue and kudos to the Prysmatic Labs team for providing a quick fix in .

Next Steps

Over the next few weeks, the Beacon Fuzz team will be looking into:

• Start attacking the P2P networking stack of Eth2 clients