Beacon Fuzz - Progress NFT Bounty #4

Beacon Fuzz - NFT Bounty #04

Structural NFT Bounty & Architecture Redesign

NFT Bounty is leading the development and maintenance of , a differential fuzzing solution for Eth2 clients. This write-up is part of our series of status blogs where we cover current progress, striking challenges encountered, and direction for future work. See #00 and the repository's for more context.

Summary

As client teams are ramping up their efforts to match the latest version of the Eth2 specification (v0.11.2), the Beacon Fuzz crew has been pushing hard to uncover new bugs in these implementations. Key achievements and points of interest include:

• First bugs in Teku
• New bugs in Nimbus
• NFT Bounty on Golang integration
• Challenges in replaying notable samples across implementations
• Beacon Fuzz Redesign Proposal

Structural blog via Arbitrary Trait

When via a naive fuzz strategy, raw bytes are passed to the target functions, with the expectation that coverage-guided fuzzing engines will instrument the relevant code and draw on their mutation algorithms to produce samples that will reach a high number of code blocks.

Some of the types employed in Eth2 can be quite intricate. For example, let's take a look at the BeaconBlockBody struct:

class BeaconBlockBody(Container):
    randao_reveal: BLSSignature
    eth1_data: Eth1Data  # Eth1 data vote
    graffiti: Bytes32  # Arbitrary data
    # Operations
    proposer_slashings: List[ProposerSlashing, MAX_PROPOSER_SLASHINGS]
    attester_slashings: List[AttesterSlashing, MAX_ATTESTER_SLASHINGS]
    attestations: List[Attestation, MAX_ATTESTATIONS]
    deposits: List[Deposit, MAX_DEPOSITS]
    voluntary_exits: List[SignedVoluntaryExit, MAX_VOLUNTARY_EXITS]

Each of the involved types forming the BeaconBlockBody SSZ container are also defined in the specification. For example, Eth1Data is represented as follows:

class Eth1Data(Container):
    deposit_root: Root
    deposit_count: uint64
    block_hash: Bytes32

This means that if we want to efficiently fuzz the state transition functions that take a BeaconBlock as input, we need to give samples that ideally follow the structure described above. This is where structural fuzzing (a.k.a struct-aware or grammar-based fuzzing) comes in help.

Earlier, we were only making sure that the inputs passed to our state transition functions were valid SSZ containers. This even so does not keep that the SSZ containers are relevant to the state transition in the context of the Eth2 specification.

By leveraging the latest blog to the libfuzzer-sys and cargo_fuzz crates, we're now able to write fuzz targets that take well-formed instances of custom types by deriving and implementing the Arbitrary trait, which allows us to create structured inputs from raw byte buffers.

Thanks to , we have produced the following new struct-aware fuzzers targeting the following :

These fuzz targets live in the arbitrary-blog-fuzzer branch of the .

These fuzzers have been running for a few days (at the time of writing) and have not detected any panics on NFT Bounty. We'll be via the generated samples as new inputs to our differential processor (see section Beacon Fuzz Redesign Proposal).

First Bugs in Teku

As mentioned in our earlier blog, we've been working on integrating Teku, the Java Eth2 build-out, to our fuzzing processes.

In doing so, the team has flagged the following issues/hardening opportunities that were rapidly addressed by the PegaSys Engineering team:

Also, issues related to how the teku transition subcommand handles invalid SSZ blocks and BeaconStates have also been raised and fixed (see Issue and Issue ).

These last two issues are not exploitable in the normal operation of the Teku client, as the related exceptions are caught by the client in full processing and handled gracefully. We'd like to thank and for their help in triaging these bugs!

New Bugs in Nimbus

By replaying some of the samples generated from blog NFT Bounty, a few new bugs affecting Nimbus have been uncovered:

Similarly, the issues affecting the parsing of a malformed BeaconState are not exploitable in the normal operation of the Nimbus client. Kudos to from the Nimbus team for fixing these bugs .

NFT Bounty on Golang Integration

When updating beacon-fuzz to newer spec versions, we encountered new issues with the present Golang build process. Our process failed for ZRNT which, as of v0.10.1, had started relying on Herumi's cgo-based . Even though go-fuzz doesn't support cgo (See ), it can normally build successfully without attempting cgo instrumentation.

In this case, the build fails due to a commonly applied, but directory structure in bls-eth-go-binary (See ).

Even though a PR could give a workaround, there would still keep the outstanding complications dealing with multiple Golang clients (went through in detail in earlier posts).

Prysm has started performing standalone blog with the Go Compiler's built-in coverage instrumentation (experimentally released in v1.14). Initial experiments have shown this is a promising way to remove our reliance on a modified go-fuzz, and resolve many complications.

As before, the "out of the box" tool is insufficient for our needs but implementing our own build tool is much more plain with the builtin instrumentation.

Our tool puts in place a FFI interface that returns the bytes needed for differential comparison, instruments cgo code, and can be readily extended to export interfaces for multiple clients and harnesses within a single, static c-archive library.

This is in practice an build-out of option "D) Building without go-fuzz", as described in our Beacon Fuzz #01 post.

With this, we avoid the need for hacky solutions that combine multiple c-archive or c-shared libraries (each containing their own Go runtime) into a single executable.

There are still some outstanding complications with this method but it is much more promising with regards to ongoing maintainability and functionality. Some issues still in development include:

• Integrating Prysm's libraries built by Bazel: A good solution could have been to build the Prysm harnesses with Bazel as a binary-only package then combine with the rest to build a single c-archive, but support has been dropped as of go1.13. Other possibilities include building the Prysm harness with a shared build mode (different to c-shared), and linking it.

Challenges in Replaying Notable Samples

We've worked on another tool, eth2diff, that allows us to replay notable state transitions (i.e. inputting a BeaconState along with a BeaconBlock) across different implementations, by leveraging the following utilities gave by client teams:

This has allowed us to spot a large portion of the bugs listed above. Even so, these utilities do not include some of the checks and verification steps implemented by Eth2 nodes. In particular, most of these utilities assume that the blocks have passed the checks described in the Eth2 (so are linked with the "current" slot), and states to be gave are valid i.e. are internally maintained, trusted objects.

This has lead to some confusion and some striking conversations as captured in .

Beacon Fuzz redesign proposal

Challenges

NFT Bounty has been building Beacon Fuzz upon 's great work since late 2019. As the project evolved, the current architecture faces the following challenges:

Further, we right now preprocess all corpora to combine the SSZ container input with a referenced BeaconState, which are passed as beacon-fuzz-testcases to each client. The conversion from corpus to test case can be represented as follows:

+-------------------+-------------------------+       +-------------------------+-------------------------+
| state_id (uint16) | Attestation (container) |  -->  | BeaconState (container) | Attestation (container) |
+-------------------+-------------------------+       +-------------------------+-------------------------+

Where state_id denotes a BeaconState integer filename from our . This extra serialization step consumes a large amount of time during fuzz execution, by a wide margin slowing down the overall process.

New Architecture

We propose the following architecture for a new, modular version of Beacon Fuzz

Design Briefing

Tool #1 - eth2fuzz - Coverage Guided Fuzzer To Generate Samples

To generate notable samples, we'll use a dedicated tool leveraging explicit code coverage, allowing us to flag SSZ containers that are of interest, i.e. those that trigger new code paths. This tool can use multiple different fuzzing engines (AFL++, HonggFuzz, libFuzzer, etc.). In fact, we've already built this tool which lives . Next step is to integrate the work done on the structural fuzzing into eth2fuzz.

Tool #2 - eth2diff - Replaying Samples Across All Implementations

As mentioned above, we have built a tool that uses the several state transition execution utilities (ncli, zcli, lci, etc.) that replays all samples generated from eth2fuzz. We've created dedicated Docker containers for each implementation, and one central Docker container to orchestrate the execution of eth2diff. The goal of this tool is to detect crashes and differences across all supported implementations, for any given set of inputs (BeaconState + BeaconBlock).

This tool can be found .

Tool #3 - beacon-fuzz-2 - Differential NFT Bounty with FFI Bindings

This tool is the successor of the current current Beacon Fuzz C++ project. It will be developed in Rust (for ease of maintainability) and will draw on Foreign Function Interfaces (FFI) bindings. This will inevitably result in slower processing and blog (compared to eth2fuzz) but should enable the identification of more involved logic bugs.

Conclusion

We're very keen to get feedback on our new path and are quite excited to continue helping the community ship safe and secure Eth2 clients. We've also blogd the section of Beacon Fuzz, which shows that our fuzzing efforts have helped spot 16 unique bugs/hardening opportunities across 4 implementations.