Asynchronous Program Execution: Dawn of a New Solana APE-och

Introduction

Amid the heated debates and announcement-packed keynotes of this year’s bustling Breakpoint conference, Solana co-founder Anatoly Yakovenko carved out time for a single impromptu and unrecorded technical workshop. Armed with a flip chart and markers, he delved into the intricacies of a topic he passionately champions as Solana's next evolution and the focal point of this article—Asynchronous Execution.

‍

‍

As an entry point to covering the ambitious vision of Asynchronous Execution (shortened to AE from here forward), we’ll begin our tour with a high-level explanation of AE’s core concepts and objectives. Then, we’ll examine Solana’s existing execution and consensus mechanisms to set a foundation for comparative analysis of future proposed architectures. From there, we’ll delve into the various proposals surrounding AE, starting with Bankless Leaders, which serve as a necessary stepping stone for this shift. We’ll also explore the earliest design proposals from 2022 and proceed chronologically to more recent "Endgame Architecture" proposals, which extend beyond AE to include design implementations for multiple concurrent block producers.

‍

AE: 10,000 Feet Overview

“That’s kinda weird to think about. All it [a validator] does is make blocks and vote. It never needs to execute any of those blocks. As it’s creating blocks, it doesn’t know what kind of blocks it’s creating… All you care about is agreeing on the ordering. It doesn’t care what the values are.”  - A.Yakovenko, Solflate Podcast

‍

Within the context of Solana’s core protocol, AE refers to the architectural approach of separating execution from consensus so they can operate independently of one another. The network does this by reaching a consensus on the order of transactions without actually executing them. Once the ordering has been agreed upon, anyone can execute the transactions to reveal the truth.

‍

Under this approach, leaders would build blocks without executing the non-vote transactions and propagate them across the network via Turbine. Validators are also free to vote on the blocks without executing non-vote transactions. Consensus is limited only to the ordering and availability of transactions, greatly reducing the time and steps required to reach consensus. To paraphrase a summary from DBA’s Jon Charbonneau—ordering determines the truth, and execution reveals it.

‍

This approach is very different from how consensus currently works, in which leaders execute transactions before packing them into blocks, and all validators replay transactions in a block before voting on it.

‍

AE is possible because:

1. Fork choice doesn't depend on program execution
2. Only one correct state can be computed, given a deterministic state transition function. Eventually, everyone will compute the same end-state
3. Invalid transactions can be dropped. The cost of failed transaction spam to the network is relatively low, consisting of the Turbine bandwidth required to propagate the transactions and the extra storage of the transactions in the ledger
4. Machines that require real-time, low-latency full-state computation, such as those operated by dedicated RPC service providers, can be provisioned explicitly for this purpose

‍

The potential advantages of AE are many. Yakovenko goes as far as to say, “Asynchronous execution is one of the rare cases where there are virtually no trade-offs.”

1. Faster block times (200 milliseconds)
2. More reliable block times
3. Lower validator requirements
4. Improved user experience (faster finality)
5. Enhanced censorship resistance
6. Reduced window for reordering transactions
7. Rollover of unused capacity between blocks
8. Providing a pathway to Multiple Concurrent Block Producers (more on this later)

‍

We will explore precisely how AE delivers these advantages throughout this article.

‍

Another way to understand AE is that it allows the vote program to operate independently from all other programs. The vote program and the system program (which is always required) are the only programs necessary for consensus within an epoch.

‍

Discussions around AE on Solana have been ongoing for several years. Anatoly Yakovenko's initial formal proposal, originally titled APEX (Asynchronous Program Execution), dates back to April 2022. Concrete details of how AE can be practically implemented on Solana are spread across multiple SIMDs and articles, all of which will be covered in this article in a mostly chronological order. Yakovenko is unquestionably the strongest public proponent and advocate of AE in the Solana community, rarely missing a chance to cover the topic in interviews and podcast appearances.

Terminology

It’s helpful to briefly define two key terms that will feature prominently in our discussion of AE—Bankless Leaders and Multiple Concurrent Block Producers (MCBP). To summarize:

‍

• Bankless Leader: a leader able to create blocks without executing transactions
• Asynchronous Execution (AE): full separation of consensus and execution
• Multiple Concurrent Block Producers (MCBP): like it sounds, multiple blocks produced concurrently within the same slot

‍

Bankless Leaders are a necessary step toward enabling full AE. Similarly, the successful implementation of AE is essential for later introducing Multiple Concurrent Block Producers (MCBP), also referred to as Multiple Concurrent Leaders (MCL). Discussion of AE is often wrapped up together with a discussion of MCBP. Yakovenko advocates for both as crucial parts of Solana’s ‘Endgame Architecture.’

Solana isn't the only blockchain investigating these architectural enhancements. Monad plans to implement AE by decoupling consensus and execution into distinct threads. Likewise, the Ethereum research community has been exploring the possibility of introducing multiple concurrent proposers for some time.

‍

Yakovenko’s proposals are still the subject of ongoing debate within the community, as implementing them would necessitate significant modifications to Solana’s core protocol. Not everyone agrees that AE is the best path forward. It is no exaggeration to say that fully implementing Yakovenko’s proposed Endgame Architecture would radically transform Solana's operations and increase the overall complexity of the protocol.

‍

Before transitioning into a more detailed examination of AE, we must refresh our knowledge of how execution and consensus currently work on Solana, paying particular attention to aspects referenced later in this article. We'll also analyze data on vote and non-vote transactions to uncover additional insights that will inform later discussion. Readers already familiar with and confident in their understanding of these functions may wish to skip this next section.

Synchronous Execution (Solana Today)

Execution

Solana employs continuous block building, which involves assembling and streaming blocks dynamically as they are created during an allocated time slot of 400 milliseconds. Leaders are assigned four consecutive slots (1.6 seconds) before rotation. For a block to gain acceptance by the network, the leader must verify and execute each transaction within the block. Furthermore, all other active consensus-partaking validators will re-verify and re-execute every transaction in the block.

‍

A leader receives transaction packets over Gulfstream via QUIC, entering the Transaction Processing Unit (TPU), the leader's core logic responsible for block production. There are three open ports over which the TPU receives packets:

• tpu: handles regular transactions such as token transfers, NFT mints, and program instructions
• tpu_vote: which exclusively handles vote transactions
• tpu_forwards: if the current leader is unable to process all transactions, it forwards unprocessed packets to the next leader through this port

‍

Before any form of ordering or execution can begin, all transaction packets undergo rigorous validation and sanity checks, including signature verification, checks for the correct number of signatures, and eliminating duplicate transactions. Note, vote and non-vote transactions have separate signature verification processes.

‍

Block-building happens in the Banking Stage. A bank is the state at a given block. Transactions are processed in parallel and packaged into ledger entries—batches of 64 non-conflicting transactions. All transactions include a complete list of accounts to which it will read and write. This design allows validators to easily select only non-conflicting transactions for execution within each entry. Transactions conflict if they write to the same account (two writes) or read and write to the same account (read + write). Conflicting transactions go into different entries and are executed sequentially, while non-conflicting transactions are executed in parallel.

‍

Six threads process transactions in parallel, with four dedicated to non-vote transactions and two handling exclusively vote transactions. Consensus votes are treated as privileged transactions within the Banking Stage—priced uniformly at 0.000005 SOL, consuming 2,100 compute units (CUs), and executed by the Vote Program. 

‍

Once transactions have been grouped into entries, they are executed by the Solana Virtual Machine (SVM). Accounts necessary for the transaction are locked; checks confirm the transaction is recent but hasn’t already been processed. The accounts are loaded, and the transaction logic is executed, updating the account states. A hash of the entry is sent to the Proof of History (PoH) service to be recorded. Upon success, all changes are committed to the bank, and the locks on each account are lifted.

‍

The protocol does not mandate the ordering of valid transactions within a block. The final state of the chain is the result of all confirmed transactions. This state can always be deterministically recreated from the blockchain’s history (i.e., replaying the ledger from Genesis or a snapshot).

‍

Each bank has a corresponding BankHash, a SHA256 hash of the following:

• Parent BankHash - the BankHash of the immediate parent block
• Account DeltaHash - a 16-ary Merkle tree root of all the accounts that have undergone state changes in the current block
• Signature Count - the total number of transaction signatures in the current block
• Last Blockhash

hashv(&[
	parent_bankhash,
	accounts_delta_hash,
	num_sigs,
	blockhash,
])

‍

The BankHash is the cryptographic commitment that validators vote on for each slot.

Consensus

Validators require three accounts to facilitate the voting process:

• Identity Account: This system account is the signer and fee payer for all vote transactions—a hot keypair stored directly on the server hardware. As the name implies, this account’s public key acts as the network identity of the validator. The identity account is required to create a vote account.
• Vote Account: This is the account to which delegators delegate their SOL. The address is used for account lookup, not for signing transactions.
• Withdrawal Account: This account is used to withdraw funds from the vote account. It can change the identity account. The private key for this account is typically stored in a secure multi-sig or cold wallet and must be unique from the validator identity and vote authority keypairs.

‍

Each vote (example) signed by the validator includes the validator’s public key and a hash identifying which block they are voting for. When validators submit correct and successful votes, they earn credits.

‍

The network doesn’t wait for all validators to agree on a newly produced block before producing the next one. As a result, it's not uncommon for two different blocks to be linked to the same parent block, creating forks. Validators must vote on these forks and use Tower BFT, a variant of the original Practical Byzantine Fault Tolerance (pBFT), to determine which one to adopt. 

‍

The network finalizes a single fork when competing ones exist, while validators abandon blocks in discarded forks. Each slot has a predetermined leader for which only that leader's block will be accepted; there cannot be two proposed blocks for a single slot. Therefore, the number of potential forks is limited to a "there/not-there" skip list of forks that can emerge at the boundaries of leader rotation slots. 

‍

Once a validator chooses a fork, it is committed to this fork until a lockout time expires, meaning it must stick with its choice for a minimum period. This mechanism incentivizes validators to vote carefully on the fork they believe has the best chance of being included, i.e., the ‘heaviest’ fork.

 

A block is considered optimistically confirmed once a two-thirds supermajority votes for it. It is deemed finalized once 31 blocks have been built on top of it. There has never been an instance in Solana's history where an optimistically confirmed block did not become finalized.

‍

When a fork is finalized, the block becomes rooted with account updates from the bank, and its ancestors are flushed to disk. Additionally, any account updates from earlier banks that are not ancestors of the finalized bank are pruned.

‍

Vote & Non-vote Transactions

As we have seen on Solana, the transaction execution processes and reaching consensus via Tower BFT are closely intertwined. Both vote and non-vote transactions follow the same pathway: they are bundled into entries, executed by the SVM, stamped within the Proof of History (PoH) stream, and disseminated to the network via Turbine. This unified process ensures consistency in how validators view the consensus state, as propagating votes solely through the gossip service could result in discrepancies across validators. Such discrepancies might result in validators having different perspectives on which fork is more likely to be correct. Continuous block building requires continuous voting, especially while previous blocks are confirmed.

‍

A side effect of this bundling is the controversy over Solana’s transaction per second (TPS) metrics. In their raw form, these include both vote and non-vote transactions, which is not the case for most industry peer networks. This has led to the adoption of a “true TPS”  metric, which excludes vote transactions, as a more accurate measure of transaction capacity.

‍

Examining recent Solana activity, we see vote transactions outweigh non-vote transactions with an average of slightly under 1,000 per block.

‍

Vote transactions account for only a small fraction of the total compute units per block, averaging slightly over 2 million CUs per block and within a consistent range between 2.1 and 2.3 million CUs. This represents just 4.5% of Solana blocks' 48 million CU compute limit.

‍

Vote transactions are very reliable regarding the time and compute they take relative to non-vote transactions. This is a crucial point we will return to later in the article. CUs of non-vote transactions are inconsistent, resulting in inefficient hardware consumption. 

‍

Guaranteeing precise execution timing in the runtime is challenging, as it’s more of an approximate measurement. On average, blocks aim to keep within 400 milliseconds, but occasional spikes occur. These spikes can be considered bugs, and efforts are continuously made to identify and fix them, leading to ongoing improvements.

‍

An epoch comprises 432,000 slots. If each were exactly 400 milliseconds, this would lead to an epoch of precisely 48 hours. Yet, as we can observe from historical data, epochs rarely, if ever, meet this target. Over the past year, epochs have commonly taken in the range of 2.1 to 2.3 days.

‍

Synchronous execution requires all staked consensus-partaking validators to be over-provisioned for the worst-case execution time in any block. Accordingly, Solana’s validator hardware requirements are relatively high.

‍

This concludes our review of how Solana consensus and execution currently work, mainly focusing on the parts relevant to later discussions. We recommend reading our previous Helius blog posts on Consensus, PoH, Turbine, and how Solana works for more comprehensive overviews. In the following sections, we will examine future changes to Solana’s protocol design.

‍

APEX, 2022

Yakovenko’s earliest formal proposal for AE dates back to April 2022, titled APEX (Asynchronous Program EXecution). This relatively short proposal puts forward practical details of how consensus can be separated from execution, introducing the concept of splitting the state into two isolated domains—the default ‘Domain 0’ and the Vote Program ‘Domain 1’. This setup separates the Vote Program from the rest of the bank.

‍

All accounts belong to a single domain; transactions cannot read or write to accounts from multiple domains. Any transactions that do so fail immediately and are ignored. The System Program exists in both domains, with the Vote Program being the only other program in Domain 1.

‍

A special system instruction (SystemInstruction::MoveDomains) provides a mechanism for accounts to move between domains once per epoch to facilitate the transfer of SOL to and from vote accounts.

‍

The proposal also introduces the concept of a ForkHash, a counterpart to the BankHash. Computing the ForkHash only evaluates the Vote Program instructions from Domain 1, with the BankHash evaluating all the non-Vote Program instructions from Domain 0. Vote instructions reference the ForkHash. BankHash computation is done outside of the root on which consensus validators vote. Previously, selecting a fork and voting on it required full evaluation and execution of all transactions in the bank. Now, the only transactions evaluated in fork selection are those involving the Vote Program.

‍

This early design leaves many unanswered questions but establishes the foundational concept of isolated domains, which will be revisited later.

Bankless Leaders

A Bankless Leader is a leader free to produce blocks without executing non-vote transactions, a crucial prerequisite for implementing AE. In a blog post titled The Road to Bankless, Anza engineer Andrew Fitzgerald builds upon an earlier Bankless Leader proposal (SIMD-005, Dec 2022, closed) by Tao Zhu, also of Anza. Fitzgerald outlines several necessary changes to introduce Bankless Leaders on Solana, specifically identifying three key categories of constraints imposed by the protocol that limit the implementation of AE: block constraints, entry constraints, and transaction constraints.

‍

Breaking any of these constraints currently makes the entire block invalid. By eliminating these restrictions, the Solana protocol would open up a path forward for AE and enhance the protocol's flexibility in block production and validation. Below is a summary of the different forms of constraint:

‍

Block Constraints: These are constraints placed on the entire block, including: 

• Blocks must include 64 ticks
• The last block entry must be a tick
• Transactions within all block entries must be within block limits

‍

Entry Constraints: The primary constraint in this category is that entries cannot contain conflicting transactions. Fitzgerald submitted SIMD 0083: Relax Entry Constraints in November last year, specifically addressing this issue. The SIMD has been accepted.

‍

Transaction-Level Constraints: The largest category which can be broken into three sub-categories:

• Static Constraints: Constraints that can be verified without knowledge of any additional state. These include signature verification, transaction size, the number of accounts listed in the read/write address list, and transaction data format compliance (i.e., a transaction must deserialize)

‍

This sub-category of transaction-level constraints does not need to be altered.

‍

• Bank-State Constraints: These include checks for the uniqueness of the transaction signature to ensure it has not already been included in a recent block and age verification (i.e., transactions must include a recent blockhash)

‍

This sub-category requires knowledge of the bank's recent state.

‍

• Account-State Transaction Constraints: The most restrictive sub-category of constraints, including address lookup table (ALT) resolution, nonce checks, fee-payer checks, and executable checks

‍

This sub-category requires knowledge of previous transactions.

‍

Fitzgerald formally proposed changes to this category of constraints in SIMD 0082: Relax Transaction Constraints, which is currently open.

‍

As Fitzgerald notes, the primary requirement for AE is that block verification cannot rely on account state. If the verification of a block requires the results of previous transactions, then the block cannot be executed asynchronously. Accordingly, it is necessary to eliminate dependencies on the account state for block verification. Furthermore, many of these constraints are unnecessarily restrictive.

‍

Relaxing these constraints does not change the current requirements for executing a transaction. It only means that block validation will not reject the entire block if one of the relaxed constraints is broken.

‍

For example, a transaction in which the signer has insufficient lamports to pay fees should not be executed in the current protocol or with proposed changes. But with the proposed changes, including such a transaction will not mark the rest of the block as invalid.

‍

Overall, Fitzgerald’s work offers a practical framework for how the banking stage could be modified to pave the way for AE, outlining many consensus-altering steps required to make AE a reality.

APExB 2023: AE meets MCBP

“Async program execution is gonna turn the whole Solana state into a rollup.”

- A.Yakovenko

‍

In March 2023, nearly a year after the initial APEX design, Yakovenko introduced a more comprehensive and detailed proposal titled Asynchronous Program Execution and Broadcast (APExB) SIMD 0023. This proposal extends beyond the implementation of AE, presenting a broader vision that includes the integration of Multiple Concurrent Block Producers (MCBP).

‍

The proposal introduces the concept of ‘builders,’ which are separate entities from leaders. Builders are staked nodes that create ‘UserBlocks’ comprised exclusively of non-vote transactions. These UserBlocks are propagated through the Turbine tree and have their own ‘UserBlockSlots.’ By default, there are two UserBlockSlots per regular slot.

‍

Builders have their own schedule independent of the leader's schedule. Multiple builders can be scheduled to build UserBlocks concurrently. Compute units are divided evenly between builders and UserBlock slots. For example, a block with two builders, two UserBlock slots, and a 48 million CU limit will have a UserBlock slot CU limit of 12 million (48/4).

‍

Leaders create blocks of consensus votes as usual following the leader schedule, with builders concurrently producing and transmitting UserBlocks of user transactions. UserBlocks specify their slot number and are signed by the builder to prevent the leader from manipulating or excluding part of the block. Once the leader receives a UserBlock over Turbine, it generates a UserBlockEntry using a hash of the UserBlock and adds this UserBlockEntry to the Proof of History (PoH).

‍

Validators cannot vote on blocks for which they have yet to receive the accompanying UserBlocks; otherwise, there is no way to guarantee that everyone can execute the data because it could be withheld. However, they can vote on leader blocks by executing just votes before executing the UserBlock transactions. Validators only execute UserBlocks on the heaviest fork and only need to submit their most recent BankHash when voting, which may be for an older parent slot. Validators vote on VoteHashes, which serve the same purpose as ForkHashes in the APEX 2022 proposal—essentially a BankHash for only vote transactions.

Builder Schedule

If there are ten UserBlock builders per network block, each will be assigned 10% of the turbine shreds and 10% of the compute available. Builders are scheduled through two processes: randomized and persistent.

‍

At the boundary of each epoch, randomized builders are assigned based on a stake-weighted selection process. Persistent UserBlock slots are allocated through a Dutch auction system and awarded to the highest bidders prepared to burn the most SOL.

Transaction Priority Ordering 

Each UserBlock is assumed to have been created simultaneously during the UserBlockSlot in which the leader encoded it. For each UserBlock within the UserBlockSlot, the transactions are ordered by priority fee before execution. If transactions from two different blocks have the same priority, they are ordered by which UserBlock appears first in the leader's PoH. This means priority fees imply execution priority. Duplicate or invalid UserBlocks transactions are skipped without state changes. If multiple UserBlockEntries contain the same transaction, the second one is skipped.

Above: In Scenario 1, Transaction B will be prioritized despite arriving later in the PoH stream. In Scenario 2, Transaction C will be prioritized as it holds the same priority fee as Transaction D, but its block is earlier in the PoH stream.

‍

Under this design, builders will capture all the MEV. The proposal leaves the question of how builders and leaders should split user transaction fees open.

‍

Builders can also create BundleTransactions, which are groups of transactions within the UserBlock designed to be executed in a single ordered batch. These transactions can also add a priority fee so that the entire bundle is prioritized for execution as a batch, similar to the current implementation of Jito bundles.

Epoch Boundaries

Stake weights influence fork choice, which are updated at each epoch boundary. This has important implications for the design. Validators must have caught up on executing non-vote transactions before they can continue voting beyond that epoch boundary. However, Yakovenko notes, “[n]odes can't fall behind that much because the overall CU limits are set for synchronous execution. But with the option of async execution, it is much easier to catch up. Raw ledger processing without dealing with the network is 20-30x times faster.”

Considerations

Having multiple builders means resource management becomes more complex. Clients can pick their nearest builder, but priority fees may differ for each UserBlock, and users will likely be unaware of which block may become saturated when sending their transactions.

‍

Benefits include the predictability of scheduling. Applications can bid for the percentage of block bandwidth they need for operation and create dedicated sequencers that guarantee eventual settlement into the chain. Including randomized UserBlock builders is crucial as it prevents persistent block builders from censoring transactions and blocking them from eventually landing.

‍

APE in Solana 2024

In an X article titled Asynchronous Program Execution (APE), published in June of this year, Yakovenko further fleshes out the concept of execution domains initially described in the 2022 APEX proposal.

‍

Execution Domains, originally Domain 0 and Domain 1, are now renamed the Vote Execution Domain (VED) and User Execution Domain (UED). Their purpose remains the same—to fully isolate consensus voting from transaction execution. Execution Domains are defined as distinct sets of programs, along with the keys and values they interact with, executed independently from other sets:

• Execution Domains can run on different threads and cores and complete at different times across physically separate machines 
• Execution Domain A cannot read or write to any values in execution Domain B 
• Domains can share state that remains consistent during the execution of either domain
• The fee payer determines the domain in which a transaction is executed
• A protocol is needed to synchronize the state between domains and facilitate the movement of keys and values

Vote Execution Domain (VED) Components

• SystemProgram: used for transfers
• VoteProgram: The core program for voting. This program is static and must be present in both the UED and the VED
• VoteProgram Sysvars: variables used for voting
• Vote Authority: Accounts authorized to vote
• Vote Fee Payer: Accounts that pay fees related to voting
• Fee Payer Funder: An updatable account that can be used to move SOL into the VED

‍

Accounts not in the VED are considered part of the UED.

‍

Vote accounts must be activated. They are included in the VED the next epoch after being activated. Once deactivated, they are removed from the VED in the next epoch. Formal processes are outlined for the movement of funds in and out of the VED.

‍

Accounts track which Execution Domain they are mapped into following a similar convention to Linux file permissions (R read, W write, and X execute). There are four valid mappings:

Only Vote Program and System Program accounts can move between VED and UED. The System Program provides an interface to move System accounts and Vote Program accounts to and from VED. This approach removes the need for an explicit FeePayerFunding account. Any System account can be remapped between VED and UED to move funds between domains.

‍

Leaders only execute the VED domain for the blocks they create, which means they may have partial or incomplete information about the status of UED fee payers. After a block is received, validators first execute VED transactions. The resulting VED state is used to compute the VED Hash, which validators then use to vote. 

‍

UED replay skips transactions with invalid fee payers. UED state updates compute the Bankhash, known as the UED Hash. If one-third or more validators submit different UED Hashes, all nodes must halt and alert operators.

‍

New Feature Activations
With consensus now decoupled from execution, the VoteProgram must be designed to handle scenarios where VED votes may cross the epoch boundaries, resulting in new feature activations occurring at different times than transactions interacting with the VoteProgram in UED.

‍

Endgame Architecture 2024

“Given the diversity of applications and core developers, planning a single major protocol change per year is worth planning. If we have to choose one, I would vote for Asynchronous Execution.” - A.Yakovenko, Endgame Architecture.

‍

In early 2024, Yakovenko published Endgame Architecture, a document asserting a bold vision of Solana’s future as a 10,000-node network with 120 millisecond block times. The document reaffirms and expands upon earlier design proposals while emphasizing the adoption of AE as the path toward achieving this ambitious vision.

‍

Bankless Leaders

To summarize:

● Leaders maintain a cache of fee-paying account balances

● If a fee payer is used as a writable account as a system transfer source or passed as a writable account along with the system program to another program, the fee-paying balance is set to 0

● Blocks are packed based on declared CUs until the block is full based on the local fee priority ordering and subtracting transaction fees from the fee-payer balance cache 

● Fee paying account balance cache is replenished by the BankHash computation 

‍

Leaders can initially obtain account balances for fee payers by querying multiple full nodes or RPCs. In the rare case that nodes provide incorrect data, the result would be failed transactions rather than a consensus failure. If the fee-payer balances are outdated, it could cause spam within blocks without impacting consensus. The cost of failed transaction spam is relatively low, primarily involving the Turbine bandwidth required to propagate the transactions and the additional storage space they occupy in the ledger. Furthermore, both Temporal and Firedancer teams are actively working on tools that mitigate spam before it reaches blocks by implementing mechanisms like filtering invalid transactions and deduplication and preventing transactions that are bound to fail due to read/write lock conflicts.

‍

This issue is easy to detect and monitor. Operators can track leader performance and assess the level of spam in blocks, allowing them to quickly resolve problems and switch to an alternate data source if necessary. Since validators are motivated to maximize their earnings, they are incentivized to maintain an accurate cache of fee-payer account balances.

Fixed-Sized Sub-Committees

Implementing AE across a large 10,000-node network necessitates the introduction of fixed-size rotating voting committees, which represents a significant shift in the consensus mechanism. This change stabilizes the resource costs of consensus voting regardless of the total number of staked nodes. Yakovenko proposes that a rotating committee of 200 or 400 nodes would be sufficient. This setup allows validators to engage in consensus voting with minimal state requirements—limited to the quorum, stake weights, and vote account balances. The memory footprint is small, and the corresponding snapshot file can be easily distributed and reinitialized upon restart.

‍

“The real irony is going to be that after async execution a Solana consensus validator could run on an FPGA. Turbine + tower for 10k vote accounts would easily fit on an FPGA. Maybe like 4 mb of state to track.” - A.Yakovenko, X post

‍

Using fixed-size rotating subcommittees in Byzantine Fault-Tolerant (BFT) protocols is an established concept already implemented by peer networks such as Tron in production. It is based on the committee sampling approach, which selects a smaller committee to handle consensus and relays the results to the entire network of replicas.

‍

An initial approach for Solana could simply allocate a set number of CUs to votes, with the top validators by stake fitting into these CUs selected each epoch to track fork choice. Other validators would still vote and earn rewards, but their votes wouldn’t be considered for fork choice.

Vote Accounts

● Vote accounts must have enough SOL to cover two epochs of votes 

● Vote transactions must be simple votes

● Withdrawing SOL from the vote account is allowed if the balance exceeds one epoch's worth of votes

● To remove all the lamports, the Vote CLOSE instruction must require an entire epoch to pass. Vote accounts are marked for CLOSE on epoch one but can only be CLOSE on epoch 2

CLOSE allows all the sol to be withdrawn and the vote account to be deleted. Once an account is marked for CLOSE, it can only be deleted and cannot be reopened

● Votes contain a VoteBankHash instead of a regular BankHash 

‍

BankHashes

Validators compute a VoteBankHash for simple vote transactions using the same format as the current BankHash, disregarding all other transactions. These VoteBankHashes incorporate the previous VoteBankHash rather than the full BankHash.

‍

For blocks optimistically confirmed by a two-thirds supermajority, validators also begin calculating the UserBankHash, which includes all state transitions except those already considered in the VoteBankHash.

‍

A BankHash is computed for every slot by combining the VoteBankHash, UserBankHash, and the previous BankHash. The top 99.5% of validators submit this BankHash as part of their vote every 100th slot. Additionally, some nodes can broadcast the BankHash through the gossip network to signal that no non-determinism has been detected.

‍

Suppose fewer than two-thirds of validators submit the full BankHash. In that case, leaders may reduce block space for user transactions and writable accounts by 50%, preventing exploits that could excessively increase replay time.

‍

State only needs to be computed once per epoch to establish the next quorum, ensuring consensus nodes and leaders remain synchronized. Execution can be aggregated and batched on machines separate from the consensus nodes. Users requiring synchronous execution (most applications and RPCs) can dedicate hardware resources to process every state transition in real-time without waiting for the broader network.

Considerations

RPC providers offering real-time state data to users do not have a signature to verify that their locally computed state matches the state computed by the broader network. However, once a fork is finalized, the network converges on a single, correct canonical state that can be deterministically computed. To minimize the risk of runtime bugs or data corruption, nodes that aim to provide accurate state data should operate multiple nodes. If any discrepancies in state execution are detected, operations should be stopped immediately.

‍

Additionally, users can submit transactions that assert the BankHash or trigger an abort. The network will only process these transactions if the computed BankHash matches the one provided to the user by their RPC provider.

‍

Conclusion

Yakovenko envisions a bold future for Solana, with block times reduced to 120 milliseconds and a dramatic increase in nodes, made possible by adopting AE. However, realizing this future requires rallying separate client teams and the wider Solana developer community around this vision and addressing the significant engineering challenges of implementing such changes. 

‍

Prominent community members have voiced concerns about many of these proposed changes. Firedancer team’s Richard Patel cautioned that "implementing asynchronous execution for block producers is quite complex and carries substantial risks." Although supportive of AE, Zano Sherwani of Jito Labs criticized MCBP, stating, “Multiple Concurrent Proposers is a terrible solution that’s looking for a problem to solve. The amount of complexity it brings to the protocol is not warranted by what it’s (if anything) trying to solve.” On a recent podcast, when asked about multiple block builders, Mert Mumtaz of Helius replied, “I’m not fully convinced.”

‍

AE undoubtedly holds the potential to deliver significant performance benefits. Shorter block times lead to faster confirmations and improved user experience while enhancing censorship resistance and reducing the window for transaction reordering. However, we must also carefully evaluate the drawbacks, including the potential increase in protocol complexity and greater reliance on RPC providers. In specific scenarios, these providers may be the only participants with an up-to-date, real-time view of the chain’s state.

‍

AE represents a major upgrade for Solana. With this article, we aim to raise awareness within the Solana developer community about the exciting changes AE will introduce and encourage more informed discussion on this important topic.

‍

Many thanks to 0xIchigo and Anatoly Yakovenko for reviewing earlier versions of this work.

‍

Additional Resources

• Consensus On Solana - Helius Blog
• Solana Executive Overview - Helius Blog
• The Road to Bankless - A. Fitzgerald
• Asynchronous Program Execution (APE) - A.Yakovenko
• Endgame Architecture - A.Yakovenko
• Asynchronous Program Execution and Broadcast (APExB) SIMD 0023

‍