Solana’s Gulf Stream: Mo Mempool, Mo problems

Actionable Insights

• The term "Gulf Stream" can be loosely defined as the process that occurs from the moment a node picks up a transaction on the network until it reaches the leader for the current slot and is received by the Fetch Stage of the TPU (Transaction Processing Unit).

• Solana stands out because it was designed from the outset to operate without a mempool. Unlike more traditional blockchains that use gossip protocols to broadly propagate transactions across the network, Solana forwards all transactions to a predetermined lead validator, known as the leader, for each slot. The leader alternates every 4 slots, and the leader schedule is known in advance by all active network nodes, ensuring efficient transaction forwarding.

• By default Solana transactions must include a recent blockhash, which can be easily requested by developers through a simple API call. A recent blockhash is valid for up to 150 slots (approximately 1 minute). After this period, it becomes stale, and transactions referencing it will be dropped by the network. This ensures unprocessed transactions cannot linger. Recent blockhashes aid in transaction deduplication. Methods also exist for developers to add durable nonces.

• Transactions in Gulf Stream are encoded and sent via QUIC streams to the leader. The adoption of QUIC in late 2022 was a significant network upgrade, replacing the previously used UDP connections. The primary driver for this change was to enhance the network’s ability to filter spam. However, the switch to QUIC has proven somewhat contentious, as Solana has experienced unprecedented levels of activity throughout 2024.

• The introduction of Stake-weighted Quality of Service (SWQoS) in early 2024 marked a profound change in how transactions reach the leader through Gulf Stream. Leaders now prioritize transaction messages proxied through other staked validators. Specifically, 80% of a leader’s capacity (2,000 connections) is reserved for staked peers, while the remaining 20% (500 connections) is allocated to transaction messages from non-staked nodes.

• Currently, more than 80% of the stake on Solana is locked up with validators running the Jito-Solana client over the original Agave client. Jito introduces an out-of-protocol blockspace auction, introducing further complexity in how transactions reach the leader. Specifically, the Jito relayer adds a 200-millisecond “speed bump” to slow the flow of incoming transaction messages, allowing searchers enough time to submit bundles.

Introduction

The term "Gulf Stream" originates from a series of introductory blog posts written by the Solana founding team in 2019, where they assigned aviation-themed titles to many of Solana’s more innovative mechanisms. In these posts, Gulf Stream was defined as Solana’s “mem-pool less transaction forwarding protocol.” However, a search for the term “Gulf Stream” in the Solana codebase today returns only a single trivial mention.

‍

In the broader context of the Solana transaction lifecycle, “Gulf Stream'' can be understood as the entire process that occurs from the moment a transaction is picked up by a network node, typically an RPC, until it reaches the leader for the current slot (i.e., when it is picked up by the “fetch stage” of the TPU). We could also conceptualize Gulf Stream as being the mirror image of Turbine, Solana’s block propagation mechanism, given that Gulf Stream is how transactions reach the leader and Turbine is how processed transactions leave the leader.

‍

First, it may be useful to define RPC (Remote Procedure Call) nodes in the context of Solana. These nodes can be thought of as gateways to interact with and read data from the network. They act as intermediaries between users and Solana’s validators. RPCs run the same software as full validators but with different settings, allowing them to accurately simulate transactions and maintain an up-to-date view of the current state (the bank). However, RPC nodes are unstaked and, therefore, non-consensus. Without stake, they cannot vote or build blocks. This setup differs from many other blockchains, where validators and RPC nodes are typically the same. With regard to Gulf Stream, we can summarize the RPC's role as receiving transactions over HTTP, converting the transactions to QUIC (more on this later), looking up the current leader’s address and port information using the leader schedule, and forwarding the transaction to the current and next leaders.

‍

Since the network’s inception, Gulf Stream has undergone at least two major upgrades, QUIC and Stake-weighted QoS, which will be discussed in detail later in this article. It is also the part of the core protocol that has arguably faced the most strain in recent years due to Solana’s unprecedented volume of network traffic. To provide context on this point, when a validator becomes the leader, they can expect inbound traffic to spike, reaching over a gigabyte per second as the entire network directs packets toward them. Handling such intense volumes of data ingress is a formidable engineering challenge.

‍

Mo Mempool, Mo Problems

The founding team’s original definition of Gulf Stream emphasizes the lack of a mempool. We can define a mempool (literally a “memory pool”) as a set of transactions that have been submitted by users and are waiting to be processed by the network. These transactions are typically not encrypted or shielded as they wait in public. They are propagated throughout the network using a gossip protocol. Depending on the network, signed transactions can potentially remain in the mempool indefinitely until the conditions for their execution are met. This is especially the case for those transactions that have set a transaction fee (price per unit of compute) significantly lower than the usual range of market rates. In extreme cases, such transactions may take days or weeks to execute if network conditions do not favor their inclusion in a block.

‍

Such a scenario is not possible on Solana, which not only has no native concept of a mempool, but all transaction messages must include a recent blockhash. Developers can easily request recent blockhashes via a JSON RPC API call to the getLatestBlockhash method. This blockhash is embedded in the transaction message and is valid for up to 150 slots, approximately 1 minute, considering each slot has a target time of 400 milliseconds. After 150 slots, the blockhash becomes stale, and transactions referencing it are dropped by the network. By default, RPCs attempt to forward transactions every 2 seconds, but once the recent blockhash expires, the transaction is dropped, guaranteeing that it will never be executed on-chain.

‍

The recent blockhash is also a way to detect and remove duplicate transactions, which on other networks would be achieved by requiring the inclusion of a nonce (a number used only once). Although methods are available for developers to add durable nonces to Solana transactions in specific niche scenarios, there is no requirement for nonces to be included as part of a standard Solana transaction.

‍

Solana’s Gulf Stream system is feasible because the leader schedule is always known ahead of time by all active nodes. Nodes update their leader schedules each time the slot height crosses an epoch boundary, roughly every 2 days. The leader schedule for an epoch is calculated from the ledger state at the start of the previous epoch, with the algorithmic process for generating the leader schedule being as follows:

‍

• Periodically use the proof of history (PoH) tick height (i.e., a monotonically increasing counter) to seed a stable pseudo-random algorithm.
• At that height, sample the bank for all staked accounts with leader identities that have voted within a cluster-configured number of ticks. This sample is called the active set.
• Sort the active set by stake weight.
• Use a random seed to select nodes weighted by stake to create a stake-weighted ordering.
• This ordering becomes valid after a cluster-configured number of ticks.

‍

Source: Solana official docs

‍

Stake-weighting ensures that trusted nodes with higher stakes have a higher chance of being chosen as the leader more frequently, while nodes with lower stakes are chosen less frequently or not at all. 

‍

‍

Stake-weighted allocation of resources is a recurring theme throughout the Solana core protocol, encompassing voting rewards, turbine trees, leader schedules, and the gossip network*. Even Gulf Stream employs stake-weighting, as will be discussed later in this article.

‍

* Solana also has a gossip network. This network is not used for transactions; instead, it functions as a control plane, distributing metadata about the blockchain's state, such as node contact information, address lists, and available ports.

‍

A QUIC Note

The first major update to Gulf Stream came in late 2022  with the adoption of the QUIC networking protocol to handle transaction message transmissions to the leader. This upgrade was prompted by the network disruptions caused by DDoS attacks and spam transactions flooding the chain during NFT mints. Following the full integration of QUIC into Mainnet-Beta with release 1.13.4, the network’s stability improved.

‍

Previously, Solana had relied upon the UDP (User Datagram Protocol) networking protocol to send transactions from RPC nodes to the current leader. While fast and efficient, UDP is connectionless and lacks both flow control and receipt acknowledgments. Accordingly, there is no meaningful way to discourage or mitigate abusive behavior. In order to effect control over network traffic, the validator’s transaction ingestion protocol (i.e. the TPU’s Fetch Stage) was reimplemented with QUIC.

Initially developed by Google in 2012, QUIC attempts to offer the best of both TCP and UDP. It facilitates rapid, asynchronous communication similar to UDP, but with the secure sessions and advanced flow control strategies of TCP. This allows for limits to be placed on individual sources of traffic so that the network can focus on processing genuine transactions. It also has a concept of separate streams; so if one transaction is dropped, it doesn’t block the remaining ones. Google has been the primary driver of QUICs adoption throughout Web2. Connections to Google servers are established using QUIC, which means many apps under Google’s umbrella are QUIC-based, such as Hangouts, Gmail, and YouTube. As a side note, QUIC is not an acronym but the actual name of the protocol. 

‍

With that being said, the effectiveness of QUIC’s implementation on Solana is open to debate. During spikes of high network traffic validators can become overwhelmed with QUIC handshakes. It’s fair to say it hasn’t been the silver bullet solution that some had originally hoped for in fixing congestion issues at the network level. It should be noted that QUIC has a low level of adoption in the blockchain industry outside of Solana’s implementation. Some members of Solana’s validator community have been open in their criticism of the protocol, believing the adoption of QUIC to have been a misstep.

‍

For a fuller exploration of QUIC, check out our full Helius blog post here.

‍

Stake-Weighted Quality of Service (SWQoS)

‍

In early 2024 Stake-weighted Quality of Service (SWQoS) was adopted as a mechanism designed to prevent spam and enhance Sybil resistance. This system allows leaders to prioritize transaction messages that are routed through staked validators. Validators with a higher stake are granted a proportionally greater capacity to transmit transaction message packets to the leader, effectively mitigating Sybil attacks from non-staked or low-staked nodes across the network. This segmentation is possible because IP addresses are verifiable through QUIC, enabling validators to prioritize and limit the traffic for specific connections. 

‍

With SWQoS, validators can lease their stake-weighted capacity to RPC nodes. In return, RPC nodes gain increased bandwidth, enabling them to achieve higher transaction inclusion rates in blocks. Notably, 80% of a leader’s capacity (2,000 connections) is reserved for Stake-weighted QoS, while the remaining 20% (500 connections) is allocated to transaction messages from other nodes. This allocation strategy resembles a system of priority lanes on a highway, where drivers pay a toll to avoid traffic.

The minimum amount of stake required to qualify as a staked peer is currently 0.04% of the total stake. As of writing the total stake is 384 million SOL, putting this minimum requirement at 15,360 SOL.

‍

SWQoS has significantly impacted the Solana ecosystem by raising the requirements to forward transactions to the leader and reducing the effectiveness of spam attacks. The change has incentivized high-traffic applications to vertically integrate their operations. By running their own validator nodes, applications can ensure privileged access to the leader, thereby enhancing their transaction processing capabilities. At Helius, we are proud to operate one of the top validators by stake on the network, enabling us to secure greater transaction inclusion. Learn more about staking with us in our detailed staking guide here.

‍

For an in-depth exploration of stake-weighted QoS, read our full Helius blog post here.

‍

The Jito Speed Bump

This article would be remiss to not mention Jito given that as of writing over 80% of the network stake utilizes the Jito validator client which adds some further complexity to how transactions typically reach the leader. The Jito-Solana validator client (Github) is a fork of the Agave client (Github), originally the Solana Labs client, that introduces an out-of-protocol blockspace auction and allows for validators to receive extra economic incentives in the form of tips.

‍

Given that a comprehensive examination of the Jito client is beyond the scope of this article, instead we will limit the scope of our analysis to how the Jito client affects the flow of normal transactions through Gulf Stream. Transactions have two possible routes —  they will flow from an RPC to a paired validator and then through to the current leader (the stake-weighted route), or they will forward directly to the leader (the open connection route). In either of these scenarios, if the leader is running the Jito-Solana client, these transactions will first be sent to the Jito-Relayer (Github), an open-source software that acts as a transaction proxy router. 

‍

Other network nodes have no notion of the Jito-Relayer; they will simply send transactions to whichever address and port configuration the leader has chosen to broadcast over the gossip network as their ingress_socket. The relayer delays transactions by 200 milliseconds before forwarding them to the leader, this “speed bump” mechanism slows the flow of incoming transaction messages and enables efficient discrete-time auctions. After 200 milliseconds the relayer optimistically releases the transactions regardless of the auction results.

Jito previously operated a canonical out-of-protocol mempool service, which has now been deprecated. When a Jito-Solana validator is the leader, searchers are still able to submit groups of atomically executed transactions, known as bundles, for other types of MEV transactions that do not rely on the mempool such as arbitrage transactions and liquidations.

‍

To learn more please refer to our Helius blog article here providing an introduction to Solana MEV.

‍

Conclusion

‍

In this article, we’ve examined various aspects of Solana’s Gulf Stream, including the QUIC networking protocol, Stake-weighted QoS, and the Jito validator setup. We’ve contrasted Gulf Stream with the more traditional mempool architecture and outlined the benefits of Solana’s approach in terms of greater efficiency and reduced latency. 

‍

There is no doubt that Gulf Stream will continue to evolve. The Solana protocol and its broader ecosystem are advancing rapidly, with significant upgrades anticipated. To give one example, Solana cofounder Anatoly Yakovenko has been a strong advocate for the implementation of multiple concurrent leaders. Multiple concurrent leaders would allow multiple nodes worldwide to simultaneously sequence user transactions, reducing latency and eliminating the need for a worst-case full global round trip before a transaction is added to the blockchain.

‍

Solana will not stand still and an exciting future awaits for the entire protocol including Gulf Stream. Consequently, developers should anticipate many further updates and optimizations to come.

‍

If you’ve read this far, thank you! Consider joining our community on Discord, following us on X, or subscribing to our mailing list below.

‍

Many thanks to Jacob Creech and 0xIchigo for reviewing earlier versions of this article.

‍

Additional Resources

‍

- Gulf Stream: Solana’s Mempool-less Transaction Forwarding Protocol - A. Yakovenko, Solana

- QUIC, a multiplexed transport over UDP - The Chromium Projects

- A Comprehensive Guide To HTTP/3 And QUIC - A. Monus

- Compilation of SWQoS Links - R. St.John, Anza

- A Guide to Stake-weighted Quality of Service on Solana - Solana Official Docs

- Solana Validator Education - Stake Weighted QoS - Solana YouTube Channel

- Jito Relayer Documentation - Jito Labs

- Asynchronous Execution, Endgame Architecture - A. Yakovenko, Solana

‍