All You Need to Know About Solana's v1.17 Update

What’s this Article About?

Solana’s network has reached a successful milestone with the super-majority adoption of 1.17, the latest version of the Solana Labs validator client. Following the recent network outage, validators restarted using version 1.17.20. At the time of writing, ~68.6% of validators are running version 1.17.21, and ~31.3% are running version 1.17.20. This new version introduces a suite of enhancements designed to bolster the network’s efficiency, scalability, and use cases. From pioneering advancements in zero-knowledge proofs to refining the Gossip protocol, v1.17 marks a pivotal step in Solana’s ongoing development.

This article includes everything you need to know about the version 1.17 update to the Solana Labs validator client. We’ll explore the testing rigor behind v1.17, the recent network outage, and the new features implemented with this update.

How was v1.17 Tested?

v1.17 has been running on testnet since October 3, 2023. It has been regularly stress-tested with high transaction loads. Solana Labs also deployed a few mainnet-beta canary nodes running v1.17 to monitor the version’s stability under real-world conditions. They have been stable for the past several months. These canary nodes’ past activity and progress can be seen by visiting the #canaries-monitoring channel in the Solana Tech Discord. Additionally, starting December 4, 2023, a small subset of volunteer mainnet-beta nodes upgraded to v1.17.

Multiple runtime fuzzers were developed to execute partially randomized transactions to catch edge cases or infrequent race conditions. These transactions were executed across multiple versions to ensure consistent performance. v1.17 has been audited several times by external auditors, which will be posted to the Solana Security Audits GitHub repository as they become available.

February Outage

On February 6, 2024, at 9:53 UTC, mainnet-beta experienced an outage that temporarily halted the finalization of blocks. This disrupted network activity for approximately five hours until consensus resumed at 14:55 UTC. This outage can be traced to a bug associated with how the network compiled and cached a program’s code for execution, specifically affecting older versions of program loaders.

The root cause stemmed from how validators handled the Just-in-time (JIT) compiled output of frequently used programs. A new caching system intended to optimize this process inadvertently introduced the fateful bug. This new system could enter an infinite recompilation loop for certain legacy programs. This stalled Solana’s consensus mechanism as most validators encountered this issue and could not continue processing transactions.

The root cause was identified quickly, as the bug shared a number of similarities with the recent devnet outage. 1.17.20 was modified to address the issue directly, and validators coordinated a network restart. The fix was twofold: a short-term solution that disabled the ability to trigger the infinite loop by deprecating the two legacy loaders that could trigger the loop and a more comprehensive adjustment to the new program caching system to prevent future similar issues.

The official report, originally published by Anza, can be found here.

ZK Token Proof Program

The ZK Token Proof Program was slated to be released with the 1.16 update. However, its activation was delayed due to extensive auditing. The Feature Gate Activation Schedule has the program marked as Pending Testnet Activation with a floor of 1.17.12.

Confidential Transfers

With the activation of the ZK Token Proof Program, Confidential Transfers will finally become available. Confidential Transfers use zero-knowledge proofs to encrypt token balances and transfer amounts for SPL tokens. The overall goal here is confidentiality rather than anonymity. Homomorphic encryption allows computations to be performed on the encrypted data without decrypting it. For example, balances can be added or subtracted without decrypting and re-encrypting the specified amounts in these operations. Thus, these computations are encrypted the same way they would be if they were applied in plaintext.

Confidential Transfers use Twisted ElGamal Encryption and Sigma Protocols for secure, private transactions without revealing sensitive information.

As an aside:

Twisted ElGamal Encryption is a simple variant of the standard ElGamal encryption scheme where a ciphertext is divided into a Pedersen commitment of the encrypted message and a decryption handle to enable the performance of hidden mathematical operations on ciphertext
Sigma Protocols are used to validate Confidential Transfers. They are a special class of zero-knowledge proofs where one party (i.e., the prover) can prove to another party (i.e., the verifier) that they know certain information without revealing it

Confidential Transfers only allow the account holding the decryption key to view their encrypted balance. The Global Auditor System is implemented for situations that require a third-party review (e.g., compliance checks or auditing). This system allows account holders to grant selective read access to specific accounts through separate decryption keys and incorporates an “auditor encryption key” for mints to facilitate secure audits.

Transactions use encrypted parameters for sender, receiver, and auditor amounts alongside proofs to ensure privacy and integrity. Account balances are split into “Pending” and “Available” to prevent front-running attacks. If not, a malicious user could send tokens to an account to invalidate proofs generated using the encrypted balance.

It is important to note that Confidential Transfers require a new key pair to be used.

Command Line Support

Command Line Interface (CLI) support for Confidential Transfers via the spl-token crate is also available. Once activated, the create-token command has been extended to include the –enable-confidential-transfers auto flag. This allows users to mint tokens with the Confidential Transfers extension enabled. There are also a number of other useful commands, such as:

configure-confidential-transfer-account - configures a pre-existing account for Confidential Transfers. Only the account owner may configure Confidential Transfers for the account
deposit-confidential-tokens - deposits tokens from a non-confidential account to a confidential account. Note that the deposited tokens will no longer exist in the account’s non-confidential balance as they’ve completely moved into the confidential balance
apply-pending-balance - moves a balance from “Pending” to “Available.” This is necessary as whenever an account receives confidential tokens from a transfer or deposit as the balance appears in the account’s “Pending” balance. Users therefore do not have immediate access to the funds and need to apply the pending balance
transfer (with the –confidential flag enabled) - transfers tokens to another account configured for confidential transfers. Note that this operation may take longer than a regular token transfer as it requires multiple dependent transactions
withdraw-confidential-tokens - withdraws tokens from an account’s confidential balance to their non-confidential balance. Ensure any pending balance has been applied before running this command to withdraw all expected tokens
update-confidential-transfer-settings - updates the confidential transfer configuration for a given token mint. It provides the option to automatically set policies for approving confidential transfers (i.e., the –aprove-policy flag) and setting an auditor’s public key (i.e., the –auditor-pubkey flag). It also has additional flags for specifying a blockhash, the Confidential Transfer authority, configuration files, fee-payer details, the JSON RPC URL, nonce account details, output format, and the token program ID

Compatibility

Note that the following extension combinations either do not work or do not make sense to combine with Confidential Transfers:

Confidential Transfer + Non-transferable
Confidential Transfer + fees (will not work until 1.18)
Confidential Transfer + Transfer Hooks (this is because these transfers can only see source or destination accounts and, therefore, cannot act on the amount transferred)

Audits

Confidential Transfers, and the Token-2022 Program more generally, have been audited extensively by firms such as Halborn, Zellic, Trail of Bits, NCC Group, and OtterSec (twice - the first audit focused on Token-2022 whereas the second audit focused on Confidential Transfers specifically).

Poseidon Syscalls

Poseidon is a zero-knowledge proof friendly family of hash functions. Poseidon hashing functions are used by most ZK-based blockchain projects, including Zcash, Mina, and Solana’s Light Protocol. Currently, computing Poseidon hashes on Solana is too expensive to be done in one transaction. Poseidon syscalls are poised to change this. They are currently slated to release with v1.17.5 pending testnet activation.

Layman’s Explanation

Imagine using an advanced calculator that efficiently solves specific types of puzzles used for secure communications. This calculator works by taking in pieces of information, chopping them up, and mixing them together in a unique way. This information is mixed up so that the original content is completely hidden but still verifiable.

This mixing process uses a special method that adds numbers and raises them to certain exponents that are part of a specific set of numbers outlined ahead of time. This method ensures that the mixing is done thoroughly and consistently every time.

Poseidon does exactly the same thing as this advanced calculator. These types of hash functions are really good at creating zero-knowledge circuits. Circuits are essentially a collection of mathematical operations. They mathematically represent one party (the prover) proving to another party (the verifier) that they know certain information without revealing it. In general, think of it as proving you know what’s inside a sealed box without actually opening it. You can prove this to the person who sealed the box with a series of hits or steps. These hits/steps are designed so they don’t give away any details about what’s in the box or how to open the box, and they can only be understood if you’ve opened the box.

This is useful for blockchains, as keeping transactions private while being able to verify their authenticity is a huge deal.

ZK Friendly Characteristics

Poseidon hash functions are considered zero-knowledge friendly for a few reasons. Namely,

Poseidon is designed to perform arithmetic operations common in zero-knowledge proof computations (i.e., addition, multiplication, and exponentiation) efficiently
Zero-knowledge proof systems require turning computational logic into a cryptographic proof. Poseidon’s design results in a lower circuit complexity than other hashing functions because of its arithmetic-friendly design, optimized S-box, customizable parameters, and low number of rounds (i.e., a sequence of operations applied to the input data or the internal state of the hash function, which is done iteratively). This means that it takes fewer steps to generate a proof for a given piece of data
Poseidon hash functions are based on a sponge construction. That is, Poseidon is designed using a class of algorithms that take in a sequence of bits of any length and produce an output of any length. This makes it extremely easy to integrate Poseidon hash functions into different zero-knowledge applications.

Comparison to Traditional Hashing Functions

Poseidon’s computational efficiency tailored towards zero-knowledge proofs offers a distinct advantage over more generalized, computationally intensive operations made by traditional hashing functions, such as SHA-256.

While secure and reliable, traditional hashing functions often lead to larger, more complex circuits within zero-knowledge proofs. This is because these hashing functions were not originally designed with the specific constraints of zero-knowledge proof systems in mind. On blockchains, zero-knowledge proofs operate within finite fields for their computations. A finite field is a set of numbers where all arithmetic operations (addition, subtraction, multiplication, and division) are performed modulo a prime number. This causes them to “wrap around” within the set and ensures that every operation stays within this set of numbers.

Traditional hashing functions, such as SHA-256, rely heavily on bitwise operations and predetermined sequences of operations. These functionalities are not directly compatible with the arithmetic operations used in finite fields. Implementing these operations within the context of a finite field would require additional steps, ultimately increasing the complexity of the circuit.

Modular Arithmetic - Clock arithmetic — Source: *BTC#: Modular Arithmetic* by Bernard Darnton

Moreover, traditional hashing functions typically use modular arithmetic based on powers of 2. This differs from the modular arithmetic of finite fields used in zero-knowledge proofs, which are often based on prime numbers. This incompatibility would require even more steps, increasing the circuit's size and complexity.

The goal of constructing a zero-knowledge proof is to create a mathematical representation of computational logic that proves knowledge of certain information without revealing it. The efficient and simple representation of this knowledge is crucial for the scalability of these proofs. Poseidon family hashing functions directly address these needs using efficient operations within finite fields, significantly reducing the required circuit creation steps. This results in a smaller, less complex circuit. Traditional hashing functions are not tailored to address the needs of building circuits directly - they are designed to be general purpose.

Integrating Poseidon syscalls in v1.17 represents a shift towards using specialized cryptographic tools for the streamlined generation and validation of zero-knowledge proofs on Solana. This results in faster transaction processing, reduced costs, and enhanced scalability for zero-knowledge computations. Coupled with Poseidon’s flexibility and customizability, generating and validating zero-knowledge proofs on Solana became much easier.

Specific Implementation

v1.17 introduces the sol_poseidon syscall - a system call that takes a 2D-byte slice input and calculates the corresponding Poseidon hash as its output. It uses the BN254 curve and takes in the following Poseidon parameters:

x^5 S-boxes (i.e., substitution boxes)
Inputs where 1 ≤ n ≤ 12
Width where 2 ≤ t ≤ 13
8 full rounds and partial rounds depending on t: [56, 57, 56, 60, 60, 63, 64, 63, 60, 66, 60, 65]

Computing these Poseidon hashes will be done with the light-posiedon crate, which is audited and compatible with Circom.

Note, in the following section, we discuss the addition of alt_bn128 syscalls. BN254 is colloquially called BN128 (for the bits of security) and alt_bn128 or alt_bn_128. Here, we are referring to the same curve.

alt_bn128 syscalls

v1.16 proposed enhanced runtime support for zero-knowledge computations, specifically 128-bit elliptic curve operations. alt_bn128 syscalls, which are crucial for generating proofs efficiently, were supposed to be released with v1.16. However, it was delayed. The Feature Gate Activation Schedule currently has alt_bn128 syscalls slated for v1.17.15 pending mainnet-beta activation and alt_bn128 compression slated for v1.17.15 pending testnet activation.

As an aside for those more interested, alt_bn128 refers to the implementation of the Barreto-Naehrig (BN-128) elliptic curve. It is a specific pairing-friendly elliptic curve that allows for efficient zk-SNARKs (Zero-Knowledge Succinct Non-Interactive Argument of Knowledge). It is considered “pairing-friendly” as it allows certain zero-knowledge computations and proofs to be done more efficiently. In Solana, alt_bn128 syscalls enable programs to leverage this curve for streamlined zero-knowledge proof verification, enhancing security and privacy. Adding alt_bn128 g1 and g2 syscalls helps facilitate Groth16 proof compression. This significantly reduces the space required per proof, optimizing space efficiently for Solana programs.

The introduction of alt_bn128 syscalls helps narrow the compatibility gap between Solana and Solidity-based contracts that rely on precompiled contracts for elliptic curve operations specified in EIP-196, EIP-197, and EIP-198. These operations (bn256Add, bn256ScalarMult, bn256Pairing) facilitate zk-SNARK verification within Ethereum’s gas limits. Solidity contracts relying on these elliptic curve operations may now find it easier to transition to or even interoperate with Solana.

Gossip Improvements

v1.17 improves Gossip message propagation efficiency by enhancing push message propagation and reducing the reliance on pull requests. Streamlining the gossip protocol's operations helps reduce resource usage for consensus validators.

For context, Solana’s Gossip Service is essential for exchanging information among validators. This includes information such as ledger heights, contact details, and consensus votes. It uses “push” and “pull” messages to share and verify information across the network. This messaging system ensures all nodes stay synchronized.

Traditionally, AccountsHashVerifier pushes its account hashes to gossip. However, no network component ever pulls this data making this process redundant. Historical operations, such as comparing account hashes from gossip to known validators’ values, have been removed with the introduction of EpochAccountsHash. The getHealth RPC method was also rewritten to no longer rely on account hashes from gossip - we’ll dive into this in the following section. Thus, as of v1.17, nothing pulls account hashes from gossip. AccountsHashVerifier has been modified to stop pushing accounts to gossip, and the functions responsible for pushing and pulling account hashes from gossip have also been removed.

getHealth

In the past, the getHealth RPC call could reflect the wrong state for a given node. This was due to a discrepancy between the solana catchup CLI command and the getHealth RPC call, where a node could appear caught up and behind simultaneously. This could result in healthy nodes being incorrectly marked as unhealthy and potentially removed from RPC pools.

The health was originally determined by comparing a local accounts hash slot published in gossip against those of other nodes, using a default comparison value of 100 slots. This could be imprecise, especially for nodes configured with values larger than 100 slots. getHealth has been rewritten to use the latest optimistically confirmed slot from the cluster (i.e., the last slot all validators have processed and confirmed by a super-majority but is not yet finalized). This offers a more precise comparison as the cluster confirmed slot can be compared against the latest optimistically confirmed bank to determine how far behind the nide is. The change offers a more granular check, reduces the chances of false negatives (i.e., healthy nodes marked as unhealthy), and provides immunity to issues affecting known validators preventing any domino effects.

The –skip-health-check flag has also been added for the wait-for-restart-window and exit commands to address the issues with getHealth. This allows validators to skip checking whether a node is healthy.

QUIC

v1.17 introduces the ability to broadcast shreds and perform repairs using QUIC. The Turbine and repair QUIC endpoints are currently disabled since they are unnecessary until testnet has fully migrated to QUIC. However, this establishes the foundation for migrating these protocols to QUIC. Several PRs have been merged to introduce this underlying functionality, including:

Asynchronous TPU Connections

v1.17 introduces asynchronous TPU client connections, significantly improving the connection cache mechanism. This update is designed to reduce transaction latency by enabling background connection setups with a default connection pool size of four. Asynchronous TPU client connections ensure smoother transaction processing without the necessary wait times associated with synchronous connections.

Streamlined Validator Startup and Snapshot Format Update

v1.17 streamlines the validator startup process by introducing a new flag and updating the supported snapshot file formats. This results in faster validator startup times, which is significant as it contributes to the network’s resilience by reducing downtime.

v1.17 introduces a new –use-snapshot-archives-at-startup flag. This allows validators to expedite the startup process by choosing between local snapshots, the on-disk local state, or automatically opting for the most recent of the two. This flag eliminates the need to process snapshots when the on-disk state is newer, resulting in faster restart times.

Previously, Solana supported various compression formats for snapshots - archive formats included bz2, gzip, zstd, lz4, tar, and none. However, the latest update narrows the choices to either zstd and lz4 to optimize for efficiency and reduce support complexity. The other formats have been deprecated for the –snapshot-archive-format argument, although validators can still read pre-existing snapshots in these formats to ensure backwards compatibility. This inherently simplifies the command line interface for solana-validator and solana-ledger-tool.

Conclusion

With the numerous feature implementations and the swift resolution to the recent network outage, Solana’s v1.17 update is a leap forward. The update ushers in unprecedented zero-knowledge support and capabilities with the release of the ZK Token Program, Poseidon syscalls, and alt_bn128 syscalls. Coupled with improvements to validators and network efficiency, this update lays a solid foundation for the next version update. This update is smaller than v1.16 and aligns with the targeted release of a new version once every three months. The release schedule for 1.18 can be found here.

If you’ve read this far, thank you, anon! Be sure to enter your email address below so you’ll never miss an update about what’s new on Solana. Ready to dive deeper? Explore the latest articles on the Helius blog and continue your Solana journey, today.