ZK Compression Keynote: Breakpoint 2024
The following is an overview and key points from the ZK Compression keynote presentation and Q&A session at the Breakpoint 2024 conference, delivered by Swen Schaeferjohann, co-founder of Light Protocol, and Nicolas Pennie, co-founder of Helius. The presentation explores ZK compression—what it is, how it works, and, most importantly, why it’s crucial for the future of Solana.
Download the Presentation slides.
ZK Compression: Send it to Zero
One thing developers quickly discover when working on Solana is that although compute is relatively inexpensive, data storage costs are high. For example, given a SOL price of $150, creating 1,000 token accounts costs around $300, while creating a million accounts would cost $300,000. This makes scaling applications to a large user base financially prohibitive, as these costs increase with the price of SOL. Additionally, state growth is a challenge faced by all stateful blockchains. As new accounts are created, rent fees must be paid to maintain them, which is costly. Currently, Solana has over 500 million accounts, with approximately one million new accounts being added daily.
ZK compression offers solutions to these issues by providing:
- Thousand times cheaper accounts
- A solution for state growth
- A foundation for ZK compute on Solana
State Compression + ZK
To understand ZK compression, let's first break down state compression, which revolves around five key stages:
- Millions of accounts → “fingerprint”
- Store fingerprint on-chain
- Store account history on the Solana ledger
- Cache the latest account state with an indexer
- Verify compressed account state via the on-chain “fingerprint”
We begin by compressing millions of Solana accounts and hashing them into a compact "fingerprint." This fingerprint is stored in an on-chain account, while the full underlying account data remains accessible on Solana’s ledger. To ensure the validity of these off-chain accounts, we employ a prover mechanism that verifies their integrity using the on-chain fingerprint. The final step in this process utilizes Zero-Knowledge SNARKs, which power the proof system responsible for verifying the integrity of these accounts.
Why ZK Compression?
Several key features make ZK compression an ideal solution for state compression. For instance, compressed accounts behave similarly to standard Solana accounts, allowing developers to leverage their existing skills and development techniques with a shallow learning curve. This ensures a lower barrier to entry and makes ZK compression easy for developers to adopt.
The API exposed by the indexer closely mirrors existing RPC calls. For instance, the getAccountInfo method directly maps to getCompressedAccount, with this one-to-one mapping extending across the entire RPC API. This consistency simplifies the integration of ZK compression with existing Solana tools and workflows.
Compressed PDAs
A key feature of this system is its support for compressed program-derived addresses (PDAs). PDAs are deterministic account addresses that can be consistently derived using a specific program address and seed. Compressed PDAs help eliminate issues such as race conditions, address collisions, or accidentally creating multiple accounts for the same program address and seed combination. This enhancement is a major reason for transitioning from the existing SPL account compression system to ZK compression.
With full support for ZK compression, the potential impact on Solana is significant. PDAs and the ability to compose programs together allow for creating more complex architectures involving multiple programs and different PDAs—mirroring the familiar development experience of building traditional applications. This can be achieved at a fraction of the cost, reducing account expenses by up to a thousand times. This dramatic cost reduction will unlock new use cases previously inaccessible due to prohibitive costs.
Composability Improvements
Everything runs directly on Solana—it is neither an L2 nor a validium. Data availability is automatically provided by the Solana ledger, ensuring that execution is verifiable and fully composable.
Zero-knowledge proofs reduce proof sizes to a constant 128 bytes, freeing up more space in each transaction for other operations, resulting in easier composability and more flexibility for additional activities. Proof sizes are consistent regardless of how many accounts are being updated, read, or written in a single transaction or how many inclusions or exclusions need to be proven. This is essential because Solana's transaction sizes are inherently small (1.2 kilobytes). A different hash function for state Merkle trees was required to achieve fast proving times, which is why we couldn't use the original SPL account compression program.
Decompression
Another key feature is decompression, which prevents lock-in and enables seamless interoperability between “regular” and compressed accounts. Any Solana account can be compressed, allowing you to reclaim the rent cost. You can also decompress a compressed account anytime, enabling easy transitions between the two forms. For example, if you have a compressed token that needs to be swapped, you can simply decompress it and use it with Jupiter to execute the swap.
Decompression enables seamless transitions between hot and cold state. For instance, if you're developing a game with various compressed cards and items, you can decompress them during battles to allow rapid state changes, then compress them again afterward to save the final outcome, such as changes in hit points. There are some limitations regarding concurrency within a single tree during a slot. Compression should be avoided if you need to update the same account multiple times within a block. In such cases, it's better to permanently decompress that specific account, functioning as a regular on-chain account. For example, an AMM pool account would be better suited as one since its state needs to be updated regularly. The system is flexible enough to interact with compressed and decompressed accounts within the same transaction.
This fully generalized solution is not limited to a single program, business logic, or application. Solana accounts can be compressed with low costs and built-in indexing support. It's fully open-source, production-ready, and adaptable to various use cases.
Compressed Tokens
Compressed tokens are not directly compatible with exchanges and DeFi platforms unless they are supported. However, this can be easily resolved by decompressing the compressed token into a standard SPL token, which can then be used in DeFi and exchanges like any regular token. This flexibility ensures no lock-in, and users can move between compressed and regular tokens as needed. Compressed tokens are ideal for cold applications requiring less frequent interaction, and regular token accounts are better suited for hot state or frequently used assets. One of the primary advantages of this system is its ability to address the state growth problem by moving cold state data into compressed storage, preserving all the benefits while managing state more efficiently.
In our initial release, we introduced compressed tokens built using the same tools you would use for any compression-enabled application. The compressed token program is identical to the Solana token program, delivering the same user experience with a thousandfold cost reduction. Token extensions are expected to go live in early 2025.
With features like decompression, you can airdrop large volumes of tokens and quickly decompress them when needed for DeFi activities. While zero-knowledge (ZK) proofs are often seen as the Holy Grail of costly computation, we've optimized the process so that transfers resemble standard SPL token transfers, reducing proving times to milliseconds. This brings production-grade consumer ZK to the forefront, significantly cutting the cost of managing token accounts.
How Does ZK Compression Work?
There are five major components in the overall system:
- A Forest of State Trees
- Light System Program
- Compressed Token Program
- Forester Nodes
- Indexers (Photon)
A Forest of State Trees
First, we have state Merkle trees, which we refer to as the “forest of state Merkle trees.” This structure creates a unique cryptographic fingerprint of a set of accounts that we can store on-chain. For example, if you have four accounts, you can recursively hash them into a Merkle tree structure.
Above: a Merkle tree structure with four accounts
The final 32-byte hash at the top provides a cryptographic guarantee of the integrity of the underlying data for all accounts. This ensures that the state of any account can be easily verified as part of the Merkle tree by comparing it against the state root. We use the term "forest" here because there are multiple Merkle trees, each with its own root.
Comparison With Existing SPL Account Compression
One major enhancement of ZK compression from the existing SPL account compression program is the ability to prove that a piece of state is part of a given Merkle tree, ensuring efficient proof generation within Solana's transaction size limits. This allows us to prove the uniqueness of a given address by managing an address tree, similar to a state Merkle tree, but with an additional feature: proving inclusion in a leaf also proves the exclusion of a specific range of numbers. This enables handling a 248-bit address space with much smaller Merkle trees, ensuring the uniqueness of addresses.
M accounts can be proven for N trees while maintaining a constant 128-byte proof size and, therefore fitting well with Solana’s 1.2kb transaction size limits. Proof generation, which happens off-chain, is more expensive than the existing SPL account compression program. However, proof verification, which occurs on-chain, remains constant and is typically cheaper in most cases than the existing compression program.
Light System & Compressed Token Programs
The protocol involves two key programs. The Light System Program is an on-chain contract that mirrors the Solana System Program. It interacts with the Merkle tree, enforcing the standard Solana account model and verifying the uniqueness of PDAs. The Compressed Token Program replicates the SPL token program, enforcing the SPL data layout within the compressed account model.
Forester Nodes
Forester nodes are responsible for managing this light forest. When you update a compressed account, a new account state is appended to the tree, while the old state is zeroed out or nullified.
This approach has two key implications. First, any state update causes a change in the root as modifications propagate up the inverted Merkle tree to the top. Second, the trees gradually fill up and eventually reach capacity, which is where the forest of light trees comes in.
Forester nodes are responsible for maintaining the state root. They asynchronously update the state root and handle the rollover of full state trees. Operating a Forester node for your own state trees is permissionless and operates similarly to running an RPC, allowing anyone to manage their own state updates. As an app developer, you have a vested interest in maintaining your compressed state, which creates a natural incentive to ensure that your state trees are properly managed. You can either choose to run your own node for self-hosting or opt to pay a provider to handle it for you.
Photon
The Photon indexer is an open-source solution designed to track and manage updates, creations, mutations, and other events related to compressed accounts on the blockchain. It listens to on-chain activity and caches the current state of these accounts. Additionally, it generates cryptographic proofs that can be used for verification or data mutation.
The production-ready Photon indexer incorporates significant improvements based on insights gained from earlier versions of compression. Its goal is to be user-friendly and accessible, whether you're an individual developer, an enterprise, or an RPC provider.
Local development has been made significantly more straightforward. It features a one-click CLI tool that seamlessly integrates with your local setup. A dev-focused explorer is also available, providing a visual interface to view compressed accounts, track their changes, and examine transaction history.
Photon generates daily snapshots, speeding up development and deployment. Instead of reindexing from genesis, developers can start from a snapshot, reducing boot and startup times to 15 minutes or less. These snapshots also enhance replication, ensuring that if an RPC provider discontinues compression support, you can easily obtain a snapshot and run the indexer independently. The risk of data loss is minimized with snapshots, which can be stored in decentralized solutions like FileCoin.
With Photon, you have the flexibility to index only a subset of data, which significantly lowers hardware requirements and reduces database size. This can be done using SQLite with just a single CLI command. Alternatively, if you're an RPC provider, you have the option to index the entire dataset. Photon is available on any Helius plan, and you can also either run it independently or request that another provider offer it.
By Developers, For Developers
Our focus is entirely on developers, and we are actively working on three major improvements explicitly tailored to enhance the developer experience.
Web3.js for Compression: The SDK is designed to work similarly to Solana web3.js but with full compression support. To initiate a compressed token transfer, you first fetch your compressed token accounts, then use those accounts to obtain a validity proof from either your RPC or a dedicated prover node. Afterward, you build the instructions just as you would with the SPL token program, specifying what you want to send, the amount, the mint, and the recipient—creating a regular Solana transaction.
Full Local Development Setup: We offer a test validator pre-configured with everything necessary for local development, including all required programs, the Photon indexer, and a local prover node. This ensures a streamlined setup process for developers.
Anchor Macros: For those familiar with Solana program development using Anchor, our goal is to make working with compressed accounts as seamless as working with regular accounts. If you've written an Anchor program before, you'll immediately recognize the familiar development experience. Anchor macros are currently a work in progress.
Airship
Airship enables developers to leverage ZK compression today with a mass token airdrop tool that's both easy and cost-effective. It offers both a user interface and a CLI, and since it's fully open-source, developers can fork and modify it to suit their needs. As more developers airdrop compressed tokens, the ecosystem will begin adopting them across various applications. However, if immediate conversion is desired, developers can decompress the tokens immediately, ensuring Airship is a ready-to-use airdrop tool for regular tokens today.
The tool automatically supports airdrops to Solana mobile holders, specific token holders, or NFT collection holders. If you have a pre-generated list of recipients, you can also import a CSV file and specify the token and amount to be airdropped. Both the CLI and UI allow you to reset your airdrop from the last saved state if you experience issues like losing internet connectivity. This is particularly useful for larger airdrops, which might take 30 to 45 minutes when distributing tokens to a large list of addresses.
A New Design Space for Apps
ZK compression addresses the state growth problem by storing only the final root hash of all accounts in the active validator's memory. This approach effectively mitigates state growth issues. Moreover, ZK compression opens up new design possibilities for applications. Here are just a few potential use cases:
- SPL token compressors
- One billion meme coins
- Prediction markets for Twitter posts
- Identifier PDAs for nodes in DePin networks
- Verifiable reward calculation and distribution
- Trust-minimized brides
- ZK identity protocols
More ideas are provided in the documentation.
ZK compression is live on mainnet and devnet today! You can get started by reading our docs and working through the examples. Additionally, we're hosting a hackathon with up to $45,000 in prizes.
If you're interested in diving deeper into ZK, check out our previous Helius blog posts for an in-depth look at the fundamentals of zero-knowledge proofs and their applications on Solana.
