Uploading Files to Shadow Drive on Solana

Overview

Solana's high cost and limited storage capacity make storing large amounts of data cost-prohibitive. Off-chain storage solutions like Shadow Drive, created by GenesysGo, have gained traction to overcome these limitations.

Shadow Drive provides a cost-efficient and performant option for storing large amounts of data on Solana. However, it requires a mainnet token, which you acquire by doing a SOL → SHDW swap.

In this tutorial, we will walk through uploading files to Shadow Drive using TypeScript.

‍Note: This test is only possible to run on mainnet, as SHDW does not have a devnet option currently available.

Prerequisites

Before getting started, make sure you have the following prerequisites installed:

• Node.js Installed

sh -c "$(curl -sSfL https://release.solana.com/v1.14.18/install)"

• Solana CLI Installed

npm install -g ts-node

• Backpack Wallet Extension

Setting up the Environment

Creating Project

1. Create a new folder from your terminal called Shadow.
2. Create a new file called upload.ts inside the Shadow folder.
3. Install the required packages:

npm install @shadow-drive/sdk @project-serum/anchor

     4. Generate a wallet using the following command:

solana-keygen grind --starts-with N:1

     5. Create a file named "helius.txt" inside the "Shadow" folder and add some content to it. In this example, we are just placing the following:

After completing these steps, your file directory structure should resemble the following:

1. Open the file named [walletAddress].json and copy the secret key. We will need to fund this wallet for the example.
2. Install the Backpack Wallet extension for your browser from the Chrome Web Store.
3. Open the Backpack Wallet extension and go to "Settings".

      4. In the settings menu, select "Wallets".

      5. Click on the "+" button at the top right to add a new wallet.

      6. Choose Solana as the wallet type and select "Import by Private key".

        7. Paste the contents of the wallet file into the import field and click "Import".

Once imported, copy the public key from the wallet. You can fund this test wallet with less than 0.03 SOL from another Solana wallet.

Getting SHDW Token

SHDW Drive requires a token called SHDW (SHDWyBxihqiCj6YekG2GUr7wqKLeLAMK1gHZck9pL6y) to create storage accounts. You can obtain SHDW tokens through a swap process using Jupiter on Solana.

1. Visit https://jup.ag/swap/SOL-SHDW to perform the SOL to SHDW token swap.
2. Enter the desired amount of SOL to swap for SHDW, and click "Swap". We only need less than 1 SHDW for this test (which equates to .01 SOL).

      3. Confirm the transaction. After the transaction is approved, the SHDW token balance will reflect in your wallet.

Steps to Build

1. Setting Connection to SHDW

In your upload.ts file, we'll import the necessary modules and establish a connection to the SHDW network. We'll create an asynchronous function named main to nest our code:

const anchor = require("@project-serum/anchor");
const { Connection, clusterApiUrl, Keypair } = require("@solana/web3.js");
const { ShdwDrive, ShadowFile } = require("@shadow-drive/sdk");
const key = require('./{wallet_address}.json');
const fs = require('fs');
const rpc = "https://rpc.helius.xyz/?api-key="
async function main() {
    let secretKey = Uint8Array.from(key);
    let keypair = Keypair.fromSecretKey(secretKey);
    const connection = new Connection(
        rpc , "confirmed"
    );    
		const wallet = new anchor.Wallet(keypair);
    const drive = await new ShdwDrive(connection, wallet).init();
		console.log(drive)
}
main();

Ensure to replace api-key in the rpc variable with a valid API key that you can obtain from the Helius Developer Portal. Also, substitute const key with the correct filename of your wallet we created before starting.

If you were to run the file now with ts-node upload.ts you will see the following produced in your terminal:

ShdwDrive {
  connection: Connection {
    _commitment: 'confirmed',
    _confirmTransactionInitialTimeout: undefined,
    _rpcEndpoint: 'https://rpc.helius.xyz/?api-key=',
...

2. Create a Storage Account

To create a storage account, add the following code inside the main function:

const newAccount = await drive.createStorageAccount("heliusDemo", "1MB", "v2");
console.log(newAccount);

Running the ts-node upload.ts command here will output the newly created storage account's ID and transaction signature.

{
  shdw_bucket: '5pzNpvGk3VK4XsqoCoJpqfXE7vioKkr7t4UEqbHkqnz1',
  transaction_signature: '4EFw2R7KNw45Rngg6MASqfhGDpeZvcaoQHApADhopD9WQxB6XMYBcshd7neyLFa6QtKZC5H9U5iTnggUuhdNDLfE'
}

Note: You may encounter an error at this step if the wallet is not funded with at least .01 SOL and 1 SHDW token.

3. Upload a File

Once we have a storage account, we can upload a file to it. Add the following code inside the main function:

const accts = await drive.getStorageAccounts("v2");
const fileBuff = fs.readFileSync("./helius.txt");
let acctPubKey = new anchor.web3.PublicKey(accts[0].publicKey);
const fileToUpload: ShadowFile = {
    name: "helius.txt",
    file: fileBuff,
    };
const uploadFile = await drive.uploadFile(acctPubKey, fileToUpload);
console.log(uploadFile);

The code above performs four primary actions:

1. It reads the local file helius.txt as a buffer.
2. Finds the storage account we created, and defines it as our account public key.
3. It creates a ShadowFile object with the file's name and data.
4. It uploads the file to the specified storage account using the uploadFile function.

We can now run the following to begin our upload:

ts-node upload.ts

Which produces the following in our console:

{
  finalized_locations: [
    'https://shdw-drive.genesysgo.net/5pzNpvGk3VK4XsqoCoJpqfXE7vioKkr7t4UEqbHkqnz1/helius.txt'
  ],
  message: '3UNhH2UMys8GxqSouVPtKLAd3rcP7m3oP8LwvwH8HSp3yChaYPSaAhjub5iV2i1LDvSU3JZ9ck322kt7QhRdbMhY',
  upload_errors: []
}

Now the file is uploaded! You can visit the finalized_location link produced, and make sure the file was uploaded correctly.

Notice the url for the file is https://shdw-drive.genesysgo.net/[storage_account_address]/[file_name]

Meaning any file uploaded to this storage account will share the initial URL, but the file name will change. You can also create separate storage accounts to store files.

You will also notice after checking your wallet: this operation only cost .00244 SHDW to complete.

Full Code

const anchor = require("@project-serum/anchor");
const { Connection, clusterApiUrl, Keypair } = require("@solana/web3.js");
const { ShdwDrive, ShadowFile } = require("@shadow-drive/sdk");
const key =  require('./wallet.json');
const fs = require('fs');

async function main() {
    let secretKey = Uint8Array.from(key);
    let keypair = Keypair.fromSecretKey(secretKey);
    const connection = new Connection(
        "https://rpc.helius.xyz/?api-key=", "confirmed"
    );    const wallet = new anchor.Wallet(keypair);
    const drive = await new ShdwDrive(connection, wallet).init(); 
		const newAccount = await drive.createStorageAccount("heliusDemo", "1MB", "v2");
		console.log(newAccount);  
    const accts = await drive.getStorageAccounts("v2");
    const fileBuff = fs.readFileSync("./helius.txt");
    let acctPubKey = new anchor.web3.PublicKey(accts[0].publicKey);
    const fileToUpload: typeof ShadowFile = {
    name: "helius.txt",
    file: fileBuff,
    };
    const uploadFile = await drive.uploadFile(acctPubKey, fileToUpload);
    console.log(uploadFile);

}
main();

Conclusion

That's it! You've successfully uploaded a file to Shadow Drive using TypeScript. This fundamental knowledge enables you to take advantage of SHDW's decentralized capabilities and provides a foundation for more advanced use cases.For instance, you could extend this tutorial by integrating file upload functionality into a web application or creating a service for decentralized file storage.Please check out the official docs for a further look into what you can do with the Shadow Drive SDK.

‍