Zecwallet Infra costs

Applicant background

Zecwallet is a desktop and mobile Zcash wallet with full support for shielded and transparent transactions. Over the last month or so, traffic to the Zecwallet's LightwalletD servers has increased ~4x (probably correlated with the price of Zcash) as old and new users refresh their wallets.

This has put a large amount of strain on the Zecwallet server infra, and Zecwallet needs to add a new server to cope with the increased traffic without which I'm worried the infra will melt if there is another large influx of users.

Motivation and overview

Zecwallet's LightwalletD server served ~25 million blocks yesterday, with peak traffic exceeding 50,000 blocks/min (Note this is all server bandwidth usage, Zecwallet clients don't have any logging, so it is hard to estimate how many new / existing users of Zecwallet) which is approx 4x what it was at the start of Jan.

This grant is based on the previous maintenance grant, but simplified down to only the basics needed to keep the lights on.

It features 3 work items:

Add a new LightwalletD server in Asia

Add a new infra setup (LightwalletD server + zcashd fullnode *2) in a primary-backup setup in a Asia AWS region. This will need 2x t3.xlarge machines (Zecwallet's LightwalletD has higher memory and disk requirements, please see previous grant application )

This work item also includes the devops needed to setup the new region

Zecwallet Fullnode

To protect against any eclipsing attacks, the zcashd nodes powering the LightwalletD run with maxconnections=500. A side effect of this is that the nodes are connected to a lot of new Zecwallet Fullnode users who are syncing the blockchain from scratch, and it ends up serving a lot of bandwidth to new Zecwallet Fullnode users.

Because of this issue, some nodes are downloading vastly more data than needed, probably causing a large corresponding increase in Zecwallet's server bandwidth costs. This was fixed here

Need to investigate and patch this in the embedded Zecwallet Fullnode zcashd node, which should help reduce server traffic costs and ease some infra pressure. (Note, this is just a hunch that this issue is causing a large amount of bandwidth to be used, I need to investigate to be sure first)

Fix checkpoints

New Zecwallet users need to sync from a known 'checkpoint' to catch up to the latest block height on a lightclient. To help with decentralization, Zecwallet has so far embedded the checkpoint (which contains the block hash and the sapling commitment root) directly into Zecwallet's code so that the users don't have to trust the LightwalletD server.

A side effect of this is that Zecwallet needs regular releases to update the embedded checkpoint. The last Zecwallet release was a over a month ago, which causes new users to download ~50k blocks just to catch up to the chain tip. This is unnecessary, and can probably be optimized away.

Technical approach

For (1), we already have deployed infrastructure, so the technical approach is just to copy the deployment to a new AWS region and reconfigure the DNS, SSl etc…

AWS Server costs

This section describes funds that are spent directly on AWS, which hosts the Lightwallet infrastructure

  • USD 2,000 / month
  • 6x AWS large instances (across 2 regions) for zcashd and LightwalletD (primary + backup)
  • 1x AWS t2.small instance for load balancing, website and other static hosting
  • AWS bandwidth costs, including zcashd serving blocks
  • SSL certs, App store developer fees and other miscellaneous one-time costs

This grant requests funding for 6 months of server costs.

For (2), we'll investigate bandwidth usage on both client and server with the patch, measure bandwidth usage, and deploy a new release if it reduces bandwidth usage

For (3), we'll add a new checkpoint to the lightclients, make new desktop + mobile releases, and look at pulling in the work done by the ECC wallet team to use the sapling root delivered from zcashd -> ligthwalletD -> Zecwallet lightclient.

I estimate this is about 5 days worth of work (for all 3 items)

Execution risks

Since this is pretty small scope and the work items are well understood, the execution risk is minimal. There is a larger risk that Zecwallet's LightwalletD is centralized, and needs to become compatible (at least on the API level) with the ECC's lightwalletD so that users can easily switch between providers.

Downsides

This grant will help ease the infra/bandwidth costs a bit, and they also make progress in reducing costs in the long run, but more can be done to address centralization and decreasing costs.

Evaluation plan

Success of this project is mainly measured around:

  • Lightclient infrastructure uptime >99.5%

Budget and justification

Code and Devops :

  • USD 7,500 (@USD 187.5/hr)
  • USD 12,000 Cloud infrastructure (USD 2,000 * 6 months)

Tipping

Tips Received
???  
ZEC

Campaign

Started
15 days ago
Funding
$19,500 
Funded through  Zomg logo
Improve Infra
2
Fix Zecwallet Fullnode and Lite client

Fix Zecwallet Fullnode and Lite client

Estimate: February 2021
Reward: $9,750
The team was awarded $9,750 in ZEC on Feb 24th, 2021, 11:15 pm.
Improve bandwidth and resource usage.