2 years of Lightwalletd Infra hosting & maintenance

Nighthawk Wallet Team has 20+ years of combined experience, with dev-ops led by Vamsi who has 10+ years experience in building & deploying scalable software with experience in handling peak demands, load balancing and triaging real time issues. Vamsi has helped setup, maintain and recently migrate Nighthawk's lightwalletd server securely, and with minimum downtime to an enterprise grade cloud service provider.

With this long term grant, we want to use our experience of deploying stable versions of lightwalletd to make Zcash services available to developers & applications in the Zcash ecosystem and maintain the services with security patches right from the platform to the endpoints.

Motivation and overview

The goal of this grant is to make available Zcash infrastructure as a test-net sandbox for new developers and stable main-net infrastructure for existing applications like Nighthawk Wallet and Zbay. This goal is in line with ZF's mission of enabling R&D and community involvement while opening up access to Zcash blockchain to other cryptocurrencies and oracles. This operation will serve the Zcash light wallet and services users.

Technical approach

Set-up 3 large 32GB firewall configured instances running on Ubuntu, Zcashd and Lightwalletd tagged releases.
1 instance configured for test-net and 2 load balanced instances configured for main-net use.
Buy a domain with pre-payment for 2 years for accessing lightwalletd infra.
Set-up a Status page to reflect the uptime of the light walled services.
Set-up a GitHub page with instructions to connect to lightwalletd.
Create documentation for community members to automate setting up similar instances via Ansible Playbooks.
Create an email to receive support requests and communications and aim for a 24 to 48hr turn-around by being available in Zcash Community Discord & Telegram groups.

Execution risks

Faults in the lightwalletd software require frequent patching to restore to working versions.
Excess server loads resulting in frequent peaks and requiring spinning up of new server instances and/or setup a proxy based load balancer.
Availability and Support from ECC Engineers to help debug and and fix issues real-time.

By having a status page and a connection to the community via social media and Zcash Form channels, we plan to keep up to date with the ongoing changes, planned & un-planned software upgrades.

Downsides

While Zcashd is a decentralized service, the Lightwalletd endpoint is centralized and may be subject to attacks and takedowns. We will have a contingency plan to switch to new-endpoints to triage issues that arise along the way. On the Availability front, issues in chain syncing in lightwalletd and Zcash upgrades need extra attention during specific times where the dev-ops may not be responsive even when Nighthawk Team members are present across North America & India time zones.

Doubts around integrity and privacy may arise and we will do our best to be transparent with any government/state requests that comes our way. Otherwise, we will adhere to no permanent logging policy (while having logging enabled for debugging purposes only).

On the stability front, we can choose the best SLA enterprise contracts, but even the best cloud providers have suffered massive unexpected downtimes which are out of our control. And continuous increased usage will require additional funding for servers.

Evaluation plan

Having a well funded infrastructure for Zcash will attract developers to test drive and create applications year round and deploy them knowing they can connect to stable services.

While we don't have year long uptime stats for Zcash services, we will be aiming for high 9s for the uptime that might be affected by software issues, maintenance turn-around time and data center issues.

Tasks and schedule

First month:

Sign long term server usage contracts.
Spin up instances and start configurations, syncing and backup security certs.
Buy and configure domain name to resolve to lightwalletd servers.
Verify end-points, test response chain info data to match with community servers.

Second month:

Work on load-balancing strategy for GRPC/HTTP services.
Set up documentation, landing pages and status page before making the lightwalled services available for public use.
Add 3rd party information request sharing policy & start maintaining warranty canary monthly.

Budget and justification

The operational costs are: $1,450/month, broken down as:

$900: $300 x 3 Large instances + Large Storage for indexed Zcash blockchain.
$300: 6 x $50/hr Dev-ops costs to support upkeep and monitor lighthttpd service, certs.
$150: Maintain, Update Warranty Canary & availability for audits
$100: Setting up domains, landing & status pages and community outreach.

Other-one time costs: Setting up a status page, buying a domain with 2 year pre-payment

Total: $1,450 * 24 months = $34,800