Last month, we were pleased that our client validators earned 19% more than other validators but we worried if it would be possible to keep that kind of momentum. September has blown away our expectations as our client validators over-earned by a staggering 28%.
We are also pleased to announce that validators under our management have gone from 138 to 211; over a 50% increase. We are proud that our pursuit for excellence is resonating with the market and provides us with scale to continue improving.
|Avg HNT earned||Annualized APY %||# of validators|
You’ll notice that we’ve greatly exceeded the projected average APY of 7.11% at the start of the month; it currently stands at 6.86%.
The actual over-earn numbers are subject to randomness, so it’s not the primary number we track in our day-to-day. We’ve been longtime proponents of monitoring “performance-to-tenure ratio” (P/T); that is to say, how many performance penalties have we taken for every tenure penalty. This metric serves as a leading indicator which helps us achieve our end-goal of over-earning.
A two-week window of performance-to-tenure ratio looks like this:
|P/T average||P/T σ||Score average|
As you can see, we take less than 18% the amount of penalties that the rest do. This is what helps the randomness eventually converge to extra earnings for you.
We are thrilled with this performance and hope it can continue. At this point, however, it comes down to what the rest of the network does. We are running multiple experiments to try to see if we can eke out better performance, but the biggest threat to our edge is the rest of the network catching up.
Finally, as mentioned in the previous update, HIP 28 was passed and enabled last month thus increasing validator rewards. Instead of being fixed, Consensus Group rewards now vary depending on the length of the epoch (up to a max). This has resulted in the average reward being roughly 5.5% higher than previously.
We’ve had an excellent second month at Argon! We have continued to improve and tweak our infrastructure. We've honed in on the hardware resourcing and architectures that perform best and deployed it on custom bare metal builds. This allows us to provide high performance servers while maintaining low rates for you. Throughout the month, we have been steadily out-earning all other validators by between 15 to 30%.
On average, the 136 validators that were staked with Argon all month earned 70.59 HNT for an APY of 8.47%. That’s over 19% better than the rest of the network which earned 59.28 HNT on average or 7.11% APY.
While the past month has shown excellent returns relative to the rest of the network, there are a few things to keep in mind moving forward.
First, we’ve seen an incredible growth in total validators over the last week and, as of writing, there are over 2,630 validators staked. This puts the projected average APY at 7.3%. That’s a significant drop from the start of August where the projected average APY was 8.7%.
Meanwhile, you’ll also notice that even with our strong performance relative to the network, our validators fell just short of 8.7%. This was due to longer epochs than expected which penalizes validator returns. Community member PaulVM drafted a Helium Improvement Proposal (HIP28) to address this and the proposal was recently ratified. In fact, we are expecting it to be live on-chain on September 10th. We anticipate this could improve earnings by 10-16%.
These two factors go in opposite directions, so we will see how it plays out! We will continue doing everything we can to run some of the best performing validators on the network.
Last week, I shared some early metrics on our relative performance to all other validators. Since then, the gap has only widened in the last 11 days as our validators have over-earned the rest of the network by 21.6% .
Since the POC interval was decreased to 300, we sustained an average performance to tenure ratio of 0.21 (over 1600+ rounds of CG). Meanwhile all other validators were averaging 0.94. This has led to an average of 12 rounds of CG per elected Argon validator as opposed to 9.8. Pro-rated monthly, that means 30 CG per month vs 24.65 or in other words, Argon validators are earning almost 13 extra HNT earned per month. This is precisely how you earn more with Argon.
We reached these numbers through hard work, constant monitoring, and tweaking over the last 4 weeks. After the first two weeks of launch, our numbers went from excellent to average. Average is not in our DNA and we are focused on providing excellence. Unlike some VaaS providers who focus on undercutting costs of the competition, our focus is on excellence and this means you make more.
Due to this, we doubled the compute resources and saw an immediate increase in performance, but the costs absorbed all of our margins from July. It was an easy decision because I treat your validators exactly as I treat my own. There’s no way I’m saving a few bucks on servers and giving up on HNT returns.
However, the last thing I wanted to do was have to raise our prices to make the business sustainable. About three weeks ago, I started investigating, testing, and working with various infrastructure providers.
I’m proud to say, as of three days ago, we’ve completed the migration to a new infrastructure provider with superior hardware AND lower cost.
Notably, we’ve abandoned VPS configurations in favor of custom bare metals builds. We’ve selected a server configuration which is uniquely tuned to our learnings from many months of testnet and the first 6 weeks of validators. Our performance has never been so strong and steady. Since Saturday (400+ rounds of CG) when we completed the migration, our average validator ratio is 0.1297 while the rest are at 0.965.
As always thanks for staking your validator(s) with us. I’ll continue doing my best to improve our edge.
After a first few excellent weeks at launch, the last couple weeks have been more average. Since then I've been optimizing our servers and have established a configuration that provides more consistent performance.
Due to this work, we've had an exceptionally good past 72 hours (since POC interval decrease). Since then our average perf to tenure ratio is 0.082. This is an Argon low which is even more impressive considering that all other validators are averaging a ratio of 1.327 during the same period. This low ratio is directly translating to longer average CG length (5.5 vs 4.74) and lower average score (5.59 vs 6.08) compared to others. In other terms, despite representing 6.70% of the elected validators, we've been in 7.69% of the rounds. Thats roughly a 14.7% over-earn during that period. Basically any way we look at the numbers, we've had a great 72 hours.
While these numbers may change with new releases, I just wanted to take a second to celebrate and let you know that I'm ever tinkering away to try to build and maintain the Argon Advantage. Thank you all for choosing us.
Yesterday Helium bumped up the CG size to 49 while also doubling the batch size, both of which had the potential to strain the validators. Due to this we have been monitoring the metrics closely. While the time-scale is short (45 elections), our Mean Performance to Tenure Ratio compared to the competition (0.1500 vs 1.618) is very statistically significant already (p-value of essentially 0). Basically, since the increase in strain, we're taking less penalty points than before while other validators are under strain and taking more.
The Performance to Tenure ratio (performance_penalty/tenure_penalty) as a measure is specifically designed to minimize noise from other random events. The Performance to Tenure Ratio basically says "for every tenure point, how many performance penalties did I incur?" This in turn will drive eligibility and longevity in CG.
The impact is that on average, we are lasting in CG more than the competition (4.513 vs 3.932). In other words, we are lasting longer in CG and earning 15% more than average. While the P-Value for this most recent period at CG=49 isn't yet statistically significant (p-value = 0.274), we believe this is due to the inherent noise from the randomness of being kicked from CG. Therefore, it is a lagging indicator, but we do believe over time as more data rolls in, the statistical significance will be clear as it was in the CG=40 period.
In short, there is a lot of randomness and probability to Consensus Earnings, we have proven that we already perform statistically better at CG=49 and at CG=40, our users were in more consensus groups in a statistically significant way. As more data rolls in, we expect this to be the same in CG=49.
After scaling CG size of 40 until CG size of 49 (Block Range: 912365 to 920403)
Argon Elected Nodes Count (percentage): 99 (0.7920) Other Elected Nodes Count (percentage): 1359 (0.7455) Argon Elections: 825 Others Elections: 9055 Argon DKG Hits: 0 Others DKG Hits: 6 Argon Performance to Tenure Ratio Mean (std): 0.4848 (1.349) Others Performance to Tenure Ratio Mean (std): 1.374 (2.561) Performance to Tenure Ratio Per Validators T-value: -5.837 Performance to Tenure Ratio Per Validators P-value: 0 Argon Elections Per Validators Mean (std): 8.333 (4.286) Others Elections Per Validators Mean (std): 6.663 (3.906) Elections Per Validators T-value: 3.766 Elections Per Validators P-value: 0.0003
After scaling to CG size 49 + doubling of txns Block Range: 920403 to current)
Argon Elected Nodes Count (percentage): 39 (0.3120) Other Elected Nodes Count (percentage): 516 (0.2830) Argon Elections: 176 Others Elections: 2029 Argon DKG Hits: 0 Others DKG Hits: 48 Argon Performance to Tenure Ratio Mean (std): 0.1500 (0.7343) Others Performance to Tenure Ratio Mean (std): 1.618 (2.832) Performance to Tenure Ratio Per Validators T-value: -8.566 Performance to Tenure Ratio Per Validators P-value: 0 Argon Elections Per Validators Mean (std): 4.513 (3.178) Others Elections Per Validators Mean (std): 3.932 (2.741) Elections Per Validators T-value: 1.110 Elections Per Validators P-value: 0.274
It’s been an exciting first 5 days of mainnet validators. Throughout this period, validators under Argon management have consistently outperformed their expected earnings by 10-30%. As of writing, we are outperforming our expected earnings by 17.5% since launch.
To try to understand whether Argon was getting lucky or whether this overperformance was deserved, I've spent the weekend digging into the data and trying to create a metric that better controls for luck and randomness and instead gives me a clear idea of what makes a validator "win".
Before I go into detail about this metric, it is worth establishing some context around the penalty score. Each validator has a penalty score. Penalties are only incurred when the validator is actively in the Consensus Group (CG). There are three types of penalties:
tenure penalty: completely unavoidable, your node accrues a penalty for every "round" of consensus that its part of
performance penalty: validators get hit with this due to poor performance, such as latencies and the ability to accomplish tasks. These are common but mitigated by good infrastructure.
DKG penalty: if a validator fails at allowing an election to happen. I think this would mostly happen if the node got knocked offline while it was elected, and thus prevented the subsequent election. This has been pretty rare and has never occured on Argon validators.
If your score is below a threshold, you will be randomly elected to a CG. As your score increases during CG, hopefully mostly due to tenure, you are more likely to be one of the 25% of CG validators evicted every round.
Due to this, as a VaaS provider, our job is to mitigate performance penalties and hopefully never incur a DKG penalty.
With all this in mind, I spent most of the weekend analyzing Server Penalty to Tenure Ratio
(Perf + DKG) / Tenure
This metric avoids noise from election luck, CG eviction luck, and variable CG size.
Since validators have been activated on mainnet, Argon is getting server penalties less than half as often as all other validators on the network. These low server penalties are translating into a much higher "elections per validator" score.
On average, our validators are in CG for an extra 1.23 rounds. That’s a 20% increase! The difference between 10% APY or 12% APY. We ran the same analysis with different date ranges (all time, since validator version 1.0.8, and since the increased gossip and short block length update) and reached similar results.
The above allows me to confidently say that our overperformance is not due to luck but instead due to our performance on the server penalty ratio. Helium’s election code is doing its job and rewarding Argon for running performant validators by keeping us in the Consensus Groups for longer. This in turn is translating into higher average returns for our users.
For the purpose of keeping the above article focused and direct, we did not include all of our research notes. We are including a few details below.
Argon’s non-elected percentage seems to be in line, or even slightly lower (0.7760 vs 0.7849), than the network average. This allowed us to rule out elections as a possible cause for the above increased earnings.
While Argon validators get server penalties far less than other validators, when they do get them, they are on average penalized roughly the same amount (1.0419 vs 1.0494).
Methods: We used Helium’s validators API to track all validators penalties. We only included validators that are currently staked. We also ran this analysis up to block 918588 (July 12th at 11:34 AM PT).
As we’ve been developing our infrastructure at Argon, we’ve been focused on reliability and scalability. We want to make sure that by the time validators are on mainnet, managing hundreds of validator nodes is old-hat to us.
We’ve been building tooling that enables us to monitor a fleet of Docker containers, check validator status, and run critical recovery operations easily (loading snapshots and manual p2p connections, for example).
In light of this, we saw the testnet reset on March 10 as an opportunity for us to put our tooling to the test. We made the decision to do so about 12 hours before the reset, so we quickly polished off a few features and got ready.
We started spinning up additional nodes about an hour before the scheduled reset. This was actually where we hit our first and biggest scaling hurdle: our Virtual Private Cloud (VPC) provider only allowed us to spin up about 90 instances before cutting us off. We had hoped for an even 100, but 90 would do.
Helium was kind enough to provide us with 1M Test Net Tokens (TNT), which meant we didn’t have to badger the faucet. With these in hand, we sent in a stake transaction every 5 seconds.
As you can see, after about five elections, we took nearly 75% of the seats.
Things were going quite well until around election 40: no new elections were happening. We checked in on the Discord and connected with the Helium team, providing details about our infrastructure and provisioning. Since we had such a large proportion of the Consensus Group under management, we wanted to make sure we hadn’t done something to break things. To help them diagnose and resolve the issue, we even provided access to a few of our consensus group nodes.
From our understanding, the peer-to-peer (p2p) seed nodes operated by Helium lacked a configuration reset and were causing p2p failures. This caused one of these early elections to drag out for a very long time and penalized those that happened to be part of the consensus group at that time. This is why we think we dropped to 25% here and lost our early edge from staking so many nodes so quickly.
Another way of looking at the data is by looking at “cumulative validator heartbeats” versus “cumulative earnings”. We chose validator heartbeats instead of validator count to account for validators that were no longer alive. It is our understanding that if all servers perform equally, then heartbeats and earnings should be roughly equal over time.
You can see that our initial ramp-up gave us an early-mover advantage. As normally only a fraction of the consensus group is replaced every election, we sustained inordinate returns as other validators joined the network, but only a fraction of the CG seats were available.
After we got ejected due to the long election, our returns returned roughly to the expected amount (where the purple line almost meets the blue line around election 50).
However, as things stabilized, we appeared to maintain a steady advantage. We are unsure whether this was dumb luck or due to our hardware or connections performing satisfactoraily.
By the end of the experiment, we had a solid 10% edge over what we would have expected, despite our early move advantage being relatively nullified around consensus group 50.
Overall, we interpret this as a very positive result for our infrastructure and tooling! We did, however, come up with many ideas on how to improve “descaling”, as tearing down 90 validators was not something we had actually thought much about until it was time.
Finally, we have resolved the scaling bottleneck with our VPC provider and will be able to spin up many more nodes before hitting the account limit!