December notches another great month for Argon validators. On average, our validators earned 58.03 HNT each which represents a 26.3% over-earn on all other validators which earned an average of 45.93 HNT (or 23.7% over-earn over all validators including Argon of 46.88 HNT).
While it is great that we continue to over-earn other validators, you may be wondering why the overall averages continue to be lower than the projected APY. With nearly 3,000 validators staked we’d expect an APY of 6% on average for the network. This past month we however only saw a network wide APY 5.62% (6.96% for Argon). The short of it is that PoCv11, released on December 13th, is causing longer election times than expected and in turn decreased earnings.
To go into technical detail: the changes enacted in PoCv11 are computationally expensive. Now, to calculate the rewards each epoch, the validators have to re-determine the proper reward-scale for each hex given which hotspots are online and which have been asserted. Since this is new functionality it has not yet been fully optimized. As it can take between 5-20 minutes to clear the reward block instead of the target 1 minute. While part of this lost time is offset by the block scaling between 50-70 seconds to hit a month average of 60 seconds, the network can only make up so much time each epoch.
While this unfortunately depresses earnings network-wide, I trust the Helium team is doing their utmost to get things back on track. It always bears reminding that this is still a relatively young project that is experiencing some expected growing pains following some exponential growth in the past year. At the beginning of 2021, there were less than 15k hotspots on the network and as we usher in 2022, we stand at 450k hotspots. This 30x growth of physical hotspots that provide IoT coverage is like nothing else in the cryptocurrency space. Overall, it’s hard to complain as Helium remains one of the top-performing crypto-assets of 2021 according to TradingView:
With light hotspots and 5G on the horizon, I’m excited to see what 2022 brings to the table and will do my best to continue running some of the most performant validators on the network.
November was another excellent month for Argon. On average, our 200+ validators earned 63.06 HNT which represents 34% more than other validators which earned 47.06 HNT (ie: 30% over average including Argon of 48.32 HNT).
However, 63.06 HNT is significantly less than last month (67.39 HNT) and there are a few factors at play:
October has 31 days, while November has 30
On November 16th, a chain halt of ~27 hours occurred, followed by a few smaller stalls the subsequent days totalling another 7 hours; 34 hours equates to roughly 1.4 days of lost earnings as well
Paradoxically, we believe these over-earn numbers partially stem from the chain outage which ultimately contributed to everyone earning less HNT this month. So while we’re proud of how we navigated the storm, we’d much rather things run smoothly and HNT income as a whole be higher.
The outage itself was a very intense period for the Helium blockchain and as an operator of a significant amount of validators, it was intense for us as well. We were in constant contact with the Helium team as we tried to move past the stall. New validator software was being released multiple times a day at points and it was critical that we deploy and validate the software to not only help the chain move forward but to make sure our fleet was positioned to earn optimally.
During this period, penalties were high. Generally, we see the network’s performance-to-tenure ratio around 1 to 1.5. During this period, the network average was over 3 and we consistently maintained half or less that. Recently, things have become stable and over the past few days the network average is around 1.64 and we stand at 0.47.
As you have likely noticed, over the past month, the price of HNT has grown quite substantially and seems to be settling in around the $40 mark. Due to this, our fee structure of $80 + 3%, which we initially set back when Argon launched in March and when HNT was still trading at about $10 and when I only had a rough sense of the infrastructure cost, is now outdated. Given the above, Argon will now only cost a 3% flat fee effective December.
November also included the first Helium vote on HIP 39 to redenominate the currency. This vote passed with 73.81% in favor (with a 66% majority required to pass). This means that one HNT will be 100,000 bones instead of 100,000,000 bones (for those familiar with Bitcoin, a bone is basically a satoshi, the smallest indivisible unit of the currency). Fundamentally, this changes nothing as the price will be split by 1,000 accordingly and the amount of bones for staking is maintained. HIP39 makes certain claims about behavioral economics which you may want to review for reasoning. And while this vote was passed on the 20th of November, given the time of year, I am guessing it won’t be implemented until the New Year.
This month has been our third month of steady and sustained over-performance compared to the network. While we were unable to match the high bar of 28% over-earn we set last month, this month, we clocked in an excellent 23% over-earn with an average per validator yield of 67.39 HNT compared to the 54.65 HNT for everybody else. This translates into a 7.93% APY vs 6.43% for other validators.
After some questions from users, I’d like to highlight just how cyclical rewards are when considering the performance of a single validator, even when you take the timeframe of a month. For example, our top 5 earners this month earned an average of 112 HNT while our lowest 5 earners made an average of 23.51 HNT. This variance might seem surprising, but looking at the penalty score and how it impacts “earning potential”, the picture is a little more balanced.
Almost all of those top earners were elected early in the month and then got re-elected for multiple cycles (5-7) throughout the month with their last ones occurring in the past few days (including one that bridged the month change). The lower earners on the other hand did not earn their first HNT until the middle of the month and subsequently got elected for fewer cycles (2-3).
When considering their penalty score, starting the month of November, the high earners have a high score average of 5.81 while the low earners have theirs at 0.82; for context, the Argon average is 3.78. As such, the low earners are carrying a low score setting them up for high “earning potential” in November. In fact, they have a good chance to be next months’ top earners. Already at time of writing, 3 out of 5 of low earners have already been elected since the month change while only the one validator that bridged the month change has been elected from the “top” group.
Last month, we were pleased that our client validators earned 19% more than other validators but we worried if it would be possible to keep that kind of momentum. September has blown away our expectations as our client validators over-earned by a staggering 28%.
We are also pleased to announce that validators under our management have gone from 138 to 211; over a 50% increase. We are proud that our pursuit for excellence is resonating with the market and provides us with scale to continue improving.
|Avg HNT earned||Annualized APY %||# of validators|
You’ll notice that we’ve greatly exceeded the projected average APY of 7.11% at the start of the month; it currently stands at 6.86%.
The actual over-earn numbers are subject to randomness, so it’s not the primary number we track in our day-to-day. We’ve been longtime proponents of monitoring “performance-to-tenure ratio” (P/T); that is to say, how many performance penalties have we taken for every tenure penalty. This metric serves as a leading indicator which helps us achieve our end-goal of over-earning.
A two-week window of performance-to-tenure ratio looks like this:
|P/T average||P/T σ||Score average|
As you can see, we take less than 18% the amount of penalties that the rest do. This is what helps the randomness eventually converge to extra earnings for you.
We are thrilled with this performance and hope it can continue. At this point, however, it comes down to what the rest of the network does. We are running multiple experiments to try to see if we can eke out better performance, but the biggest threat to our edge is the rest of the network catching up.
Finally, as mentioned in the previous update, HIP 28 was passed and enabled last month thus increasing validator rewards. Instead of being fixed, Consensus Group rewards now vary depending on the length of the epoch (up to a max). This has resulted in the average reward being roughly 5.5% higher than previously.
We’ve had an excellent second month at Argon! We have continued to improve and tweak our infrastructure. We've honed in on the hardware resourcing and architectures that perform best and deployed it on custom bare metal builds. This allows us to provide high performance servers while maintaining low rates for you. Throughout the month, we have been steadily out-earning all other validators by between 15 to 30%.
On average, the 136 validators that were staked with Argon all month earned 70.59 HNT for an APY of 8.47%. That’s over 19% better than the rest of the network which earned 59.28 HNT on average or 7.11% APY.
While the past month has shown excellent returns relative to the rest of the network, there are a few things to keep in mind moving forward.
First, we’ve seen an incredible growth in total validators over the last week and, as of writing, there are over 2,630 validators staked. This puts the projected average APY at 7.3%. That’s a significant drop from the start of August where the projected average APY was 8.7%.
Meanwhile, you’ll also notice that even with our strong performance relative to the network, our validators fell just short of 8.7%. This was due to longer epochs than expected which penalizes validator returns. Community member PaulVM drafted a Helium Improvement Proposal (HIP28) to address this and the proposal was recently ratified. In fact, we are expecting it to be live on-chain on September 10th. We anticipate this could improve earnings by 10-16%.
These two factors go in opposite directions, so we will see how it plays out! We will continue doing everything we can to run some of the best performing validators on the network.
Last week, I shared some early metrics on our relative performance to all other validators. Since then, the gap has only widened in the last 11 days as our validators have over-earned the rest of the network by 21.6% .
Since the POC interval was decreased to 300, we sustained an average performance to tenure ratio of 0.21 (over 1600+ rounds of CG). Meanwhile all other validators were averaging 0.94. This has led to an average of 12 rounds of CG per elected Argon validator as opposed to 9.8. Pro-rated monthly, that means 30 CG per month vs 24.65 or in other words, Argon validators are earning almost 13 extra HNT earned per month. This is precisely how you earn more with Argon.
We reached these numbers through hard work, constant monitoring, and tweaking over the last 4 weeks. After the first two weeks of launch, our numbers went from excellent to average. Average is not in our DNA and we are focused on providing excellence. Unlike some VaaS providers who focus on undercutting costs of the competition, our focus is on excellence and this means you make more.
Due to this, we doubled the compute resources and saw an immediate increase in performance, but the costs absorbed all of our margins from July. It was an easy decision because I treat your validators exactly as I treat my own. There’s no way I’m saving a few bucks on servers and giving up on HNT returns.
However, the last thing I wanted to do was have to raise our prices to make the business sustainable. About three weeks ago, I started investigating, testing, and working with various infrastructure providers.
I’m proud to say, as of three days ago, we’ve completed the migration to a new infrastructure provider with superior hardware AND lower cost.
Notably, we’ve abandoned VPS configurations in favor of custom bare metals builds. We’ve selected a server configuration which is uniquely tuned to our learnings from many months of testnet and the first 6 weeks of validators. Our performance has never been so strong and steady. Since Saturday (400+ rounds of CG) when we completed the migration, our average validator ratio is 0.1297 while the rest are at 0.965.
As always thanks for staking your validator(s) with us. I’ll continue doing my best to improve our edge.
After a first few excellent weeks at launch, the last couple weeks have been more average. Since then I've been optimizing our servers and have established a configuration that provides more consistent performance.
Due to this work, we've had an exceptionally good past 72 hours (since POC interval decrease). Since then our average perf to tenure ratio is 0.082. This is an Argon low which is even more impressive considering that all other validators are averaging a ratio of 1.327 during the same period. This low ratio is directly translating to longer average CG length (5.5 vs 4.74) and lower average score (5.59 vs 6.08) compared to others. In other terms, despite representing 6.70% of the elected validators, we've been in 7.69% of the rounds. Thats roughly a 14.7% over-earn during that period. Basically any way we look at the numbers, we've had a great 72 hours.
While these numbers may change with new releases, I just wanted to take a second to celebrate and let you know that I'm ever tinkering away to try to build and maintain the Argon Advantage. Thank you all for choosing us.
Yesterday Helium bumped up the CG size to 49 while also doubling the batch size, both of which had the potential to strain the validators. Due to this we have been monitoring the metrics closely. While the time-scale is short (45 elections), our Mean Performance to Tenure Ratio compared to the competition (0.1500 vs 1.618) is very statistically significant already (p-value of essentially 0). Basically, since the increase in strain, we're taking less penalty points than before while other validators are under strain and taking more.
The Performance to Tenure ratio (performance_penalty/tenure_penalty) as a measure is specifically designed to minimize noise from other random events. The Performance to Tenure Ratio basically says "for every tenure point, how many performance penalties did I incur?" This in turn will drive eligibility and longevity in CG.
The impact is that on average, we are lasting in CG more than the competition (4.513 vs 3.932). In other words, we are lasting longer in CG and earning 15% more than average. While the P-Value for this most recent period at CG=49 isn't yet statistically significant (p-value = 0.274), we believe this is due to the inherent noise from the randomness of being kicked from CG. Therefore, it is a lagging indicator, but we do believe over time as more data rolls in, the statistical significance will be clear as it was in the CG=40 period.
In short, there is a lot of randomness and probability to Consensus Earnings, we have proven that we already perform statistically better at CG=49 and at CG=40, our users were in more consensus groups in a statistically significant way. As more data rolls in, we expect this to be the same in CG=49.
After scaling CG size of 40 until CG size of 49 (Block Range: 912365 to 920403)
Argon Elected Nodes Count (percentage): 99 (0.7920) Other Elected Nodes Count (percentage): 1359 (0.7455) Argon Elections: 825 Others Elections: 9055 Argon DKG Hits: 0 Others DKG Hits: 6 Argon Performance to Tenure Ratio Mean (std): 0.4848 (1.349) Others Performance to Tenure Ratio Mean (std): 1.374 (2.561) Performance to Tenure Ratio Per Validators T-value: -5.837 Performance to Tenure Ratio Per Validators P-value: 0 Argon Elections Per Validators Mean (std): 8.333 (4.286) Others Elections Per Validators Mean (std): 6.663 (3.906) Elections Per Validators T-value: 3.766 Elections Per Validators P-value: 0.0003
After scaling to CG size 49 + doubling of txns Block Range: 920403 to current)
Argon Elected Nodes Count (percentage): 39 (0.3120) Other Elected Nodes Count (percentage): 516 (0.2830) Argon Elections: 176 Others Elections: 2029 Argon DKG Hits: 0 Others DKG Hits: 48 Argon Performance to Tenure Ratio Mean (std): 0.1500 (0.7343) Others Performance to Tenure Ratio Mean (std): 1.618 (2.832) Performance to Tenure Ratio Per Validators T-value: -8.566 Performance to Tenure Ratio Per Validators P-value: 0 Argon Elections Per Validators Mean (std): 4.513 (3.178) Others Elections Per Validators Mean (std): 3.932 (2.741) Elections Per Validators T-value: 1.110 Elections Per Validators P-value: 0.274
It’s been an exciting first 5 days of mainnet validators. Throughout this period, validators under Argon management have consistently outperformed their expected earnings by 10-30%. As of writing, we are outperforming our expected earnings by 17.5% since launch.
To try to understand whether Argon was getting lucky or whether this overperformance was deserved, I've spent the weekend digging into the data and trying to create a metric that better controls for luck and randomness and instead gives me a clear idea of what makes a validator "win".
Before I go into detail about this metric, it is worth establishing some context around the penalty score. Each validator has a penalty score. Penalties are only incurred when the validator is actively in the Consensus Group (CG). There are three types of penalties:
tenure penalty: completely unavoidable, your node accrues a penalty for every "round" of consensus that its part of
performance penalty: validators get hit with this due to poor performance, such as latencies and the ability to accomplish tasks. These are common but mitigated by good infrastructure.
DKG penalty: if a validator fails at allowing an election to happen. I think this would mostly happen if the node got knocked offline while it was elected, and thus prevented the subsequent election. This has been pretty rare and has never occured on Argon validators.
If your score is below a threshold, you will be randomly elected to a CG. As your score increases during CG, hopefully mostly due to tenure, you are more likely to be one of the 25% of CG validators evicted every round.
Due to this, as a VaaS provider, our job is to mitigate performance penalties and hopefully never incur a DKG penalty.
With all this in mind, I spent most of the weekend analyzing Server Penalty to Tenure Ratio
(Perf + DKG) / Tenure
This metric avoids noise from election luck, CG eviction luck, and variable CG size.
Since validators have been activated on mainnet, Argon is getting server penalties less than half as often as all other validators on the network. These low server penalties are translating into a much higher "elections per validator" score.
On average, our validators are in CG for an extra 1.23 rounds. That’s a 20% increase! The difference between 10% APY or 12% APY. We ran the same analysis with different date ranges (all time, since validator version 1.0.8, and since the increased gossip and short block length update) and reached similar results.
The above allows me to confidently say that our overperformance is not due to luck but instead due to our performance on the server penalty ratio. Helium’s election code is doing its job and rewarding Argon for running performant validators by keeping us in the Consensus Groups for longer. This in turn is translating into higher average returns for our users.
For the purpose of keeping the above article focused and direct, we did not include all of our research notes. We are including a few details below.
Argon’s non-elected percentage seems to be in line, or even slightly lower (0.7760 vs 0.7849), than the network average. This allowed us to rule out elections as a possible cause for the above increased earnings.
While Argon validators get server penalties far less than other validators, when they do get them, they are on average penalized roughly the same amount (1.0419 vs 1.0494).
Methods: We used Helium’s validators API to track all validators penalties. We only included validators that are currently staked. We also ran this analysis up to block 918588 (July 12th at 11:34 AM PT).
As we’ve been developing our infrastructure at Argon, we’ve been focused on reliability and scalability. We want to make sure that by the time validators are on mainnet, managing hundreds of validator nodes is old-hat to us.
We’ve been building tooling that enables us to monitor a fleet of Docker containers, check validator status, and run critical recovery operations easily (loading snapshots and manual p2p connections, for example).
In light of this, we saw the testnet reset on March 10 as an opportunity for us to put our tooling to the test. We made the decision to do so about 12 hours before the reset, so we quickly polished off a few features and got ready.
We started spinning up additional nodes about an hour before the scheduled reset. This was actually where we hit our first and biggest scaling hurdle: our Virtual Private Cloud (VPC) provider only allowed us to spin up about 90 instances before cutting us off. We had hoped for an even 100, but 90 would do.
Helium was kind enough to provide us with 1M Test Net Tokens (TNT), which meant we didn’t have to badger the faucet. With these in hand, we sent in a stake transaction every 5 seconds.
As you can see, after about five elections, we took nearly 75% of the seats.
Things were going quite well until around election 40: no new elections were happening. We checked in on the Discord and connected with the Helium team, providing details about our infrastructure and provisioning. Since we had such a large proportion of the Consensus Group under management, we wanted to make sure we hadn’t done something to break things. To help them diagnose and resolve the issue, we even provided access to a few of our consensus group nodes.
From our understanding, the peer-to-peer (p2p) seed nodes operated by Helium lacked a configuration reset and were causing p2p failures. This caused one of these early elections to drag out for a very long time and penalized those that happened to be part of the consensus group at that time. This is why we think we dropped to 25% here and lost our early edge from staking so many nodes so quickly.
Another way of looking at the data is by looking at “cumulative validator heartbeats” versus “cumulative earnings”. We chose validator heartbeats instead of validator count to account for validators that were no longer alive. It is our understanding that if all servers perform equally, then heartbeats and earnings should be roughly equal over time.
You can see that our initial ramp-up gave us an early-mover advantage. As normally only a fraction of the consensus group is replaced every election, we sustained inordinate returns as other validators joined the network, but only a fraction of the CG seats were available.
After we got ejected due to the long election, our returns returned roughly to the expected amount (where the purple line almost meets the blue line around election 50).
However, as things stabilized, we appeared to maintain a steady advantage. We are unsure whether this was dumb luck or due to our hardware or connections performing satisfactoraily.
By the end of the experiment, we had a solid 10% edge over what we would have expected, despite our early move advantage being relatively nullified around consensus group 50.
Overall, we interpret this as a very positive result for our infrastructure and tooling! We did, however, come up with many ideas on how to improve “descaling”, as tearing down 90 validators was not something we had actually thought much about until it was time.
Finally, we have resolved the scaling bottleneck with our VPC provider and will be able to spin up many more nodes before hitting the account limit!