[Referendum: MR10]: Expediting RT-2302 to bring Dynamic Fees to Moonbeam Sooner to deal with recent storage bloat

Summary: Abnormal behavior is causing significant performance degradation to the network. I propose expediting the deployment of RT 2302 to Moonriver by using the OpenGov whitelist track to ensure transactions on the network are properly priced based on how much gas is being used. This is a measure to slow the transactions that are clogging the network and increasing the size of the chain state.

(See Original MR Referendum 8 post for RT 2302 here.)

Recently, community members have observed some abnormal behavior on the Moonbeam Network including block saturation, rapid increase in storage size, degraded collator performance and delays in transactions being accepted into blocks.

Upon further analysis, increased usage originating from the Xen crypto project appears to be a significant contributor. The project uses a “proof of participation” concept for minting its XEN token, but because Moonbeam currently has a flat base fee (100 Gwei per transaction, regardless of network utilization); these transactions are not getting properly priced. On March 23rd, the volume of transactions originating from XEN significantly increased with the launch of XENFT ERC-721s; which allow the automation of creating XEN ERC-20 tokens in bulk.

The “proof of participation” mechanism is generating a lot of storage and saturating the network with tiny smart contract creations whose sole use is proving the participation of the user. What’s more, users will create new addresses in order to participate repeatedly without having to wait for the “waiting period” to pass. As a result, it is significantly increasing the size of the chain state. Prior to XEN stepping up its efforts, the chain state was growing by about 1Mb per day. Currently it’s growing at roughly 500Mb every single day. The Moonbeam chain state is currently sitting at 16GB in size.

This will rapidly start to create significant performance degradation - we’re already seeing block saturation and if this continues, synchronizing the chain will take weeks, producing blocks will be slower, fees will increase and it will take a long time for transactions to make it into a block, which is obviously bad for the entire ecosystem.

I want to be clear that I’m not offering an opinion on the merits of the philosophy behind the XEN project, XENFTs or their utility. But as it currently stands, without properly pricing these transactions, it presents a serious stability issue for the chain.

As it happens, before becoming aware of this issue, the Moonbeam dev team had already been working on Runtime 2302 (in fact, the deployment to Moonriver is currently in democracy), which would enable dynamic fees - this would ensure that these transactions would be properly priced based on how much gas is being used; and the chain storage can’t be so easily grown at an ultra low cost.

As a concerned community member and given the urgency of the situation, I am proposing that we accelerate the deployment of RT 2302 to Moonriver by using the OpenGov whitelist track to have it deployed as soon as possible. Then, it is proposed to fast track the deployment to Moonbeam, targeting a deployment after 1 day of being live in Moonbeam.

In terms of the storage problem itself, the dev team will need to do some additional work post RT 2302 to prevent abuse.

Moonbeam core developers continue to investigate other solutions to solve the underlying storage issue (which will only be partially mitigated by the dynamic fees). Any solutions would likely need to be made in a future runtime upgrade.

Here are some charts to put this into perspective:

8 Likes

I’ve read the proposal and fully support it.

To give just one anecdote on how this has impacted CertHum, one of our Moonbeam RPC nodes was low on disk space and we had it as planned work this weekend to upgrade. Unfortunately, yesterday afternoon that server hit 100% full and we had to perform emergency maintenance – our estimates of how long we could wait were skewed. Overall we have taken a lesson from the event, however, it showed us how the currently low fees can be exploited and have impacts not immediately apparent.

5 Likes

From an indexer’s perspective, every new address, user or contract, is another dimension to track and has real costs.

+1 for expediting to allign economics with fair use

4 Likes

I am in full support of this as well.

4 Likes

as a fellow collator I can only agree to the proposal and the points mentioned

1 Like

In speaking with various service providers and stake holders, it sounds as though we’ve reached the point where it’s becoming challenging to use some of their operational tools, etc.

It is being proposed that we shorten the delay between Moonriver and Moonbeam to so that RT2302 is deployed to Moonbeam approximately 24 hours after Moonriver.

1 Like

Moonriver Referendum is up:

2 Likes

Full support - this is trash and impacting the network for teams looking to do legit applications

1 Like

I am in favor of this proposal because I have observed that numerous community members txs are getting stuck, requiring them to wait for hours, or even days. this is because many of them do not comprehend the need to speed up the gas or use aggressive gas settings

I fully support the proposal.

I have updated the post to propose that RT2302 would be deployed to Moonbeam 1 day following Moonriver and invite community members to comment on this.

1 Like

I’m considering potential issues that may arise due to the varying network loads between Moonriver and Moonbeam. I’m uncertain whether testing something on Moonriver for one day would suffice to ensure everything is working correctly. Your thoughts and opinions are welcome

Upgrade on Moonriver has been enacted and will apply in about 20 minutes or so.

I understand your concern about the security of it.
The current proposed Runtime has been tested carefully like all the previous ones (which includes lot of rigourous tests but also deployment on multiple internal networks for each runtime).

We usually have a delay to ensure new features are well understood by the developers on Moonriver and to gather feedback before moving to Moonbeam.

For this runtime, it doesn’t include a lot of features impacting the developers. Some of them impact the community, like the dynamic gas fee, but this feature was already deployed to Alphanet 2 releases ago, and to Moonriver 1 release ago, so it was tested there.

No testing is good enough, not enough time is sufficient to get a 100% guarantee but we will never sacrifice the security of the chain for any other aspect.

I hope this answers your concerns

2 Likes

Hey @turrizt - definitely appreciate the concern here.

I understand that the pros and cons were carefully considered. Aside from the thorough testing that @Alan_PureStake mentioned, a 24 hour period will allow for several rounds to pass with smooth block production, etc. Smoke tests monitoring Moonriver should alert for any abnormal behavior.

The urgency here is that there is concern that certain thresholds could be hit that could prove much more difficult to recover from. The analogy of the Frog being slowly boiled alive was used and I think quite apt.

After careful consideration, the Moonbeam Council and Tech Committee have elected to move forward with the fast tracking and the community is urged to weigh in by voting:

1 Like

Thank you for taking the time to address my concerns and provide such a detailed answer, Alan! your explanation has been very helpful and makes a lot of sense, we truly appreciate it!

1 Like

I completely agree with you, Aaron. Thank you for providing such a detailed explanation, it’s much appreciated. in fact, i decided to ask the question on behalf of the entire community because i believe it’s important that everyone has a clear understanding of the situation. Thank you again for your quick and informative answer!

I’m in full support as well

read the proposal and fully supported

Solving issues related to explosive growth in network utilisation is first world problems :grinning:

Things looking bullish for moonbeam, well done