MEMO Decentralized Storage Fully Explained: Enabling Cross-Chain Integration of Ethereum, Solana, and Other Ecologies
1. Preface
The blockchain explosion has lasted for three years, with the increasing scale of the Ether ecosystem, more and more data generated, the pressure of processing transactions on the chain has increased abruptly, and some of today’s Layer2 programs (OP/ZK rollup) have become a very good way to expand the capacity at the moment. However, the easing of the pressure on the chain is accompanied by the continuous expansion of developers and transaction volume, the size of the rollup is getting bigger and bigger, and the amount of data that needs to be uploaded is also rising at the same time. At this time, the problem arises that, on the one hand, the burden on the chain of Ether is increasing dramatically, while on the other hand, the cost of Layer2 is also increasing gradually. In order to alleviate the data storage pressure of Ether and Layer2, and at the same time ensure data availability, MEMO decentralized storage network gives the answer.
2. Introduction to MEMO
2.1 EVM-compatible MEMO decentralized storage network
MEMO decentralized storage network was constructed in 2017, at first it was constructed as a blockchain bottom layer storage facility, with the help of its own developed data middleware protocol, it constructed a dApp container that can be compatible with EVM, FVM and other mainstream environments at the same time, which is mainly used for storing all kinds of data generated by the blockchain itself as well as applications on the chain, such as text, images and audio-visuals, etc., and it supports permanent storage just as the Arweave supports permanent storage.
2.2 Reducing storage pressure in the Layer2 scheme
Later, with the big explosion of Ethernet ecosystem, the actual transaction volume is much larger than the processing capacity of the chain, and Layer2 also started to develop, OP/ZK rollup gradually became the main solution of Layer2, but with the rising data volume, the pressure of data storage of Rollup also came up, thanks to the decentralized node network that has already begun to take shape, MEMO has created a new generation of storage + data MEMO has created a new generation of storage+data availability parallel expansion solution. The huge space not only has the ability to store a large amount of data, but also can be specialized in storing Rollup’s transaction data at a lower cost, ensuring that the transaction data is stored in MEMO’s network and allowing the nodes on the chain to easily access and view the transaction data.
On the one hand, MEMO builds Rollup data storage solution based on decentralized storage system, on the other hand, MEMO also realizes the ecological integration of cross-chain storage and interaction based on data middleware protocol.
The following will detail the technical architecture, role information, storage technology, storage cost, and application scenarios of the MEMO decentralized storage network, so that you can understand exactly the technical route of the entire MEMO ecosystem.
3. Network Architecture
3.1 Information about each role and storage process
3.1.1 Role information
In the organizational structure of MEMO, three roles are designed: User, Provider, and Keeper, where User is the storage consumer, Provider is the storage space provider, and Keeper is the coordination manager. The registered address of the role will become the unique user characteristic that is identified and saved in the smart contract, subject to the management of the role information system inside to the smart contract, which matches the service and establishes the transaction, and imposes penalties on the role that violates the rules.
Of these three roles, Keeper acts as a third-party auditor, periodically challenging Providers and verifying that they are storing data intact, which is the focus of Rollup’s data availability solution.
3.1.2 Storage process
The data storage of MEMO is mainly accomplished by the above three roles cooperating with each other, and the specific storage process is as follows:
When a User initiates a request for storage data, the file is first sharded, and then the Keeper will assign multiple Provider storage execution nodes based on the reputation mechanism. The execution nodes will select storage devices based on the reputation and resource status of each device, pass these sharded data to them, and then record the corresponding metadata in the chain. According to the storage proof mechanism, Keeper needs to periodically verify the storage proof to Provider, if Keeper fails to verify the storage proof, the storage node will be penalized.
3.2 Data validation
3.2.1 Validation Principles
Each data validation effort is initiated by the Keeper, which generates the VRF key pair and sends the VRF public key to the corresponding data store Provider while saving the VRF private key locally, which means that the Provider utilizes the VRF public key to validate the reliability of the function. Similarly, the random numbers generated by the VRF computation and the response evidence will be sent to the Provider together for the subsequent verification process. In other words, Keeper as the computation party holds the private information and Provider as the verification party holds the public information, thus making Keeper’s randomization process verifiable.
After ensuring the verifiability of the randomization phase, the randomization process also needs to be publicly verified. It can be seen that the essence of the collusion attack lies in the fact that the process related to random numbers is too secretive and its generation process is not transparent, which makes it difficult to verify the reliability of the generated random numbers. The verifiable random function utilizes a commitment scheme similar to public key cryptography, in which the prover guarantees unpredictability through the private key and the black-box mode function, and the verifier guarantees verifiability using the public key and the fully public function output, thus achieving an effective combination of the two necessary properties.
Figure : Interaction flow of roles in the proof of validation
3.2.2 Validation process
1. Data pre-processing
Before uploading the data, the User first preprocesses the data; the User generates public and private information based on the parameter calculations, the public information is sent to the Keeper for storage, and the private information is saved locally by the User; and the User needs to sign the outsourced data that will be stored in the Provider to generate the corresponding label information, and then the User combines the outsourced data and the label information and sends them to the Provider. After that, the user combines the outsourced data and label information and sends them to the Provider.
User slices the outsourced data to be stored in the data store into data slices and names the indexes of the data according to the incremental integer, then generates labels for signing according to the data slices and the indexes, after which the user combines the outsourced data with the label information and sends it to the Provider.
Figure: user’s computation of data blocks to generate labels
2. Generate challenge information
Keeper receives the public information from the User, which is equivalent to accepting the challenge commission from the User, and therefore chooses a kind of information that will change and the change is unpredictable as the input for the subsequent computation.Keeper generates its own key pair of verifiable random function and sends the challenge information to the Providers, which is the public key generated by Keeper according to the verifiable random function. The challenge message is the public key generated by Keeper based on the verifiable random function.
3. Generate proof of data holdings
Upon receiving the challenge message, the Provider first validates it, and if the validation passes, it generates the corresponding data holding evidence and sends it back to the Keeper.
4. Certification
Finally, the Keeper performs a two-step verification of the received evidence of data holdings in sequence based on the stored public information of the user, which can indicate that the Provider has correctly and completely stored the outsourced data when and only when the results of both steps of the verification are passed.
3.3 Data security
3.3.1 Data encryption
1. Asymmetric encryption: User uploads data and generates a private key, only User’s own private key can decrypt the data
2. Digital Signature: User encrypts the private key of the asymmetric cryptography algorithm and generates a message digest, which is attached to the message; Provider gets the digital signature and the message content before the signature, and uses the public key of the asymmetric cryptography algorithm distributed by the User to check whether the message content before the signature has been tampered with in the transmission process or in the distribution process, and thus confirms the identity of the User. The identity of the User can thus be confirmed.
3.3.2 Fault tolerance mechanisms
Based on the volume and application characteristics of each data, MEMO adopts the fault-tolerant mode combining multi-copy and corrective redaction code, with multi-copy technology adopted for metadata with small volume, while corrective redaction code fault tolerance is adopted by default for data with large volume. However, the user has the right to choose the data independently, and can choose either the corrective action code or the multi-copy mode. This flexible multi-level fault-tolerance mechanism is to effectively improve the storage space utilization, and secondly, it can effectively protect the user’s right to choose independently.
Because the data and metadata are only volume differences and functional differences, and both are stored in the storage layer in the decentralized storage model, i.e., both are stored in a low-trust environment, the correctness of the data also needs to be verified at this stage, and for the same considerations as those in the metadata fault-tolerance mechanism, the correctness verification strategy of BLS signatures is used for the data.
3.3.3 Data restoration
RAFI Risk-Aware Data Failure Confirmation Strategy: RAFI is able to accelerate the remediation process by optimizing the two steps of risk classification and confirmation strategy to quickly identify data that is at high risk of loss.
4. Storage costs: a simple description, storage, reading, deletion and other costs and principles of realization
4.1 Cost: The cost of storage is determined by the size of the valid data stored by the user and the actual computing power consumed by the storage provider.
4.2 Conditions: The user deploys a payment contract and transfers sufficient funds for payment to the contract. When the payment condition is triggered, the contract will automatically calculate the amount and implement automatic payment.
4.3 Strategy: Delayed Payment + Interval Payment, Delayed Payment mechanism ensures the active contract of Keeper and Provider, while Interval Payment ensures the user benefit of Provider.
5. Application Scenarios
5.1 Decentralized database and dApp containers across chains
In the service layer, MEMO provides a decentralized database and a dApp container based on MEMO’s decentralized cloud storage system. This container is currently compatible with EVM and FVM environments, which ensures that the dApp data is stored in a user-controlled manner and has autonomous control over its data. Through it, user data is stored in a location controlled by the user himself, the dApp requires user authorization to read the data, and each user has his own access to the dApp URL, which is resolved through decentralized DNS. Transactions are settled through the corresponding on-chain payment method without the need for currency conversion.
5.2 Layer2 — Data Availability Solutions
MEMO’s data availability scheme is to help the Layer2 project outlaw the direct storage of transaction instructions on Ether, and dump the transactions in batches to the MEMO system and then publish the storage index corresponding to the transaction batches on Ether, and nodes can access the transaction data from MEMO through the value of this index.
The advantage of this solution is the lower storage cost and higher throughput of the MEMO system, in addition to the fact that all other raw data generated by calculations and transaction orders will be stored in the MEMO system and can be retrieved at any stage by virtue of the index values.
Cooperation Case: In 2021, MEMO and Metis, a star L2 project, entered into a cooperation to provide data packaging and batch processing services, and in 2022, the two sides deepened the cooperation, and Metis transaction data was gradually uploaded to MEMO’s system.
5.3 Data storage services
Provide the basic storage facilities built by each ecosystem, with the help of MEMO storage API you can build storage applications in other ecosystems for storing application data or digital assets generated in the process of use, including NFT, text, images, etc., and provide wallet services, in addition to blockchain storage MEMO also supports the construction of Internet storage.
Application examples: the current storage class net disk application EthDrive developed by MEMO community developers has been online, mainly for individual users to provide data storage, cross-chain transmission and asset management functions, EthDrive official website: ethdrive.net
6. Summary
In terms of basic data storage services or data availability, MEMO is currently in the application stage. Compatibility with EVM and other environments and easing the pressure on the Ethernet ecosystem are a good start for MEMO, and the continuous expansion of the node network will bring MEMO continuous storage space and more comprehensive storage services.
Over the years, MEMO team insists on blockchain technology research, focusing on finding a way out for data storage in the Web3.0 era, and now solves some of the Layer2 storage pressure and data availability problems, and provides the basic functions of storage in the new era. With the strong entry of AI, the real blockchain technology will also usher in its own era, and with more and more builders, the ecosystem may usher in a more complicated situation, MEMO should stick to its heart, maintain good data interaction and help the ecological prosperity.
