Storing the Merkle Tree in a compact way

Allow me to introduce this simple idea that could be useful ...

-The Intuition was some discussion on Utreexo project about storage saving and some traversing issues in handling the UTXOS Merkle Tree/ forest; that is N internal nodes need to be stored along with 2N pointers (left&right), + maybe 1 more pointer in the leaves special nodes to handle different traversing options (insert, delete, & differently proof fetch that traverse aunt or niece node according to your implementation https://github.com/mit-dci/utreexo/discussions/316) . Then, I thought of a simple idea that gets rid of all the pointers; specially appealing when we have all trees are full (complete) in the forest, but can work for any Merkle Tree:

- 2D array with variable row size; R[j] is of length (N/2^j) -For example when N=8 nodes R[0]=0,1,2,...,7 R[1]=8,9,10,11 R[2]=12,13 R[3]=14 . -We can see that total storage is just 2N-1 nodes, no need for pointers, and traversing could be neat in any direction with the right formula:

-Pseudo code to fetch proof[i] ...

//direction to know + or -
If ((i mod 2)==0) drct=1;
            else drct=-1;
// first, the sibling node
proof[i]=R[0,i+drct]

//add the rest thru loop
For(j=1; j≤logN; j++)
 { index= i/(2^j)+drct;
    proof[i]=Add(R[j,index]);
 }

-In fact it's just the simple primitive approach of transforming a recursion to an iteration, and even if Utreexo team solved their problem differently I thought it is worth telling as it can work for any Merkle Tree . Thanks for your time, Shymaa M Arafat

It could be viewed as the simple complete tree to 1D array with no pointers described in lecture 8 here https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-006-introduction-to-algorithms-spring-2020/lecture-notes/index.htm starting from min 15 in this video https://youtu.be/Xnpo1atN-Iw

Since all trees in Utreexo forest are full binary trees, this is perfect to use, and we can save *76*10⁶*2*size of pointer(probably4bytes)* *~600MB *with almost no effort.

However, I suggest to put it in a 2D array to make it more easy to handle (the indexing math) as we, different than the lecture, traverse in many ways ( normally to delete or insert, and the parent siblings for the proofs)

I wrote more details here https://bitcointalk.org/index.php?topic=5360009.0

On Thu, Sep 16, 2021, 14:37 Vincent <vincent.palazzo@protonmail•com> wrote:

> Hi.
>
> Thanks for the reference, but I missed where you want save space with this
> compression on the Merkle Tree.
>
> Regards.
>
> Vincent.
> vincenzo.palazzo@protonmail•com
> https://github.com/vincenzopalazzo
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, September 16th, 2021 at 5:15 AM, shymaa arafat <
> shymaa.arafat@gmail•com> wrote:
>
> Allow me to introduce this simple idea that could be useful ...
>
> -The Intuition was some discussion on Utreexo project about storage saving
> and some traversing issues in handling the UTXOS Merkle Tree/ forest; that
> is  N internal nodes need to be stored along with 2N pointers (left&right),
> + maybe 1 more pointer in the leaves special nodes to handle different
> traversing options (insert, delete, & differently proof fetch that traverse
> aunt or niece node according to your implementation
> https://github.com/mit-dci/utreexo/discussions/316)
> .
> Then, I thought of a simple idea that gets rid of all the pointers;
> specially appealing when we have all trees are full (complete) in the
> forest, but can work for any Merkle Tree:
>
> - 2D array with variable row size; R[j] is of length (N/2^j)
> -For example when N=8 nodes
> R[0]=0,1,2,...,7
> R[1]=8,9,10,11
> R[2]=12,13
> R[3]=14
> .
> -We can see that total storage is just 2N-1 nodes,
> no need for pointers, and traversing could be neat in any direction with
> the right formula:
>
> -Pseudo code to fetch proof[i] ...
>
> //direction to know + or -
> If ((i mod 2)==0) drct=1;
>             else drct=-1;
> // first, the sibling node
> proof[i]=R[0,i+drct]
>
> //add the rest thru loop
> For(j=1; j≤logN; j++)
>  { index= i/(2^j)+drct;
>     proof[i]=Add(R[j,index]);
>  }
>
> -In fact it's just the simple primitive approach of transforming a
> recursion to an iteration, and even if Utreexo team solved their problem
> differently I thought it is worth telling as it can work for any Merkle Tree
> .
> Thanks for your time,
> Shymaa M Arafat
>
>
>