Datacenter outage

On August the 14, 2021 Digital Ocean have encounter an outage in their datacenter in New York. This outage have affect all the nodes in a single datacenter. If you own one of those nodes or if your node datacenter undergoes an interruption please follow this procedure.

The usual symptoms of a network split are :

• Your node stop prunning at a specific block and stop.
• Your node block height beeing different than the network one ( see https://explorer.nkn.org )

If all nodes in a datacenter have lost their connections to the main network ( aka nodes outside that datacenter -i.e. network separation-)

In this case they will form a little private network. You need to restart them to join back the main net.

But there is one important caveat: since it has been a while before the outage, BEFORE you start your nodes, please turn off pruning of your node by adding the line StatePruningMode: "none" in your node's config.json.

This is because if the private net continues to produce blocks, your nodes will have a forked local ledger. If your nodes have too many forked blocks, when it starts, it will prune pass the fork point, and once that happens, your node cannot rollback to the correct ledger. Turning off pruning before restarting nodes will prevent that from happening. If, unfortunately, your node has pruned those blocks, you would need to use a chaindb snapshot or let it sync from zero