monero/ZMQ.md

The Current/Future Status of ZMQ in Monero

ZMQ Pub/Sub

Client ZMQ_SUB sockets must "subscribe" to topics before it receives any data. This allows filtering on the server side, so network traffic is reduced. Monero allows for filtering on: (1) format, (2) context, and (3) event.

• format refers to the wire format (i.e. JSON) used to send event information.

• context allows for a reduction in fields for the event, so the daemon doesn't waste cycles serializing fields that get ignored.

• event refers to status changes occurring within the daemon (i.e. new block to main chain).

• Formats:

  • json

• Contexts:

  • full - the entire block or transaction is transmitted (the hash can be computed remotely).
  • minimal - the bare minimum for a remote client to react to an event is sent.

• Events:

  • chain_main - changes to the primary/main blockchain.
  • txpool_add - new publicly visible transactions in the mempool. Includes previously unseen transactions in a block but not the miner_tx. Does not "re-publish" after a reorg. Includes do_not_relay transactions.
  • miner_data - provides the necessary data to create a custom block template Available only in the full context.

The subscription topics are formatted as format-context-event, with prefix matching supported by both Monero and ZMQ. The format, context and event will never have hyphens or colons in their name. For example, subscribing to json-minimal-chain_main will send minimal information in JSON when changes to the main/primary blockchain occur. Whereas, subscribing to json-minimal will send minimal information in JSON on all available events supported by the daemon.

The Monero daemon will ensure that events prefixed by chain will be sent in "chain-order" - the prev_id (hash) field will always refer to a previous block. On rollbacks/reorgs, the event will reference an earlier block in the chain instead of the last block. The Monero daemon also ensures that txpool_add events are sent before chain_* events - the chain_* messages will only serialize miner transactions since the other transactions were previously published via txpool_add. This prevents transactions from being serialized twice, even when the transaction was first observed in a block.

ZMQ Pub/Sub will drop messages if the network is congested, so the above rules for send order are used for detecting lost messages. A missing gap in height or prev_id for chain_* events indicates a lost pub message. Missing txpool_add messages can only be detected at the next chain_ message.

Since blockchain events can be dropped, clients will likely want to have a timeout against chain_main events. The GetLastBlockHeader RPC is useful for checking the current chain state. Dropped messages should be rare in most conditions.

The Monero daemon will send a txpool_add pub exactly once for each transaction, even after a reorg or restarts. Clients should use the GetTransactionPool after a reorg to get all transactions that have been put back into the tx pool or been invalidated due to a double-spend.