πŸ‡Parallel Indexing using RabbitMQ

By parallelizing the indexing process, you can significantly reduce the time required to index blockchain data

Parallelizing Indexing with RabbitMQ

Due to the need to index many entities, the process can take significant time for each block. To achieve real-time indexing, a smaller set of entities can be indexed separately by different workers. After indexing a block, the resulting objects are sent to RabbitMQ. Each type of entity has its own routing key, ensuring the message is delivered to the appropriate queue while maintaining indexing order.

For example, to index logs, blocks and transactions are required. In the first worker, indexed blocks and transactions are sent to RabbitMQ with the routing key block. The second worker, which has its own queue configured for the routing key block, receives this message and starts indexing logs using the already indexed blocks and transactions. Meanwhile, the first worker continues indexing the next set of blocks.

Steps for Parallel Indexing

  1. Set Up RabbitMQ Queues and Routing Keys

    • Define queues for each entity type.

    • Assign a routing key for each entity type to ensure messages are delivered to the correct queue.

  2. Configure Workers

    • Assign specific entities to be indexed by each worker.

    • Ensure each worker listens to the appropriate RabbitMQ queue.

  3. Indexing Workflow

    • Worker 1: Indexes blocks and transactions, then sends the indexed data to RabbitMQ with the routing key block.

    • Worker 2: Listens to the queue for block routing key, receives the indexed blocks and transactions, and begins indexing logs.

    • Worker 1: Continues to index the next set of blocks and transactions in parallel.

This approach allows for efficient and scalable real-time indexing, leveraging RabbitMQ for task distribution and coordination among workers.

By parallelizing the indexing process, you can significantly reduce the time required to index blockchain data, ensuring timely and accurate updates across your GuruNetwork.ai products.

Last updated