Table of Contents
In this article, you will discover everything you need to know about indexers and even learn how to build your own. So, let’s dive in!
Blockchains and their Nature
Blockchain data is optimized for serialized writes, one block at a time, as the chain is being created. Querying the blockchain for data about a specific block or account is fairly straightforward, known as a “narrow” query. However, if we need to query data across multiple blocks, it can become cumbersome as we have to aggregate results from various single-block queries. These are what we call “wide” queries.
A blockchain is a distributed database, and a smart contract (decentralized application, dApp) is an application that runs on a virtual machine inside a blockchain. It’s important to understand that smart contracts should not be considered as a “backend.” While some applications may consist solely of smart contracts, building a dApp with only smart contracts is usually not feasible.
Smart contracts have limitations when it comes to interactions, such as user notifications and integration with third-party applications. However, the nature of a blockchain requires it to be deterministic. A blockchain knows the state at a given time, which is a block in the case of blockchains. Think of it as snapshots. The blockchain takes snapshots of its state with every block. Users can call smart contracts for a specific block, and the blockchain guarantees that the execution will always produce the same result for the same block.
The deterministic nature of a blockchain prevents it from being influenced by external (off-chain) variables. It is impossible to make API calls from within a smart contract. A blockchain and smart contract are isolated from the external (off-chain) world.
Blockchains are excellent at applying requested changes to the state in a decentralized manner. However, to observe the changes, you need to actively pull information from the network.
Instead of abstract explanations, let’s look at an example.
The dApp has a helper deployed off-chain, and this helper can send an email with a copy of an e-book. But how do we trigger the helper?
Getting the Data from a Blockchain from the External World
NEAR blockchain provides a JSON-RPC endpoint for users to interact with the blockchain. Through the JSON-RPC API, users can call smart contracts and view data from the blockchain.
In our example, we can make our helper pull a Block every second, then retrieve all the Chunks and analyze the Transactions included in the Block. We check if there is a transaction to our smart contract with a “buy an e-book” function call. If we find such a transaction, we need to ensure its success before sending the e-book to the user.
After completing the process, we can trigger the helper’s code to send the user an email with the purchased e-book.
This approach is called the pull model of getting data. There is nothing wrong with this approach, but sometimes it may not be the most convenient or reliable.
Not all data is available through the JSON-RPC. For example, Local Receipts are not stored in NEAR node’s internal database and are therefore not accessible through the JSON-RPC.
Indexer
An indexer is an implementation of the push model of getting data. Instead of actively pulling data from the source, your helper waits for the data to be sent to it. The data is complete, allowing the helper to analyze it immediately without the need for additional pulls to obtain more details.
In our example, the helper becomes an indexer that receives every Block, along with Chunks, Transactions, and their statuses. The helper analyzes the data and triggers the code to send the user an email with the purchased e-book.
Indexers and “Wide” Queries
As mentioned earlier, indexers listen to the stream of data from the blockchain and can immediately filter and process the data according to defined requirements. This ability makes them useful for simplifying the execution of “wide” queries. For instance, a stream of data can be written to a permanent database, allowing later analysis using a convenient query language like SQL. This is exactly what Indexer for Explorer does.
Another scenario where “wide” queries are essential is when using a seed phrase to recover one or more accounts. Since a seed phrase represents a signing key pair, the recovery process involves finding and recovering all accounts associated with the matching public key. NEAR Indexer for Explorer stores this data in a permanent database, enabling NEAR Wallet to perform such “wide queries.” This would be impossible with JSON-RPC alone.
Summary
We hope this article has helped you understand the concept of indexers and decide whether you need one for your application.
What’s Next?
We encourage you to learn more about the Lake Indexer project. Please proceed to the Tutorials section to learn how to build an indexer in practice.
Alternatively, there are a few third-party indexers that are tightly integrated with the NEAR ecosystem. You can explore your data sourcing options, including The Graph, Pagoda, Pipespeak, and SubQuery, under data tools.
