Indexing: It ain't a sexy business, but it's honest work.
No one talks about indexing, yet it's one of the most important aspects of crypto. Ready to explore this field in very simple terms?
What is the common point between indexing and logistics? This is not a sexy business, but it’s extremely useful. Not sure if it this convince you to continue reading lol, but bare with me. That will be cool I promise.
Super formally, indexing is a method of organizing items based on a common property or specific rule. For example, dictionaries index words alphabetically from A to Z. Similarly, health tip books arrange content according to symptoms, illnesses, or the drugs used for treatment.
This is basically the same in blockchains. For example, it allows you to track and inspect a specific NFT collection, rather than the entire set of NFTs traded within that blockchain.
But searching for data was not like this in the past. In fact, blockchains were never built for indexing! This caused a lot of problems. A whole lot, bro.
I want you to think of a blockchain as a dictionary. Give it a name you like. I’m naming mine “NAIROLF”.
NAIROLF is an English dictionary that has been collecting every English word said since its creation. Whenever a new word was said, it was immediately added to NAIROLF.
The first word said is…
“Crypto is fuckin hoooooge”.
These words are kept in the dictionary after confirming that it is English — hoooooge is English, right?
Every single person on Earth would contribute to the dictionary. Fast forward to 28,476 years later, NAIROLF dictionary is filled with lots of new and old words.
You, as a curious person, don't know the meaning of “Encryption” and open NAIROLF to look for its meaning. Where will you open the book? How will you spot Encryption in the dictionary?
You have no clue! That’s the problem with blockchains too, bro.
They store data in immutable blocks that are arranged according to their date of creation. To find a specific kind of data on any blockchain, you will look through each block and check each data within it.
As of writing, Ethereum has a height of over 20,000,000 blocks and processes 1.156 million transactions each day. That’s a shit ton of work if you wanna find data from 7 years ago.
Indexing is also a lot of work. To index data on a blockchain, every piece of data created and added to the blocks must be checked to see if it contains the specific information you’re looking for—every single time!
Using the nodes of blockchains, which are already working hard to validate blocks and build the network, was impossible. If an employee is already working incredibly hard and you keep adding pressure on their shoulders, he will burn out. Well, the same goes for nodes.
Fortunately, some projects have developed solutions dedicated to solving the problem of accessing blockchain data. Let’s explore a few of them.
The Graph is one of the forerunners of indexing protocols and projects related to querying blockchain data.
It uses subgraphs, which are like narrow snapshots of blockchain information. These snapshots can focus on specific transactions or blocks, tailored to meet your specific needs.
Imagine a subgraph as a personalized guidebook. You create this guidebook to find exactly what you're interested in within a vast blockchain. When you give this guidebook to The Graph, it follows the instructions closely. It checks through its own network's nodes, which are constantly watching transactions and smart contracts being added to blocks in real-time.
Using your guidebook, The Graph then creates a special database. This database holds exactly the information you asked for—like transaction details or smart contract actions.
Later on, you can easily search this database. It lets you quickly find and use the exact blockchain data you need for your own projects or to understand how things are happening in the blockchain world.
There are other projects that are very similar to Graph, but are made with some added features.
Goldsky also provides projects and chains with subgraphs. This helps to organize and store data in a way that makes it super easy to search, access, and retrieve.
Another intriguing product offered by Goldsky is Mirror.
It used to be a real puzzle to figure out how to take data on the blockchain and use it for a decentralized application. You had to call external APIs, or pull data from a provider if you prefer. This was complex, costly and not that efficient.
With Mirror, you can create so-called pipelines that stream on-chain data to your off-chain database in real time. So, instead of you having to pull the data from the chain, it is directly pushed to you.
It’s like living a life of getting water to your kitchen with a bucket and with a water pipeline system. See the convenience and speed added there?
Your application can then use this database to query real-time data more easily. This also allows your application to combine on-chain and off-chain data in its database, making it possible to build more complex use cases.
You’ve grasped it: indexing is extremely important yet often overlooked in discussions on CT. It’s like the hidden machinery that keeps the whole system running without receiving the recognition it deserves. If many of today’s applications function the way they do, it’s thanks to these behind-the-scenes indexing protocols…
Disclaimer: This content is presented to you on an “as is” basis for general information and educational purposes only, without representation or warranty of any kind. It should not be construed as financial, legal or other professional advice, nor is it intended to recommend the purchase of any specific product or service. You should seek your own advice from appropriate professional advisors. Digital asset prices can be volatile. The value of your investment may go down or up and you may not get back the amount invested. You are solely responsible for your investment decisions and Crypto Made Easy is not liable for any losses you may incur. This material should not be construed as financial, legal or other professional advice.