Welcome back for another instalment of the SubQuery Mastery Series, where we help you navigate and master the SubQuery Network.
When it comes to indexing, performance is a crucial factor. So how can you optimise your SubQuery project to speed it up? In this article, we’ll delve into practical tips and tricks for writing efficient queries in SubQuery, highlighting the best practices for indexing and data retrieval. By leveraging these strategies, developers can enhance their SubQuery indexers to deliver quicker responses, reduce resource consumption, and improve overall user experience. Whether you're a seasoned developer or just getting started, these insights will help you navigate the intricacies of query optimization in SubQuery.
We’ll start out by covering how you can more efficiently query your project, and then dive deeper into improvements to your project to maximise your indexing speed.
Hot Tips for Query Performance
Improving your indexer query performance is essential because it directly impacts the speed and efficiency of data retrieval, which is crucial for maintaining a responsive and reliable application. Optimised queries not only reduce the load on your database but also enhance the overall user experience by providing faster access to necessary information. Firstly here are some hot tips to be aware of for your queries that will greatly improve performance:
- Cursor-based pagination is far more efficient compared to first/offset/after pagination
- Only query the necessary fields from GraphQL
- For large data tables, avoid querying totalCount without adding conditions
- Add indexes where possible to the database tables for columns that you often filter or sort on
Another thing you want to do is restrict query complexity. GraphQL is extremely powerful, but one of the downsides is that it allows users to make somewhat unrestricted calls that result in complex SQL that can really affect performance for other users. We provide some controls for this that you can read about here.
You should also consider reading this excellent guide from Apollo on how they recommend you secure your GraphQL API from malicious queries.
Boost Indexing Speed
There are a few typical issues that we see developers run into time and time again. Check them out below and once you implement the fixes, you may see your SubQuery project performance has a noticeable improvement.
- Avoid using blockHandlers where possible. Using block handlers slows your project down as they can be executed with each and every block. Use them only if you need to and consider adjusting project architecture.
- If you must use a block handler, ensure that you carefully optimise every code path called by it. As it will be executed on each block the total time that it might take will increase linearly as the chain grows.
- Instead, can you use the modulo filter to run a handler only once to a specific block? This filter allows handling any given number of blocks, which is extremely useful for grouping and calculating data at a set interval. For instance, if modulo is set to 50, the block handler will run on every 50 blocks.
- Always use a dictionary (we can help create one for your new network if you want one). You can see examples of how to create a dictionary in the dictionary repository.
- Use as strict filter conditions in your mapping handlers (within the project manifest) as possible to reduce the number of events/transactions that need to be processed. Create filters as specific as possible to avoid querying unnecessary data.
- Set the start block in your project manifest to when the contract was initialised or when the first event/transaction occurs - no point indexing blocks with nothing useful in them!
- Use node worker threads to move block fetching and block processing into its own worker thread. It could speed up indexing by up to 4 times (depending on the particular project). You can easily enable it using the -workers=<number> flag. Note that the number of available CPU cores strictly limits the usage of worker threads. Read more here.
Review Your SubQuery Project Architecture
If your project requires indexing all the blocks, transactions alongside more specific data, consider dividing it into separate SubQuery projects responsible for different data sources. If such separation is possible it can provide better development experience and efficient workflow. This decision can be compared to a design decision between micro-services and monolith project architecture.
We recommend this approach, because it takes time to index all the blocks and it can slow down your project significantly. If you want to apply some changes to your filters or entities shape you may need to remove your database and reindex the whole project from the beginning.
A common example is creating a large project that indexes everything so you can perform internal analysis on your contracts, and then much smaller and optimised project for indexing the key data for your dApp. The larger project that indexes everything might never change and so you can avoid costly reindexing, while the smaller optimised project will change as your dApp matures and can be reindexed much faster.
Other Improvements
But wait there’s more! If you’re indexer is not running like you’d expect, and none of the tips in this article have helped so far, then check out the further improvements outlined in our docs.
And if you’re stuck, just get in touch! You can join our Discord and message our community full of developers on the #sdk-technical-support channel, or you can reach out directly to support@subquery.network.
We can wait to see what you build!
About SubQuery
SubQuery Network is innovating web3 infrastructure with tools that empower builders to decentralise the future. Our fast, flexible, and open data indexer supercharges dApps on over 200 networks, enabling a user-focused web3 world. Soon, our Data Node will provide breakthroughs in the RPC industry, and deliver decentralisation without compromise. We pioneer the web3 revolution for visionaries and forward-thinkers. We’re not just a company — we’re a movement driving an inclusive and decentralised web3 era. Let’s shape the future of web3, together.
Linktree | Website | Discord | Telegram | Twitter | Blog | Medium | LinkedIn | YouTube
