API rate limit errors on bulk imports
#7
We are hitting 429 responses when importing more than 500 records in a single batch. Is there a documented rate limit we should design around?
Solution
meowOwnerThe documented limit is 100 requests per minute per API key, and bulk imports count each record as its own request today. For anything over a few hundred records, batch them into chunks of 50 with a short delay between chunks, or reach out and we can raise your limit temporarily for a one-time migration.
7 replies
meowOwner✓ Solution
The documented limit is 100 requests per minute per API key, and bulk imports count each record as its own request today. For anything over a few hundred records, batch them into chunks of 50 with a short delay between chunks, or reach out and we can raise your limit temporarily for a one-time migration.
meowOwnerThe documented limit is 100 requests per minute per API key, and bulk imports count each record as its own request today. For anything over a few hundred records, batch them into chunks of 50 with a short delay between chunks, or reach out and we can raise your limit temporarily for a one-time migration.Chunking into 50s fixed it for us too, thanks for the tip.
We ended up just queuing the import overnight when traffic is low, works fine as a stopgap.
meowOwnerThe documented limit is 100 requests per minute per API key, and bulk imports count each record as its own request today. For anything over a few hundred records, batch them into chunks of 50 with a short delay between chunks, or reach out and we can raise your limit temporarily for a one-time migration.One more thing, does the raised limit apply permanently or just for the one migration?
meowOwner
Jane CooperOne more thing, does the raised limit apply permanently or just for the one migration?Just for the one-time migration window we agree on, your limit reverts to the standard 100 per minute afterward unless you ask us to keep it raised.
Chunking helped us too, though we ended up going a bit smaller, batches of 25, to stay well under the limit with headroom.
Kai TanakaChunking helped us too, though we ended up going a bit smaller, batches of 25, to stay well under the limit with headroom.Good tip, will try a smaller batch size if we hit this again.
Sign in to reply to this question.
Powered by Forumcat