Pagination
Pagination approach used by Claret API endpoints with large datasets.
Most API GET
endpoints for Claret will automatically paginate 1000 records at a time. If you expect to receive fewer records, then pagination will not apply to your pull.
If, however, you will be expecting more than 1000 records per API call, the following details should be taken into account:
The current_index
value will be required for each paginated data request after the initial GET
request is made. On the first API call, the current_index
will default to 0
, and does not need to be included in the query parameters. The second and subsequent requests, however, will require this parameter to be set.
The value of the current_index
parameter can be retrieved from the next_index
value that will be returned with each paginated response. So, for each batch of 1000 records that are returned within the context of paginated data, a next_index
key will be included until no additional data remains. Once the final batch of 1000 (or fewer) records is returned, the next_index
value will be null
. So long as the next_index
key is not null
, it's value must be used as the current_index
parameter value on the next request.
Let's use an example of a paginated request that will contain 2,222 records total.
For the initial API call to the GET
endpoint, no additional query parameters (beyond that API endpoint's essential requirements) need to be passed through. The response from this initial API call will contain the following key
=> value
pairs within the JSON:
Let's break this down.
total
represents the total number of records that will be returned throughout the course of the paginated request/response cycle. Here, that value is 2222.from
represents the starting number of the total records (1 through 2222) for this particular response. As this is the very first pull, in this example, this value is 1 (the starting point).to
represents the ending number of the total records (1 through 2222) for this particular response. As this is the very first pull, in this example, this value is 1000 (the first 1000 of the 2222 total records).per_page
represents the total number of records contained in each paginated batch. Here, we can see that we are receiving 1000 records at a time.previous_data_count
represents the amount of records we have already received from prior paginated API calls to this same endpoint. As this is the first pull, that number is 0.data_count_remaining
represents the number of records that still need to be retrieved based on the total that will be returned along with the amount already retrieved using theper_page
counter. As this is the first pull of 1000 records out of a total of 2222 records, this value informs us that we still have 1222 records remaining to retrieve.current_index
represents the index that was used to initiate this batch of 1000 records. For the initial pull, that value is0
, which is what we see here.
Important Note: This will not correspond to the actual number of records or amount of API calls. It is a very specific index that ensures all data being retrieved is the most recent, and is based on primary keys in the database. So, be sure to always reference it as the next_index
value.
next_index
represents the value that should be included as thecurrent_index
parameter value in the next API call. For illustrative purposes here, that value is 9127.success_count
represents the total number of records that were retrieved successfully for this response.
Now... as this data pull contains more than 1000 records, and our next_index
value is not null
, we can hit the same API endpoint as the initial pull, but this time, we will include an additional query parameter of current_index
with a value of 9127 (¤t_index=9127
) - which was the next_index
value returned in the first call.
Our paginated response values will be included in this next response as follows:
Once again, the next_index
value is not null
, and the data_count_remaining
is > 0
. So, the next and final request should contain the current_index
parameter with a value of 14555 (¤t_index=14555
), which is the next_index
value from the most recent response. This will provide us with our final batch of paginated records, and the pagination JSON would look like this:
Finally, data_count_remaining
is 0 and next_index
is null
, so the full paginated data load has been retrieved, and no additional GET
requests are necessary.
Last updated