Pagination

Pagination approach used by Claret API endpoints with large datasets.

Most API GET endpoints for Claret will automatically paginate 1000 records at a time. If you expect to receive fewer records, then pagination will not apply to your pull.

If, however, you will be expecting more than 1000 records per API call, the following details should be taken into account:

The current_index value will be required for each paginated data request after the initial GET request is made. On the first API call, the current_index will default to 0, and does not need to be included in the query parameters. The second and subsequent requests, however, will require this parameter to be set.

The value of the current_index parameter can be retrieved from the next_index value that will be returned with each paginated response. So, for each batch of 1000 records that are returned within the context of paginated data, a next_index key will be included until no additional data remains. Once the final batch of 1000 (or fewer) records is returned, the next_index value will be null. So long as the next_index key is not null, it's value must be used as the current_index parameter value on the next request.

Let's use an example of a paginated request that will contain 2,222 records total.

For the initial API call to the GET endpoint, no additional query parameters (beyond that API endpoint's essential requirements) need to be passed through. The response from this initial API call will contain the following key => value pairs within the JSON:

"total": 2222,
"from": 1,
"to": 1000,
"per_page": 1000,
"previous_data_count": 0,
"data_count_remaining": 1222,
"current_index": 0,
"next_index": 9127,
"success_count": 1000

Let's break this down.

• total represents the total number of records that will be returned throughout the course of the paginated request/response cycle. Here, that value is 2222.

• from represents the starting number of the total records (1 through 2222) for this particular response. As this is the very first pull, in this example, this value is 1 (the starting point).

• to represents the ending number of the total records (1 through 2222) for this particular response. As this is the very first pull, in this example, this value is 1000 (the first 1000 of the 2222 total records).

• per_page represents the total number of records contained in each paginated batch. Here, we can see that we are receiving 1000 records at a time.

• previous_data_count represents the amount of records we have already received from prior paginated API calls to this same endpoint. As this is the first pull, that number is 0.

• data_count_remaining represents the number of records that still need to be retrieved based on the total that will be returned along with the amount already retrieved using the per_page counter. As this is the first pull of 1000 records out of a total of 2222 records, this value informs us that we still have 1222 records remaining to retrieve.

• current_index represents the index that was used to initiate this batch of 1000 records. For the initial pull, that value is 0, which is what we see here.

Important Note: This will not correspond to the actual number of records or amount of API calls. It is a very specific index that ensures all data being retrieved is the most recent, and is based on primary keys in the database. So, be sure to always reference it as the next_index value.

• next_index represents the value that should be included as the current_index parameter value in the next API call. For illustrative purposes here, that value is 9127.

• success_count represents the total number of records that were retrieved successfully for this response.

Now... as this data pull contains more than 1000 records, and our next_index value is not null, we can hit the same API endpoint as the initial pull, but this time, we will include an additional query parameter of current_index with a value of 9127 (&current_index=9127) - which was the next_index value returned in the first call.

Our paginated response values will be included in this next response as follows:

"total": 2222,
"from": 1001,
"to": 2000,
"per_page": 1000,
"previous_data_count": 1000,
"data_count_remaining": 222,
"current_index": 9217,
"next_index": 14555,
"success_count": 1000

Once again, the next_index value is not null, and the data_count_remaining is > 0. So, the next and final request should contain the current_index parameter with a value of 14555 (&current_index=14555), which is the next_index value from the most recent response. This will provide us with our final batch of paginated records, and the pagination JSON would look like this:

"total": 2222,
"from": 2001,
"to": 2222,
"per_page": 1000,
"previous_data_count": 2000,
"data_count_remaining": 0,
"current_index": 14555,
"next_index": null,
"success_count": 222

Finally, data_count_remaining is 0 and next_index is null, so the full paginated data load has been retrieved, and no additional GET requests are necessary.