← Blog
Perfex CRMAPIIntegrations

Perfex CRM REST API Pagination, Filtering, Rate Limits

Perfex CRM REST API pagination, sorting, field selection and rate-limit headers - the working pattern that gets ten thousand records out without a 429.

A nightly job pulls invoices out of the CRM into a reporting database. It ran for a year against 400 invoices and finished in nine seconds. The client grows, the invoice table passes 11,000 rows, and the job comes back with a PHP memory exhaustion, a 60-second gateway timeout, or a wall of 429 responses halfway through. Nothing about the CRM changed. What changed is that a request shape which was accidentally fine at 400 records is the wrong shape at 11,000. Perfex CRM REST API pagination is the first of four levers that fix it - the others being sorting, field selection and date-range filtering, which exist on every list endpoint for exactly this reason.

We build and sell the REST API module for Perfex CRM, so weigh the recommendations accordingly. The patterns below are ordinary REST discipline; the specifics - parameter names, limits, headers - are that module’s.

Perfex CRM REST API pagination: page size is not the lever

Every list endpoint shares a unified toolkit for pagination, sorting, field selection and date-range filtering, and it is fully backwards compatible. That compatibility is precisely the trap. An integration written before the toolkit existed sends none of those parameters and keeps getting everything - right up to the day the result set outgrows memory_limit on the PHP process building the JSON.

Two things matter more than page size.

Ordering has to be explicit. Without a sort you are relying on whatever order the database happens to return, and that is not guaranteed to be stable between two requests. Sorting is available as ?sort=-datecreated for newest first; drop the minus for oldest first.

Newest-first paging silently loses records. If you page by offset while new rows are being created, every insert shifts the whole set down by one and pushes a record you have not read yet across a page boundary. You never see it, and nothing errors. Page ascending by creation date instead, so new rows always land after your cursor rather than in front of it. Better still, freeze the window with a date range before you start.

For the exact pagination parameter names, read GET /api/openapi on your own install rather than copying a snippet out of an article. It describes 74 paths and 144 operations, at the version you are actually running.

LeverParameterWhat it buys you
Paginationsee the OpenAPI documentBounded memory and response time per request
Sorting?sort=-datecreatedDeterministic ordering across pages
Field selection?fields=id,companySmaller payloads, cheaper JSON decode
Date rangecreated_after / created_beforeIncremental pulls and restartable backfills

Field selection is the cheapest win in the whole exercise

?fields=id,company is one query parameter and it usually saves more wire time than any amount of page tuning. A customer record carries a lot of columns; a reporting warehouse typically stores four. The rest is serialised by the CRM, pushed over TLS, and thrown away by your parser.

Three knock-on effects:

  • Response caching for GET requests is part of the module. A narrow, repeatable request shape is a more useful cache entry than a wide, ad-hoc one.
  • The response transformer can wrap responses in a standardised JSON format, filter fields by query parameter and automatically strip sensitive ones for privacy. An absent field is therefore not automatically a bug - it may have been removed deliberately on the way out.
  • A field you never asked for cannot regress on you. Narrow selection makes the integration immune to new columns upstream.

Date-range filters turn a full sync into an incremental one

created_after and created_before are what convert a nightly full export into a delta. Keep a high-water mark from the last successful run, ask for everything created after it, and the 11,000-row problem becomes a 30-row problem on every night except the first.

For that first run - the backfill - do not send one enormous range. Chunk it by month. Each window becomes a fixed, restartable unit of work: if the run dies in July you resume at July, not at the beginning, and you can space chunks days apart on a busy install.

Now the limitation. These filters are about creation, not modification: an invoice created in March and marked paid in August still carries a March creation date, so a created_after sync will never see that update. Two honest options:

  1. Reconcile on a window. Keep the nightly delta, and once a month re-pull a rolling recent period in full to catch status changes.
  2. Stop polling for changes and let the CRM push them. Webhooks 2.0 carries 124 events across 22 resource groups, bridged to 75 core Perfex actions so both admin-side and API-side changes fire. Deliveries are HMAC-signed with a timestamp (X-Perfex-Signature) for verification and replay protection, and can run through a background queue that retries with exponential backoff.

The rule we use: pull for backfill, push for change capture. Polling an API to discover that nothing happened is the most expensive way to learn nothing.

Perfex API rate limit headers beat guessing at a sleep

Every response carries X-RateLimit headers, which means a client never has to guess where it stands. Log them for one complete run before tuning anything. You will usually find either that the sync is nowhere near the ceiling and the real cost is payload size, or that it burns the allowance in ninety seconds and then coasts.

Read the headers by prefix and log whatever arrives, rather than hard-coding one name into a parser that throws the day the set changes. Back off on the server’s numbers, not on a fixed one-second sleep, which is too slow when you have headroom and too fast when you do not.

Two caps sit behind those headers and they are not the same thing:

  • Request limiting is a rate. You are going too fast right now.
  • Quota limiting is a budget. You have spent your allowance for the period, and waiting a few seconds does not help.

Both are configured per API key, alongside an expiry date, on a single screen. The practical consequence: give a one-off backfill its own token. A 40,000-record import can exhaust a quota steady-state traffic never came close to, and if it shares a key with your live website form handler it takes that down too.

One more distinction, because it produces an identical-looking stall: brute-force protection throttles repeated failed authentications per IP. If your retry loop is hammering a rejected key the cause is a 401, not a 429 - and those need completely different fixes.

Batch operations and idempotent retries, not bigger pages

Reads are the easy half. The write side is where a naive integration suffers: one HTTP request per record, meeting every limit at once.

Batch operations execute up to 50 in a single request, with per-operation results and continue-on-error. Five thousand writes stop being five thousand requests and become one hundred. Two caveats follow directly from that design:

  • Continue-on-error means the envelope can return successfully while operations inside it failed. You must read the per-operation results. A 200 on a batch is not fifty successes.
  • A batch of 50 fails as a group of 50 when the network drops, which makes the next point non-optional.

Idempotency-Key on POST replays the stored response instead of re-executing an identical retry, within a 24-hour window. That is the only real answer to a request that succeeded server-side and died on the wire on its way back. Without it a timeout becomes a duplicate invoice, and duplicates in billing data cost far more to clean up than a slow sync.

Three smaller details remove whole classes of follow-up calls:

  • record_id comes back in every create response, so you can store the local-to-CRM ID mapping immediately instead of searching for the record you just created.
  • Tolerant partial updates on 13 resources ignore unknown payload fields rather than erroring, so you can send your object as it stands. The honest flip side: a misspelled field name is discarded in silence, so validate names against the specification, not against a successful HTTP status.
  • Server-side totals for invoices, estimates, proposals and credit notes are computed from line items, taxes, discounts and adjustments, so you never reimplement that arithmetic or disagree with the CRM about what an invoice is worth. POST and PUT also accept the same items array shape that GET returns.

Check the contract, then measure the run

Guessing at parameter names is how integrations acquire folklore. That OpenAPI 3.0 document carries 498 typed request fields across 35 endpoints - import it into Postman, Insomnia or Stoplight, or generate a typed client from it. The Postman collection (144 requests in 25 folders) is generated from the same specification, so it cannot drift from the product. When a call behaves differently from your code, reproduce it in the Swagger playground inside Perfex; if it works there, the difference lives in your client.

The order to build a bulk sync in

  1. Fetch GET /api/openapi and confirm the parameter names for the specific endpoint you are calling.
  2. Issue a separate API key for the backfill, with its own quota and an expiry date on the day the import should finish.
  3. Page ascending by creation date, with an explicit sort, never newest-first.
  4. Add ?fields= listing only the columns you actually store.
  5. Chunk the historical import into monthly created_after / created_before windows, and record which windows completed.
  6. Log the X-RateLimit headers for one full run before touching concurrency or page size.
  7. Convert writes into batches of up to 50, and read the per-operation results, not the HTTP status alone.
  8. Attach an Idempotency-Key to every POST.
  9. Once the backfill is done, move change capture onto signed webhooks and keep the nightly pull only as reconciliation.
  10. After a week in production, read the per-API-key statistics and the request log, and resize the limits from measurement rather than assumption.

The decision rule: if you are about to raise a page size, stop and ask whether those records needed to cross the wire at all. Nearly every time the answer is a narrower field list or a tighter date window - and the request you never send is the one that can never time out.