Bulk Files vs. API Pulls: The Cost Math Nobody Shows You
The real cost is not the license. Run the math on rate limits, re-pulls, and stale data before you sign.
- written by
- Adewale ThomasMixRank
- published
- 15 October 2026
- reading time
- 6 min
- filed under
- Data Infrastructure
Bulk files and API pulls are the two ways B2B data reaches your systems: complete dataset transfers, or record-by-record queries. The math that picks between them is rarely the license fee. It is the cost of re-moving, re-storing, and re-processing the same records every time your pipeline runs, plus what a stale answer costs the decision it feeds.
What Are Bulk Files and API Pulls?
A bulk file is the whole dataset, or a declared slice of it, delivered as one transfer: a set of Parquet or CSV files, or a shared table you query in place. An API pull is a question: you send filters, the vendor returns matching records a page at a time, and a rate limiter decides how fast the conversation can go. Same data, different physics. One moves the bytes once. The other moves records many times, one request at a time.
How the Delivery Choice Changes Your Bill
- 01Count what you actually touch: A real-time lookup touches one record. A model or a rebuild touches most of them. Most pipelines are somewhere in between, and the honest first step is measuring where yours sits.
- 02Price the pull loop: Records per page, divided into total records, divided by the rate limit, equals hours per full refresh. For a 120M-record dataset at 1,000 records per page and 10 requests per second, that is 120,000 calls and about three and a half hours, every time you want a clean copy.
- 03Price the file loop: One transfer to load the dataset, then incremental diffs for whatever changed. The same refresh that took hours of scheduled API calls becomes a file drop and a diff.
- 04Add the hidden line items: Re-processing records that did not change. Retry logic and pagination bugs. Queueing jobs around a rate ceiling. And freshness lag: the gap between when data changed and when your next permitted pull was noticed.
- 05Choose per workload, not per vendor: Lookups and small slices go to the API. Anything that touches most of the dataset, joins multiple datasets, or trains a model goes to bulk files. Teams that run both keep each one for what it is good at.
Bulk Files vs. API Pulls
| Bulk files | API pulls | |
|---|---|---|
| Delivery | Whole dataset or a declared slice in one transfer | Matching records, paged per request |
| Full refresh, 120M records | One transfer, then diffs | ~120,000 calls, about 3.5 hours at typical rate limits |
| Best for | Models, joins, rebuilds, anything touching most of the data | Lookups, small slices, real-time checks |
| Freshness | As of the file build; hourly files land changes fast | As of the call |
| Rate limits | None once you hold the file | Typically 10 requests per second; retries and queues become your problem |
| Cost shape | Storage and compute on your side, flat | Flat or metered, plus engineering time to respect the ceiling |
| What breaks at scale | Disk space | Pagination bugs, retry storms, stale caches |
Key Benefits of Doing the Math Before You Sign
You stop paying to re-read records that did not move. Here is a concrete case. MixRank updates 120M+ people profiles a month out of 1.20B+ total. A pipeline that re-pulls the full file through an API every month moves ten times more data than actually changed. Bulk delivery with incremental diffs moves the 120M.
Rate limits stop being your architecture. Ten requests per second is a generous ceiling for a lookup service and a hard wall for a dataset build. Every hour of every night, someone on your team will be scheduling around it. Holding the file deletes the entire problem class.
Joins get cheap. If your analysis crosses datasets such as people, companies, jobs, technographics, apps, and developer data, doing the join through APIs means orchestrating calls across all of them. Doing it on files is a SQL statement on tables you already hold.
You get a citable snapshot. A file is a point in time you can name, store, and reproduce. An API answer reflects whatever the system believes today, and if the vendor reclassifies history, your two extracts of "the same" query can disagree. For audits and published research, that difference matters.
Freshness lands where it matters. Hourly files keep the bulk path honest for large-scale work, and the API stays the right tool for the one-record lookup. Teams running both answer "what changed today" and "who is this person" without forcing one tool to do both jobs.
Bottom Line
The license price is the line item vendors compete on. The bill that actually hurts is the one on your side: hours of rate-limited pulls, compute spent re-reading unchanged records, and pipelines built around a ceiling you do not control. Run the workload math before you sign. If your work touches most of the dataset, bulk files are not a delivery preference. They are the difference between a data pipeline and a queue.
MixRank prices both the same so the math stays yours: API access and bulk delivery come with one flat license, refreshed hourly across every dataset, so you can run the file loop, the pull loop, or both, without the delivery method quietly taxing the choice.
Adewale Thomas
Adewale Thomas writes about how technology and workforce data answer the questions buyers actually ask, instead of the ones marketing hopes they wonder about. At MixRank, he spends his time between sales-call transcripts, query logs, and the datasets themselves.
