Sync Only New Jobs
Use the postedAfter parameter to fetch only jobs posted since your last sync, reducing API calls and processing time.
Instead of fetching all jobs every time, use postedAfter to only get jobs posted since your last sync. This reduces API calls and processing time.
How It Works
The postedAfter parameter filters jobs to those posted after a Unix timestamp (seconds):
# Jobs posted in the last 24 hours
curl -H "X-API-Key: $JOBVEN_API_KEY" \
"https://api.jobven.com/v1/public/jobs?postedAfter=1735257600"
The Pattern
- First sync: Fetch all jobs (or start from a recent date)
- Save timestamp: Record when you started the sync
- Next sync: Use saved timestamp in
postedAfter - Repeat: Each sync only fetches new jobs
Examples
JavaScript
import fs from 'fs';
const TIMESTAMP_FILE = './last_sync.txt';
function getLastSync() {
try {
return parseInt(fs.readFileSync(TIMESTAMP_FILE, 'utf8'));
} catch {
// First run - start from 7 days ago
return Math.floor(Date.now() / 1000) - (7 * 24 * 60 * 60);
}
}
async function syncNewJobs() {
const lastSync = getLastSync();
const syncStart = Math.floor(Date.now() / 1000);
const jobs = [];
let cursor = null;
do {
const params = new URLSearchParams({
postedAfter: lastSync.toString(),
limit: '100'
});
if (cursor) params.set('cursor', cursor);
const response = await fetch(
`https://api.jobven.com/v1/public/jobs?${params}`,
{ headers: { 'X-API-Key': process.env.JOBVEN_API_KEY } }
);
const { data, meta } = await response.json();
jobs.push(...data);
cursor = meta.nextCursor;
} while (cursor);
// Process jobs here (save to database, etc.)
// Save timestamp for next sync
fs.writeFileSync(TIMESTAMP_FILE, syncStart.toString());
console.log(`Synced ${jobs.length} new jobs`);
return jobs;
}
Python
import os
import time
import json
from pathlib import Path
import requests
TIMESTAMP_FILE = Path('./last_sync.json')
def get_last_sync():
try:
return json.loads(TIMESTAMP_FILE.read_text())['timestamp']
except (FileNotFoundError, KeyError):
# First run - start from 7 days ago
return int(time.time()) - (7 * 24 * 60 * 60)
def sync_new_jobs():
last_sync = get_last_sync()
sync_start = int(time.time())
jobs = []
cursor = None
while True:
params = {'postedAfter': last_sync, 'limit': 100}
if cursor:
params['cursor'] = cursor
response = requests.get(
'https://api.jobven.com/v1/public/jobs',
headers={'X-API-Key': os.environ['JOBVEN_API_KEY']},
params=params
)
response.raise_for_status()
data = response.json()
jobs.extend(data['data'])
cursor = data['meta'].get('nextCursor')
if not cursor:
break
# Process jobs here (save to database, etc.)
# Save timestamp for next sync
TIMESTAMP_FILE.write_text(json.dumps({'timestamp': sync_start}))
print(f"Synced {len(jobs)} new jobs")
return jobs
Tips
- Handle duplicates - Jobs near the sync boundary may appear twice. Use upsert operations when saving to your database.
- Add error handling - Don't update the timestamp if the sync fails, so the next run retries from the same point.
- Schedule it - Run the sync on a schedule (cron, task scheduler, etc.) appropriate to your needs.