Salary.com · Present
Job Data Pipeline
Job data crawling and AI processing
- Role
- Software developer
- Stack
Overview
C#/.NET tooling that crawls and normalises job postings from many sources, then an LLM pipeline that turns thousands of records a day into structured, validated data.
Company work: details stay at the level I can share publicly.
What I built
- C#/.NET tooling to crawl and normalise job postings from multiple sources, with rate limiting, retries and deduplication.
- An automated LLM prompt pipeline that extracts structured fields from thousands of job records daily, with batching, schema validation and fallback handling.
- Unit tests, prompt regression against a golden dataset and integration tests, plus monitoring of pipeline quality and cost.
System design
Polite, resilient crawlers
Per-source rate limits and retries keep collection steady, and normalisation plus deduplication turn many formats into one clean record.
Batched LLM extraction
Records are sent in batches to keep throughput high and cost predictable.
Schema validation and fallbacks
Every model response is validated against a schema. Invalid output goes through fallback handling instead of polluting the dataset.
Prompts under test
A golden dataset catches prompt regressions before they ship, next to unit and integration tests for the code around them.
Gallery
