How do I collect tech jobs across the internet?

It's easy to get lost in the world of web scraping these days - there's a huge variety of tools and services out there. But my goal was specific: build a system that's cheap and easy to maintain, and that collects tech jobs from across the internet so I can serve developers fresh, enriched listings.

I started by collecting company profiles from the US and EU (mostly US) and building a simple spreadsheet database with company names and websites. The goal: find active businesses that have remote tech jobs, so I'd know exactly who to crawl.

Once I had that database, the next question was: how do I collect jobs from those companies without building a separate scraper for each one? Maintaining a custom scraper per company website is simply impossible at scale.

The answer: an LLM.
Why not use an LLM to find each company's career page, then extract the raw text from the page -> feed it to the LLM -> get structured JSON back that goes straight into my database?

And yeah - it worked. But there's one major downside you have to admit: LLMs hallucinate, and you won't get perfectly structured data 100% of the time.

So I built a multi-level enrichment pipeline: feed the job description to the LLM -> extract the structured data -> then run a simple regex extractor over the raw text to catch anything the LLM missed and fill in the gaps in the JSON.

How do I crawl and keep tech jobs fresh?

Instead of building a crawler from scratch, I used an open-source crawler that already had everything I needed, paired with BullMQ as the queue system. Each company gets its own job in the queue that periodically checks whether its listings are still live or have been taken down.

That's what I'm trying to deliver:
1. Fresh jobs - checking daily whether each listing is still alive
2. Verified jobs - filtering out ghost jobs so you don't waste time on postings that were never real

How much do I spend on enrichment?

Right now I'm using DeepSeek as my main AI provider, and their pricing works well for me - enriching 250K jobs cost me around $90.

Cache hits are doing a lot of the heavy lifting here. If you're building something similar, it's worth structuring your prompts specifically to maximize cache hit rates.

Are these tech jobs reliable?

Yes. I verify every company before adding it to the database, and every job posting goes through LLM verification with a quality score. At the bottom of each job description you'll find the red flags the LLM detected - things like mismatched seniority level vs. requirements, role mixing, or excessive demands. If the list is empty, the listing looks reasonable.

How am I different from LinkedIn or Indeed?

LinkedIn and Indeed optimize for volume - anyone can post anything, and stale or fake listings can sit there for months. I'm optimizing for the opposite:

1. Every job is checked daily. If it's gone from the company's career page, it's gone from my platform.
2. Ghost jobs get filtered out. Big boards have no incentive to remove them - companies pay to post. I have every incentive to remove them, because wasted applications are the whole problem I'm solving.
3. Every listing is enriched. Salary signals, remote scope, timezone requirements, red flags - extracted and structured so you can filter on what actually matters, instead of reading 40 paragraphs to find out the "remote" job requires 3 days in an office in Ohio.

I'm not trying to have the most jobs. I'm trying to have the jobs that are actually worth applying to.