Job Radar

Two sources, one record shape

Remotive supplies JSON with job descriptions. Python.org provides an HTML listing page. Its importer extracts the available fields and categories without following individual job pages.

Both importers produce the same record shape. Source-specific company and location fields become shared fields. Remotive descriptions are converted from HTML to plain text, removing scripts, styles, and other non-content elements while retaining readable text.

Both sources use title keywords to infer seniority, with explicit precedence when several labels match. An unrecognized title becomes unknown. Remotive records are marked remote; Python.org’s remote flag comes from title and location wording, including checks for explicit negations.

Malformed records are skipped. A failed request or an unrecognized source response fails the import. These are separate outcomes: incomplete data should not be mistaken for a successful fetch from a broken source.

What counts as a duplicate

For the MVP, the practical priority was preventing repeated imports from creating duplicate offers.

The shared persistence function checks incoming URLs against stored records and URLs already handled in the batch. A database unique constraint provides a second check. Each insert uses a savepoint, allowing a recognized URL conflict to be skipped while the remaining records are processed.

Identity is based on the exact URL. Existing URLs are skipped without updating their records. Tracking parameters and URL aliases are not unified.

This prevents repeated insertion of the same URL, but it does not identify every duplicate vacancy. The same role can appear on different websites with different URLs—and possibly different titles—and remain as separate records.

From stored records to API responses

SQLAlchemy maps a single job model to PostgreSQL, with schema changes managed through Alembic. The stored timestamp records when an offer was added locally, not when the source published it.

The API supports remote and seniority filters, text search, and limit/offset pagination. Text search checks title, company, location, and description; search wildcard characters are treated literally. Results can be ordered by date added or match score, with explicit tie-breaking rules.

OpenAPI and Swagger expose the request and response structure. The frontend reads the same listing endpoint, keeping retrieval and ranking behavior in the backend.

Imports run through protected HTTP endpoints or a one-shot CLI. The CLI attempts both sources with separate database sessions, so one source failure does not stop the other attempt.

A score for the roles I’m looking for

The score uses four inputs: title, description, remote status, and seniority.

A relevance check runs first. A record needs a matching developer-role title or a Python, FastAPI, or SQLAlchemy signal before it receives points. Eligible records gain points for technology matches, a relevant title, remote work, and junior or intern seniority. Senior and leadership levels subtract points.

Python contributes 20 points; FastAPI and PostgreSQL contribute 10 each. Each technology counts once, so repeated keywords cannot inflate the score. URLs and source metadata are removed from the scoring text.

The result is calculated when records are returned, rather than stored. The same inputs under the same rules produce the same score, bounded between 0 and 100. It expresses keyword relevance to this profile, not hiring probability or semantic suitability.

Ordering exposes a practical trade-off. Date ordering and pagination run in SQL. Score ordering loads all filtered candidates, calculates their scores in Python, sorts them, and then selects the requested page.

What is verified

The pytest suite covers normalization, repeated imports, URL conflicts, scoring rules, filters, pagination, and ordering. It also checks malformed source responses, import authentication, and source-failure isolation in the CLI.

External responses are mocked, and database tests use SQLite. PostgreSQL migration SQL is rendered and the driver is checked, but the suite does not exercise a live PostgreSQL server.

GitHub Actions is configured to run pytest under Python 3.12 on pushes and pull requests targeting main. Docker Compose supplies a local application and PostgreSQL environment; database migrations remain an explicit step.

Where the current approach stops

• Existing offers are not refreshed. Reimports leave stored content unchanged, and offers are not removed when they disappear from a source.
• Source detail varies. Python.org listing summaries provide fewer scoring signals than full descriptions. Keyword-based seniority and remote inference can miss ambiguous wording.
• Score sorting processes every filtered candidate. Its work grows with that result set, even when the requested page is small.

These are boundaries of the current implementation, not all deliberate strategies established at the beginning.

What I learned, and what I would change next

Scoring and duplicate handling were the most educational parts of the project. They required working with both the data available and the limits of what it could establish.

I would add more job sources next: more useful offers, but also more varied data to test normalization, deduplication, and scoring assumptions. That variety could expose edge cases that do not appear with only two sources.