W1---Opportunity-Intelligence/docs/LRS.md

LRS collector

Run from this repository with Python 3.10 or newer; no dependencies or credentials:

python -m unittest discover -s tests -v
python tools/inspect_lrs.py
python w1.py lrs

On this workstation Python is not on PATH. Commands executed use this interpreter:

& 'C:\Users\Gharke\AppData\Roaming\uv\python\cpython-3.12.11-windows-x86_64-none\python.exe' -m unittest discover -s tests -v
& 'C:\Users\Gharke\AppData\Roaming\uv\python\cpython-3.12.11-windows-x86_64-none\python.exe' w1.py lrs

The sandbox required approval for the installed interpreter and public HTTP. Ordinary local execution requires only Python and HTTPS access.

Public discovery

https://jobs.lrs.com/ exposes ordinary links to 87 distinct requisitions, plus a matching count in section.job-list. Discovery uses this index, not the three seed URLs, guessed IDs, search engines, private APIs, or accounts. robots.txt returned 404: no crawl restrictions were published there. This is public-page access, not a claim of a special license. Each run rechecks robots rules and honors disallow, crawl delay and request rate. Requests are sequential and at least 0.5 seconds apart. No applications, outreach or commitments are made.

HTTP errors, missing title/identity, unexpected content, and index count/layout changes produce a nonzero exit and error manifest. A future paginated index requires explicit implementation; count mismatches fail instead of truncating. Missing optional fields are individually flagged.

Malformed JSON and conflicts

The sample contains schema.org JobPosting. The MIME attribute is application/ld+json; HTMLParser decodes attribute entities, so matching application/ld+json works. The original inspection parser already handled that encoding; the actual remaining defect is invalid JSON inside the script.

Literal newlines in the description string and unescaped HTML attribute quotes make strict json.loads fail. The fallback uses visible job HTML. Complete metadata fields outside the broken description are individually decoded and marked untrusted, never represented as valid repaired JSON. Unrecoverable fields remain absent. Valid JSON-LD objects, arrays and @graph are supported.

AI Architect 46369 has visible added date September 3, 2026 and advertised pay $85–$120/hour, while metadata claims January 18, 2017, expiry March 18, 2017 and USD 40/hour. Visible type Contract also conflicts with metadata FULL_TIME. The collector retains each source value separately and flags review needs. Compensation sentences are preserved verbatim, not labeled verified. Organization mentions identify recruiters in the description, not a confirmed underlying client/employer; metadata organization evidence is also preserved.

http_availability reports HTTP access; hiring_status remains unconfirmed. Neither historical metadata expiry nor an accessible page proves whether hiring is active. New records remain unreviewed.

Outputs and deduplication

Each run creates runs/YYYY-MM-DD/lrs-HHMMSS/ using UTC:

• manifest.json: success/errors, counts, timestamps, request log, snapshot SHA-256 hashes and the pipeline hash used for comparison.
• listing.html, numbered job HTML files, robots text if available: original response bytes, linked from the request log with original/final URLs.
• discovered.json: index URLs and listing evidence before detail extraction.
• lrs.json: all successfully extracted observations and conflicts.
• new_candidates.json: observations absent from the seed pipeline.
• existing_observations.json: fresh evidence alongside original reviewed records.

--pipeline PATH selects another comparison file. --output PATH selects a new directory, which must not already exist. Partial failures preserve successful observations and snapshots but exit 1 with success: false.

Deduplication uses LRS requisition identity, treating the known LRS and Codeworks job hosts as aliases. Unrelated organizations with the same numeric ID remain distinct. Duplicate index links collapse while retaining URLs and evidence. data/opportunities.json is read only: no reviewed status, rationale or exclusion is overwritten. Candidate output is a staging queue, not an automatic pipeline merge. “New” means absent from that pipeline, not newly posted since the previous run. Repeated runs return the same candidates until reviewed and added. Semantic duplicates across different requisition IDs require human review.

Offline tests cover the real saved sample and index, encoded MIME, valid and malformed JSON, missing fields, changed layouts, counts, independent discovery, namespace/alias deduplication and preservation of manual decisions.

Verified execution: September 17, 2026

The original pre-refactor regression command passed all 10 tests (2 USAC and 8 LRS), and the original lrs_discovery.py command exited 0. Their current equivalent commands are shown above. The actual executable was the absolute Python path above. Earlier bare python attempts failed because it is not on PATH; initial sandbox HTTP/interpreter access required escalation.

The successful run is lrs-162640: 16:26:40.825–16:27:51.704 UTC, approximately 71 seconds. It discovered and collected 87 distinct jobs, produced 84 candidates absent from the pipeline, and matched 3 existing records. No HTTP/detail extraction errors occurred. Pipeline SHA-256 before and after was b68aea9436f835375d4f08a46f09fcc06a13da9280a0de7f7faed63913247e00.

85 pages contained malformed JSON-LD. All 87 had conflicting posting dates and historical metadata expiry. 55 had header/metadata employment-type conflicts; 77 had multiple compensation sources requiring review. 10 had no extracted dollar-denominated compensation paragraph and 8 lacked a recognized recruiter mention in the visible description; those missing values are flagged. Snapshots remain available for review, including prose distinctions between contract work and full-time headers that require interpretation.

Examples: 46404, Software Engineer – Backend, Codeworks, Waukesha WI, added September 17, 2026: header Full-time, description advertises a contract at $55–$65/hour, metadata USD 40/hour. 46369, AI Architect is an existing pipeline record with the conflicts described above. Neither example is confirmed actively hiring.

Recommended next step: review and qualify the 84 staged candidates, resolve compensation/engagement conflicts for promising roles, and add approved records to the pipeline before implementing scheduled runs and change detection.