LinkedIn Extractor

About this project

LinkedIn Extractor is a high-performance browser automation tool that captures full-page LinkedIn profile screenshots and extracts structured JSON profiles using GPT-4o vision.

The Challenge

Standard HTML web scraping for LinkedIn is incredibly fragile because the site changes its DOM structure constantly and deploys strong anti-scraping measures. Parsing text profiles via typical selectors breaks weekly, requiring constant developer maintenance.

The Solution

We designed a visual extraction pipeline that relies on browser screenshots rather than HTML structure. A headless Chrome instance orchestrated by Selenium captures high-definition full-page screenshots of LinkedIn profiles, which are then analyzed by OpenAI GPT-4o Vision to extract name, education, work experience, and contact info in a structured JSON payload.

Headless Chrome orchestration with anti-detection

Headless Chrome orchestration with anti-detection headers via Selenium.

High-resolution full-page screenshot stitches of

High-resolution full-page screenshot stitches of dynamic profile pages.

Vision AI parsing pipeline extracting

Vision AI parsing pipeline extracting detailed career histories.

Structured output validator enforcing rigorous

Structured output validator enforcing rigorous JSON schemas.

AI Intelligence

GPT-4o Vision model parses raw visual screen pixels, understanding complex multi-column layouts and formatting changes without relying on HTML selectors.

System Architecture

Python FastAPI backend that exposes webhook endpoints to run scraper workers and trigger structured GPT-4o calls.

Database & Caching

MongoDB stores extraction histories and raw images, cached with Redis for deduplicating scraper runs.

Messaging

Webhooks send structured JSON outputs directly to target client databases, with Slack alerts for execution summaries.

Frontend

Backend

Deployment

Containerized in Docker, running on AWS ECS with custom Chromium buildpacks.