Job Description

Client: Adobe

Location: USA (remote, must be able to work Mountain Time hours)

Duration: 2-4 months to start

Title: Data Engineer

Key Skills: Data Engineer, Spark, SQL, Experience building production ETL/data-integration pipelines, schema, Databricks

Cybersecurity Experience/ security schemas/ vulnerability scanners would be a HUGE plus

Description:

Data Engineer-Vulnerability Data Integration Engineer

Deliverables

Two production-ready source integrations (ingestion, normalization, merge, ticketing) with validation/reconciliation checks and documentation.

Summary

We operate a large-scale security data platform on a modern lakehouse that normalizes vulnerability findings from multiple scanners into a common, standardized schema and drives downstream ticketing. We are onboarding two new scanner sources - one cloud-security-posture scanner and one web-application (DAST) scanner - and need an engineer to build and productionize those integrations end to end.

Key Responsibilities

• Design and build incremental ingestion pipelines (watermarking, deduplication) for the two new sources against upstream data tables.
• Map each source's native findings into the platform's common security schema, including severity/risk normalization and first-seen/last-seen handling.
• Solve asset/target correlation for each source - associating findings with the right infrastructure or application entities and their ownership.
• Integrate findings into downstream ticketing and exception-handling workflows, including which findings are ticketable and how ownership is attributed.
• Merge new sources into a central large-scale table safely (schema evolution, unique-key handling, collision checks) and validate propagation into reporting/history views.
• Reconcile output against each source's console (counts, coverage, dedup), and produce documentation and operational runbooks.

Work as an embedded member of the existing security data engineering team (not an isolated solo contributor) - collaborating day-to-day with the platform/security engineering team and other team members on shared schema conventions, merge patterns, and table ownership so new-source integration work lands safely alongside everyone else's changes.

Required Qualifications

• Strong distributed data-processing skills (Spark or equivalent) and advanced SQL; comfortable with large-table upserts/MERGE, schema evolution, and incremental/watermark patterns.
• Experience building production ETL/data-integration pipelines that normalize heterogeneous sources into a shared schema.
• Solid data modeling; able to reason about join fan-out, deduplication, and entity correlation at large scale.
• Self-directed, but collaborative - comfortable working as an embedded member of a larger existing engineering team, coordinating with other team members, and taking an integration from upstream discovery through validated production output.

Proficient in SQL (complex joins, window functions, CTEs) and Python (PySpark) for building and maintaining production data pipelines.

Hands-on experience with Databricks (notebooks, jobs/workflows, SQL warehouses) and Delta Lake tables - MERGE/upsert patterns, schema evolution, clustering/partitioning, and table maintenance (OPTIMIZE, VACUUM).

Preferred

• Experience integrating commercial security/vulnerability scanners and/or working with standardized security schemas.
• Ticketing/workflow-automation integration experience (e.g., issue trackers).
• Application-security (DAST) domain awareness for the web-scanner workstream.

Job Tags

Similar Jobs