Senior Data Engineer
6 - 10 years
Philadelphia, PA
Full-time
Hybrid
1 month ago
About the role
We are seeking an experienced Senior Data Engineer to serve as the onshore technical lead for the EXL augmented support team. In this role, you will lead a team of 6–8 Data Engineers (onshore/offshore) through the full repointing lifecycle: SQL syntax conversion (Teradata to Snowflake), Airflow variable and connection updates, script modification, parallel run validation, and production cutover.
KEY RESPONSIBILITIES
Repointing Execution & Delivery
Team Leadership
REQUIRED SKILLS & EXPERIENCE
Technical Skills (Must Have)
Delivery & Leadership Skills
PREFERRED BACKGROUND
WHAT WE'RE LOOKING FOR
A hands-on technical lead who can walk into a live migration program, immediately understand script-level complexity, own execution accountability, and keep a distributed team moving at pace — without needing the SI Partner to hold their hand.
KEY RESPONSIBILITIES
Repointing Execution & Delivery
- Lead script-level repointing across 593 downstream scripts (PySpark, BTEQ, Shell, SQL, IICS, SSIS) and 154 Databricks MLOps scripts from Teradata/SQL Server/Minio to Snowflake.
- Execute SQL syntax conversion for Teradata BTEQ scripts to Snowflake-compatible SQL, leveraging utility frameworks provided by the SI Partner (Infosys).
- Drive Airflow variable and connection updates across all migrated workflows, ensuring pipelines point to Snowflake endpoints post-cutover.
- Manage the script modification and change management process, coordinating production-to-parallel sync within the required 1-business-day SLA.
- Own the deployment and sign-off process for each migration wave, maintaining runbooks and go/no-go checklists per domain group.
- Manage data copy requests for Minio-related assets and coordinate initial load and incremental sync activities with the SI Partner.
Team Leadership
- Lead and coordinate a team of 6–8 Data Engineer I and Data Engineer II resources (onshore and offshore) across parallel migration workstreams.
- Assign scripts per complexity band (S/M/L) to team members, track daily progress against the LOE estimates, and escalate blockers promptly.
- Review code changes, validate SQL conversion quality, and enforce engineering standards before submission to EBI team for review.
- Mentor junior engineers on Snowflake-specific constructs (virtual warehouses, stages, Snowpark, streams & tasks) and Teradata-to-Snowflake syntax differences.
- Conduct daily stand-ups and provide weekly migration status reports to the Program Manager and client stakeholders.
REQUIRED SKILLS & EXPERIENCE
Technical Skills (Must Have)
- 8+ years of data engineering experience; minimum 3 years hands-on with Snowflake (architecture, SQL dialect, performance tuning, RBAC).
- Proven experience migrating Teradata workloads to Snowflake — including BTEQ/ANSI SQL translation, macro conversion, and stored procedure migration.
- Strong PySpark and Python skills; experience working with Databricks is required (MLOps pipelines a strong plus).
- Working knowledge of Apache Airflow — DAG management, variable configuration, connection updates across environments.
- Familiarity with IICS (Informatica Intelligent Cloud Services) and/or SSIS (.dtsx) pipelines in a migration context.
- Solid understanding of data lake and object store patterns (Minio, S3, ADLS) and how they integrate with Snowflake external stages.
- Experience with SQL Server, Teradata, and/or Netezza as source platforms is required.
- Strong Git proficiency; experience with CI/CD pipelines and code change coordination across environments (dev → parallel → prod).
Delivery & Leadership Skills
- Proven experience leading a team of 5+ engineers in a fast-paced, time-bound migration or repointing program.
- Ability to manage scope across complexity bands (S/M/L scripts) and balance team capacity against tight delivery timelines.
- Strong stakeholder management — able to communicate migration status, risks, and escalation paths clearly to both technical and business audiences.
- Experience working in a client-facing consulting or managed services model is strongly preferred.
- Comfortable coordinating with an SI Partner — understanding scope boundaries, handoff points, and utility dependencies without micromanaging.
PREFERRED BACKGROUND
- Domain experience in US Telecom — CDR processing, subscriber data, network inventory, or revenue assurance datasets aligned to EBI workloads (NRD, Compensation, TechOps, CB, etc.).
- Snowflake SnowPro Core or Advanced Data Engineer certification.
- Prior experience delivering migrations , consulting firm, or analytics-driven managed services engagement.
- Exposure to data observability and reconciliation tooling (Great Expectations, Monte Carlo, or custom validation frameworks).
- Familiarity with Snowflake Snowpark, Streams & Tasks, or Dynamic Tables for replacing legacy Teradata batch patterns.
- Experience with Databricks Unity Catalog or MLOps frameworks in an enterprise telecom context.
WHAT WE'RE LOOKING FOR
A hands-on technical lead who can walk into a live migration program, immediately understand script-level complexity, own execution accountability, and keep a distributed team moving at pace — without needing the SI Partner to hold their hand.
- Someone who can read a Teradata BTEQ script and a Snowflake execution plan in the same breath.
- A communicator who can run a 15-minute daily sync with EBI stakeholders and translate yesterday’s reconciliation failures into a clear resolution path.
- A professional who understands the rhythm of parallel run validation — tolerates expected data mismatches, drives iterative reconciliation, and knows when to escalate versus resolve.
Skills
EngineeringInformation TechnologyHospitals and Health Care