Associate Director, Data Engineering

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us. About the Organization The Clinical & Non-Clinical Data Organization at Eli Lilly and Company is responsible for the design, build, and operation of enterprise data platforms that power drug discovery, clinical development, and regulatory submissions. Data Hub is building a robust Data Strategy to make Lilly's Clinical and Non-Clinical data AI-ready and audit-ready, delivering scalable, governed, and reusable data products that accelerate how medicines reach patients. Our data engineering organization sits at the intersection of science, technology, and patient impact — connecting Clinical and Non-Clinical data across the full chain, from ingestion to consumption. Job Description The Associate Director, Data Engineering is a senior product-ownership, platform-management, and people-leadership role that owns the quarterly roadmap and backlog for one or more data engineering delivery teams, the health and cost of the underlying data platform, and the data strategy and AI roadmap for a Databricks-oriented practice. Hands-on data engineering experience is expected and non-negotiable — this is a practitioner who has built pipelines, models, and platforms directly, and stays technically current enough to review architecture and pressure-test build-vs-buy calls first-hand. This role defines and implements the data KPI and metrics framework the organization runs on, enables AI adoption across data teams beyond its own delivery pods, and owns the data products built on the Databricks lakehouse. A further mandate is accelerating data onboarding and access, so new sources and consumers move from request to trusted, governed availability in days, not months. As an M1-level people leader, this individual manages a team of data engineers and/or product owners and partners closely with architecture, platform, and senior business leaders on scope and multi-year direction. Core Responsibilities Strategic Leadership & Data/AI Roadmap (Databricks-Oriented) Proactively surface strategic topics to senior leadership based on first-hand data engineering experience — platform constraints, technical debt, build-vs-buy trade-offs, and emerging AI/engineering tooling — rather than waiting to be asked. Define the data strategy and AI roadmap for a Databricks-oriented data engineering team, identifying the data products, lakehouse assets, and AI/ML capabilities that matter most over the next 12–24 months. Own the product roadmap for Databricks-native capabilities — lakehouse architecture, Unity Catalog governance, Delta Live Tables pipelines, MLflow/feature-store patterns— with hands-on proof points, not vendor-slide recommendations. Product Ownership & Roadmap Leadership Own sprint- and quarter-level backlog prioritization applying product-management rigor (RICE/WSJF-style scoring, stakeholder discovery, success metrics) to balance business value, technical debt, effort, and risk. Bring forward strategic topics — informed by direct engineering experience — that shape the roadmap rather than simply respond to it (e.g., platform limitations, emerging AI capabilities). Facilitate PI Planning, backlog grooming, and sprint reviews; serve as the voice of both the internal customer and the platform in delivery ceremonies. Platform Management Own the data platform as a product — define its roadmap, SLAs/OLAs, and reliability targets, treating internal engineering teams and data consumers as its customers. Manage platform capacity, cost, and performance (compute/storage sizing, FinOps discipline, cost-per-workload tracking) to keep the platform scalable and cost-efficient. Standardize and rationalize the tooling landscape — reduce redundant components and own the platform's build-vs-buy and vendor evaluations. Own lifecycle management of platform components — versioning, deprecation, and migration planning — so platform evolution doesn't disrupt delivery teams. Data KPIs & Metrics Define and implement the data KPI and metrics framework for the organization — data quality, pipeline reliability, freshness/latency, lineage coverage, and cost-to-serve — with clear owners and targets. Instrument dashboards and reporting (e.g., in Databricks AI/BI or equivalent) so KPIs are visible to engineering teams and business stakeholders, not buried in a backlog tool. Tie KPIs to business outcomes and review trends regularly with the team and stakeholders, using them to drive continuous improvement, not static reporting. AI Adoption Enablement Across Data Teams Champion and enable AI adoption across data teams enterprise-wide, not only within owned pods — building the playbooks, training, and support that let other teams adopt AI-assisted engineering with confidence. Stand up a community of practice for AI-assisted and agentic engineering across data teams, sharing reusable patterns, guardrails, and lessons learned. Track engineering efficiency and AI-adoption metrics (cycle time, automation coverage, tool adoption rate) and prioritize backlog items that improve them. Data Acceleration & Solution Delivery Prioritize data product delivery work that reduces time-to-data and time-to-insight for downstream consumers. Own backlog items that advance Data Capability Maturity — data contracts, self-service access, and reusable data products at scale. Data Onboarding & Access Acceleration Redesign and own the data onboarding pipeline so new sources are cataloged, quality-checked, and available as trusted data products in days, not months. Streamline data access provisioning — self-service, role/attribute-based requests (e.g., via Unity Catalog), and automated approvals — to cut time-to-access. Track onboarding and access-acceleration metrics (time-to-onboard, time-to-access) and prioritize work that measurably shortens them. People Leadership Directly manage a team of data engineers and/or product owners, including hiring, onboarding, and performance appraisals. Coach team members on product thinking and platform ownership, and serve as their career advisor, partnering with HR on development and IDPs. Stakeholder Engagement & Collaboration Partner with architecture, test engineering, and business/product leaders to align priorities and dependencies; represent the team in PI Planning. Communicate roadmap, platform trade-offs, and priorities clearly to both technical and business audiences. Process Leadership & Continuous Improvement Implement and refine a standardized framework for backlog management, platform governance, and delivery execution (Agile/Kanban). Contribute to a growing community of practice among data product owners and platform leads; support Lill