Note: The job is a remote job and is open to candidates in USA. Cognizant is a leading technology company that helps clients modernize technology and transform experiences. They are seeking a highly skilled Azure Databricks Administrator to manage and optimize enterprise-scale Azure Databricks environments, ensuring security and governance while collaborating with various teams.
Responsibilities
- Design, deploy, and administer Azure Databricks workspaces utilizing VNet injection, private endpoints, and secure cluster connectivity
- Configure and maintain cluster policies, Databricks Runtime versions, autoscaling, and workload management standards
- Optimize platform performance through cluster tuning, Spark configuration management, and Delta Lake best practices
- Perform capacity planning and resource forecasting to support enterprise growth
- Drive cost optimization initiatives including right-sizing, auto-termination policies, reserved instances, and spot instance utilization
- Troubleshoot and resolve complex platform, cluster, job execution, and connectivity issues
- Design, implement, and administer Unity Catalog metastores, catalogs, schemas, external locations, and volumes
- Establish and enforce fine-grained access controls across enterprise data assets
- Support data lineage tracking, auditing, and governance requirements
- Govern managed and external data assets while promoting best practices for data management
- Integrate Azure Databricks with Microsoft Entra ID for authentication and Single Sign-On (SSO)
- Configure and manage SCIM provisioning for automated user and group lifecycle management
- Implement secure secret management using Azure Key Vault-backed secret scopes
- Define and manage access controls across Unity Catalog, Databricks workspaces, and Azure Data Lake Storage Gen2
- Support security audits, compliance reviews, and governance initiatives
- Architect and maintain secure networking solutions utilizing VNets, NSGs, private endpoints, and firewall controls
- Ensure secure connectivity between Databricks, Azure services, and on-premises environments
- Manage DNS resolution, routing configurations, and network security best practices
- Design and implement Azure Data Factory (ADF) integration patterns with Azure Databricks
- Support orchestration of enterprise data pipelines and notebook execution workflows
- Partner with DevOps teams to establish and maintain CI/CD pipelines
- Utilize Databricks REST APIs and automation frameworks to streamline platform administration
- Provide guidance on Azure Data Lake Storage Gen2 architecture and configuration
- Implement secure access patterns using service principals, credential passthrough, and Microsoft Entra ID integration
- Troubleshoot storage connectivity and access management issues
- Implement monitoring and alerting using Azure Monitor, Log Analytics, and Databricks native monitoring tools
- Monitor service availability, platform health, resource utilization, and job performance
- Support disaster recovery, business continuity, and resiliency planning efforts
- Plan and coordinate Databricks Runtime upgrades and platform maintenance activities
- Communicate service impacts and operational risks to stakeholders
- Develop and maintain architecture documentation, diagrams, standards, and operational runbooks
- Establish best practices for platform governance, security, cost optimization, and performance management
- Provide technical guidance, training, and knowledge transfer to internal teams
- Collaborate with data engineering, data science, infrastructure, networking, security, and operations teams
- Act as the subject matter expert for Databricks platform administration and governance
- Participate in architecture reviews and provide technical leadership for strategic initiatives
Skills
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent work experience
- 5+ years of experience supporting cloud-based data and analytics platforms
- Hands-on experience administering Azure Databricks in enterprise environments
- Strong experience with Azure Databricks
- Strong experience with Azure Data Factory (ADF)
- Strong experience with Azure Data Lake Storage Gen2 (ADLS Gen2)
- Strong experience with Microsoft Entra ID
- Strong experience with Azure Key Vault
- Strong experience with Azure Networking (VNets, NSGs, Private Endpoints)
- Strong experience with Azure Monitor and Log Analytics
- Experience implementing and managing Unity Catalog
- Strong understanding of cloud security, RBAC, IAM, encryption, and governance principles
- Experience with SCIM-based identity provisioning
- Proficiency in Python, SQL, PowerShell, and Azure CLI
- Experience supporting CI/CD initiatives using Azure DevOps, GitHub Actions, or similar platforms
- Excellent analytical, troubleshooting, and problem-solving skills
- Strong verbal and written communication skills
- Experience supporting large-scale enterprise Data & AI platforms
- Knowledge of Spark architecture, Delta Lake optimization, and Databricks APIs
- Experience with Infrastructure as Code (IaC) tools and cloud automation frameworks
- Experience working within regulated or highly governed environments
- Microsoft Certified: Azure Solutions Architect Expert
- Databricks Certified Data Engineer Associate or Professional certification
Benefits
- Medical, Dental, Vision, and Life Insurance
- Short-Term and Long-Term Disability Coverage
- 401(k) Plan with Company Contributions
- Health Savings Account (HSA) and Flexible Spending Accounts (FSA)
- Paid Time Off (PTO), Company Holidays, and Sick Leave
- Paid Parental Leave
- Employee Assistance Program (EAP)
- Professional Development and Certification Opportunities
- Employee Wellness and Recognition Programs
- Benefits are subject to eligibility requirements and may vary by location and employment status.
Company Overview