Cloud Engineer
wlgore
Job Description
- Work closely with Solution Architects, Data Engineers, Data Analysts, and Data Scientists to support their data processing needs and use cases.
- Set up, configure, and manage Azure Databricks workspaces and clusters to support data processing and analytics needs, as well as monitor, optimize, and balance the performance and cost efficiency of Databricks clusters, by scaling resources and managing cluster configurations.
- Administer data integration and governance capabilities across Azure Databricks, inclusive of Unity catalog, Microsoft Fabric, and Power BI to ensure trusted, compliant data access.
- Implement and manage infrastructure as code using Bicep templates to deploy and maintain Azure resources consistently and securely.
- Ensure timely execution of data pipelines by creating and managing job schedules in Databricks to automate data processing and ETL workflows.
- Implement security measures (e.g., access control, encryption, auditing) to comply with required data protection policies and regulations.
- Apply AI Ops and platform automation techniques to proactively identify performance issues, reduce operational incidents, and improve system reliability.
- Partner with applicable teams to monitor the health and performance of data pipelines and clusters, troubleshoot issues and resolve errors to maintain system reliability.
- Manage user access and permissions ensuring appropriate access levels are assigned to users and roles.
- Execute routine platform maintenance including upgrades, performance tuning, backup, and disaster recovery planning, in addition to implementing data backup and disaster recovery strategies to ensure data availability and continuity in the event of an outage or failure.