Data Integration Engineer
virtusa
Job Description
Python - Advanced level (pandas, requests, boto3 libraries)
• API Integration - REST APIs, authentication (OAuth, API keys), error handling
• Cloud Platforms - AWS (S3, EC2, IAM) proficiency
• Databricks/Spark - PySpark, Delta Lake, SQL optimization
• Data Pipeline Tools - Airflow, Databricks Workflows, or equivalent orchestration
• SQL - Complex queries, window functions, performance tuning
Security & Enterprise Knowledge
• Understanding of security telemetry (logs, alerts, events, metrics)
• Familiarity with at least 2-3 security platforms (JAMF, Intune, Qualys, CrowdStrike,
Okta)
• Experience with AWS CloudTrail or similar cloud audit logs
• Knowledge of data security and compliance (encryption, access controls)
Additional Competencies
• Version control (Git)
• CI/CD pipeline familiarity
• Data quality and validation frameworks
• Performance tuning and optimization mindset
Education Qualificaiton
BE, BTech, MCA
Roles & Responsibilities
Data Integration & Connectivity
• Design and build connectors between enterprise security platforms (JAMF, Intune, Qualys,
CrowdStrike, Okta, AWS CloudTrail) and Databricks
• Establish secure API connections and authentication mechanisms with each source system
• Implement data extraction logic using REST APIs, SDKs, and native connectors
• Handle varying data formats and transformation requirements across heterogeneous sources
Data Pipeline Development
• Develop automated Python scripts for continuous data ingestion
• Build orchestration workflows using Apache Airflow, Databricks Workflows, or similar tools
• Implement error handling, retry mechanisms, and monitoring for pipeline reliability
• Ensure near real-time or scheduled data synchronization based on requirements
AWS S3 & Data Staging
• Configure S3 bucket architecture and data partitioning strategy
• Land raw security telemetry data in S3 with proper folder structures
• Implement lifecycle policies and data retention strategies
• Ensure proper encryption and access controls
Databricks Platform Configuration
• Create and optimize Databricks tables
Qualifications
Mandatory Skills
Databricks & Python
Job Description
Core Technical Skills
• Python - Advanced level (pandas, requests, boto3 libraries)
• API Integration - REST APIs, authentication (OAuth, API keys), error handling
• Cloud Platforms - AWS (S3, EC2, IAM) proficiency
• Databricks/Spark - PySpark, Delta Lake, SQL optimization
• Data Pipeline Tools - Airflow, Databricks Workflows, or equivalent orchestration
• SQL - Complex queries, window functions, performance tuning
Security & Enterprise Knowledge
• Understanding of security telemetry (logs, alerts, events, metrics)
• Familiarity with at least 2-3 security platforms (JAMF, Intune, Qualys, CrowdStrike,
Okta)
• Experience with AWS CloudTrail or similar cloud audit logs
• Knowledge of data security and compliance (encryption, access controls)
Additional Competencies
• Version control (Git)
• CI/CD pipeline familiarity
• Data quality and validation frameworks
• Performance tuning and optimization mindset
Education Qualificaiton
BE, BTech, MCA
Roles & Responsibilities
Data Integration & Connectivity
• Design and build connectors between enterprise security platforms (JAMF, Intune, Qualys,
CrowdStrike, Okta, AWS CloudTrail) and Databricks
• Establish secure API connections and authentication mechanisms with each source system
• Implement data extraction logic using REST APIs, SDKs, and native connectors
• Handle varying data formats and transformation requirements across heterogeneous sources
Data Pipeline Development
• Develop automated Python scripts for continuous data ingestion
• Build orchestration workflows using Apache Airflow, Databricks Workflows, or similar tools
• Implement error handling, retry mechanisms, and monitoring for pipeline reliability
• Ensure near real-time or scheduled data synchronization based on requirements
AWS S3