Note: The job is a remote job and is open to candidates in USA. SoftStandard Solutions is seeking a Data Engineer to work on developing and maintaining data pipelines and ETL workflows. The role involves collaborating with various teams to deliver high-quality data solutions and ensuring data security and compliance standards are met.
Responsibilities
- Design, build, and maintain scalable data pipelines and ETL workflows
- Develop and optimize data ingestion processes from multiple data sources
- Build and maintain data lakes and data warehouses
- Optimize SQL queries and improve data processing performance
- Collaborate with Data Scientists, Analysts, and Software Engineers to deliver high-quality data solutions
- Implement data validation, monitoring, and quality checks
- Troubleshoot production issues and ensure high availability of data pipelines
- Ensure data security, compliance, and governance standards are followed
- Automate repetitive data engineering tasks and deployment processes
- Document data architecture, workflows, and technical specifications
Skills
- 5+ years of experience as a Data Engineer
- Strong proficiency in Python and SQL
- Experience with ETL/ELT development and data integration
- Hands-on experience with Apache Spark, PySpark, and Kafka
- Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform (GCP)
- Knowledge of data warehousing concepts (Snowflake, Redshift, BigQuery, Synapse)
- Experience with workflow orchestration tools like Apache Airflow
- Familiarity with relational and NoSQL databases (PostgreSQL, MySQL, MongoDB, DynamoDB)
- Understanding of data modeling, data governance, and data quality best practices
- Experience with version control tools such as Git
- Strong analytical, troubleshooting, and problem-solving skills
- Experience with Databricks
- Knowledge of CI/CD pipelines for data engineering
- Exposure to containerization technologies like Docker and Kubernetes
- Experience working in Agile/Scrum environments
- Experience with dbt
- Knowledge of Terraform or Infrastructure as Code (IaC)
- Experience with streaming technologies and real-time data processing
- Exposure to machine learning data pipelines
Company Overview
Company H1B Sponsorship