AWS Data Engineer Python, Dataiku (San Francisco) Job at Compunnel, Inc., San Francisco, CA

REtoc0hudUhZSjRjNmJwd2t3SlovWWJmZnc9PQ==
  • Compunnel, Inc.
  • San Francisco, CA

Job Description

We are looking for an experienced AWS Data Engineer to design, develop, and maintain scalable data pipelines using AWS native technologies.

This role involves working closely with technical analysts, stakeholders, data scientists, and other team members to ensure data quality, optimize storage solutions, and support data platform modernization efforts.

The ideal candidate will have hands-on experience with cloud migration, data transformation, and automation strategies.

Key Responsibilities

  • Lead and support data platform modernization projects.
  • Design and develop robust, scalable data pipelines using AWS native services.
  • Optimize ETL processes for efficient data transformation and performance.
  • Migrate workflows from on-premise environments to AWS cloud, ensuring data quality and consistency.
  • Design automations and integrations to resolve data inconsistencies and quality issues.
  • Perform system testing and validation to ensure successful integration and functionality.
  • Implement security and compliance controls within the cloud environment.
  • Conduct data validation checks pre- and post-migration to ensure completeness, consistency, and accuracy.
  • Collaborate with data architects and developers to document manual workflows and design automation strategies.
Required Qualifications
  • Minimum of 7 years of experience in data engineering using AWS technologies (AWS Glue, Python, Snowflake, S3, Redshift).
  • Strong experience in designing and developing scalable data pipelines.
  • Proficiency in Snowflake for data transformations and ETL optimization.
  • Experience with both streaming and batch data pipeline architectures.
  • Familiarity with DataOps concepts and CI/CD pipeline setup on AWS.
  • Hands-on experience with Databricks and a willingness to expand capabilities.
  • Knowledge of Dataiku for data preparation and workflow automation.
  • Strong problem-solving and analytical skills.
  • Bachelors or Masters degree in Computer Science or a related field.
Preferred Qualifications
  • Experience with AWS EMR, Lambda, and other cloud-based data engineering tools.
  • Exposure to data governance and compliance frameworks.
#J-18808-Ljbffr

Job Tags

Full time,

Similar Jobs

Adecco US, Inc.

VFR Pilot Job at Adecco US, Inc.

**VFR Travel Pilot - Monroeville, AL****Be the Difference. Fly to Save Lives.****Sign-On Bonus: Up to $100,000****Location: Monroeville...  ...**cutting-edge equipment**+ Work alongside **healthcare and aviation professionals** who share your passion+ Experience the reward... 

Gifted Healthcare

Travel MedSurg Licensed Practical Nurse (LPN) Job at Gifted Healthcare

 ...travel assignmentsfor all overtime hours worked (excluding local, PRN, CA, Govt assignments). Weekly Pay - We pay you weekly so you...  ...your life. Responsibilities Licensed Practical Nurses (LPN) plan and manage patient care specific to each patient. The LPN... 

Amada Senior Care NorthShore

Polish Speaking Caregiver Job at Amada Senior Care NorthShore

Job Description Job Description Job Summary:We are seeking a compassionate Caregiver to join our team. The Caregiver will provide assistance to individuals in their daily living activities, ensuring their comfort and well-being. Responsibilities:- Assist clients...

HealthMarkets Insurance Agency

Licensed Insurance Agent Job at HealthMarkets Insurance Agency

 ...succeed on your terms with HealthMarkets. Disclaimer: Licensed Insurance Agents with HealthMarkets Insurance Agency are self-employed acting as independent contractors and not as employees. HealthMarkets independent licensed insurance agents shall for all... 

Powerwell

Lead Food Scientist I Job at Powerwell

 ...Powerwell is a leading recruitment firm helping our clients discover top talent. Were recruiting for a Lead Food Scientist I. About Our Client: Our clients purpose is unlocking the potential of people, nature, and science to nourish the world. We consistently...