Senior Data Engineer
Job Summary
We are looking for a Senior Data Engineer to design and optimize data solutions in the cloud. In this role, you will work independently to understand business needs, build data pipelines, and integrate AI models (including OpenAI custom agents) to enhance workflows. You’ll be responsible for developing ETL processes, connecting various data sources, and managing AWS cloud infrastructure. Strong experience in data engineering, prompt engineering, and cloud architecture is essential.
Key Responsibilities
- – Gather and analyze business requirements to define data solutions.
- – Design, develop, and maintain ETL pipelines, pulling data from diverse sources and transforming it for business use.
- – Work extensively with OpenAI custom agents and prompt engineering to automate data retrieval, analysis, and processing.
- – Implement AI-driven workflows to enhance data processing, analytics, and decision-making.
- – Architect and manage AWS cloud infrastructure, choosing the right services (e.g., AWS Lambda, S3, Redshift, Glue, etc.).
- – Define best practices for data storage, processing, and deployment in a cloud environment.
- – Ensure scalability, security, and performance optimization in AWS.
- – Collaborate with stakeholders to refine business logic and optimize data operations.
- – Troubleshoot and optimize data pipelines, AI models, and cloud deployments.
Requirements
- – 5+ years of experience in data engineering and working with large datasets.
- – Strong expertise in ETL processes, data transformations, and working with multiple data sources.
- – Familiarity with OpenAI API, custom agents, and prompt engineering.
- – Advanced knowledge of AWS cloud architecture, including experience with Lambdas, S3, Redshift, Glue, Step Functions, and other AWS services.
- – Strong programming skills in Python, SQL, and relevant data engineering frameworks.
- – Experience designing scalable, cloud-native data pipelines and defining infrastructure strategies.
- – Ability to work independently, gather business requirements, and deliver solutions with minimal supervision.
- – Good verbal and conversational English communication skills.
- – Strong problem-solving skills and ability to optimize data workflows efficiently.
Please apply for the position by sending your CV to hr@aobyte.com, mentioning the position you are applying for in the title.