AI Summary
Seeking a Senior Data Engineer to architect and build data pipelines on AWS, focusing on ingestion, orchestration, and a Single Source of Truth. Responsibilities include containerized Python development, AI API integration, and owning the data lifecycle from extraction to UAT. Requires 5+ years of data engineering experience, expert Python skills, and deep AWS data infrastructure knowledge. This autonomous role demands strong communication, problem-solving, and pragmatism in managing cross-team dependencies.
Key Highlights
Owns ingestion, pipeline orchestration, and Single Source of Truth architecture for new product modules.
Builds containerized data pipelines on AWS, integrates with AI APIs, and manages the data lifecycle.
Requires strong autonomy, expert Python, deep AWS data infrastructure experience, and AI API integration.
Key Responsibilities
Construct core ingestion pipelines to pull high-volume metrics from various APIs into the AWS S3/Glue/Athena SSoT.
Write and maintain containerized Python workers (RQ/Celery with Redis) on EKS for data processing.
Work directly with the Bedrock LLM layer and internal SDKs to ensure seamless data flow into the AI orchestration/scoring layer.
Extend existing Python SDKs to cover new services.
Own User Acceptance Testing (UAT) for the data layer, ensuring high quality without dedicated QA support.
Work alongside internal DevOps/platform teams for CI/CD and IaC, adapting to internal standards.
Technical Skills Required
Benefits & Perks
100% remotely with flexible hours
Dedicated annual learning budget
Competitive compensation
Nice to Have
Experience with Auth0 and openfga for ReBAC/RBAC implementation.
Worked on marketing specific pipelines, building audiences, providing metrics and sourcing data for Analytics/BI.
Proficiency with AI-assisted development tools and agentic coding workflows.
Job Description
AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
WHY JOIN US
If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!
About The Role
We are looking for a Senior Data Engineer to own ingestion, pipeline orchestration, and the Single Source of Truth architecture for new product modules. You will build containerized data pipelines on AWS, integrate directly with AI APIs, and manage the data lifecycle from extraction through user acceptance testing. This role requires strong autonomy, since it operates without dedicated QA support during the initial build phase.
What You Will Do
- Construct the core ingestion pipelines to pull high-volume metrics from the YouTube Data API, YouTube Creator API, Google Trends, and Later Social into the AWS S3/Glue/Athena SSoT.
- Write and maintain containerized Python workers (RQ/Celery with Redis) on EKS to handle data processing.
- Work directly with the Bedrock LLM layer and internal SDKs to ensure data flows seamlessly into the AI orchestration/scoring layer.
- Extend existing Python SDKs (e.g., gads-audience-insights) to cover new services like the ContentCreatorInsightsService.
- Own the User Acceptance Testing (UAT) for the data layer, ensuring high quality and catching edge cases without relying on a dedicated QA team.
- Work alongside internal DevOps/platform teams for CI/CD (GitHub Actions, ArgoCD) and IaC (Terraform), adapting quickly to internal standards.
Interested in remote work opportunities in Data Science? Discover Data Science Remote Jobs featuring exclusive positions from top companies that offer flexible work arrangements.
- 5+ years of heavy Data Engineering or Backend experience focused heavily on data pipelines and infrastructure.
- Expert-level Python skills, specifically in building containerized microservices and workers (ECS/EKS).
- Deep experience with AWS data infrastructure, specifically S3, AWS Glue, and Amazon Athena.
- Proven, hands-on experience integrating AI APIs, specifically AWS Bedrock, Claude, or Gemini, and handling external data firehoses.
- High comfort level working in a Mac/Linux terminal environment (Bash, Makefiles).
- Strong communication and interview skills; you must be able to confidently defend your technical choices and pass a rigorous, high-bar technical interview.
- Extreme pragmatism and patience when dealing with cross-team dependencies (like waiting on DevOps for access/provisioning).
- Proficiency with AI-assisted development tools and agentic coding workflows to accelerate engineering productivity.
- Upper-intermediate English level.
Browse our curated collection of remote jobs across all categories and industries, featuring positions from top companies worldwide.
- Experience with Auth0 and openfga for ReBAC/RBAC implementation.
- Worked on marketing specific pipelines, building audiences, providing metrics and sourcing data for Analytics/BI.
- Growth without limits: build your skills through mentorship, internal TechTalks, challenging projects, and a dedicated annual learning budget
- Competitive compensation: get recognition that reflects your skills and impact, with regular performance and compensation reviews
- Flexibility: work 100% remotely with flexible hours that support focus, autonomy, and a healthy work rhythm
- Meaningful, modern projects: build impactful products using modern technologies alongside global teams and leading brands
- Collaborative culture: join a supportive environment with zero micromanagement where ideas are welcomed and contributions are recognized
- Well-being & support: access local well-being programs and people-focused support tailored to your location
Similar Jobs
Explore other opportunities that match your interests
Visa Sponsorship
Relocation
Remote
Job Type
Full-time
Experience Level
Not Applicable
BairesDev
Brazil
Visa Sponsorship
Relocation
Remote
Job Type
Full-time
Experience Level
Not Applicable
BairesDev
Brazil
Visa Sponsorship
Relocation
Remote
Job Type
Full-time
Experience Level
Not Applicable
AgileEngine
Brazil