BigData engineering pipeline
for data ingestion

Project description:

The client has on-premises storage infrastructure, where the files with the genetic analysis data are placed.

We have designed and implemented the AWS technology stack-based, BigData pipeline, which transfers raw data from on premises storage to the AWS S3 storage and later to the database deployed on AWS infrastructure. The pipeline includes CHECKSUM validation of each file and its insertion into the database through the EKS jobs. The pipeline also contains a retries mechanism and notification using the AWS SES (Simple Email Service).

Technologies:

• Python, AWS Stack: S3, Lambda, EKS, ECR, Cloudwatch, SES.