[vc_row row_type=”3″ blox_image=”20599″ align_center=”aligncenter” page_title=”page-title-x” blox_dark=”true” parallax_speed=”8″][vc_column]

---

---

[vc_column_text]

BigData engineering pipeline
for data ingestion

[/vc_column_text]

---

---

[/vc_column][/vc_row][vc_row][vc_column width=”1/4″]

---

---

[/vc_column][vc_column width=”1/2″]

---

[vc_column_text]Genetic testing and rare disease detection and analysis company based in Germany.[/vc_column_text]

---

[/vc_column][vc_column width=”1/4″][/vc_column][/vc_row][vc_row][vc_column width=”1/3″][vc_single_image image=”21717″ img_size=”550×350″][/vc_column][vc_column width=”1/3″][vc_single_image image=”21774″ img_size=”550×350″][/vc_column][vc_column width=”1/3″][vc_single_image image=”21775″ img_size=”550×350″][/vc_column][/vc_row][vc_row][vc_column]

---

[vc_column_text]

About this project

[/vc_column_text][vc_row_inner][vc_column_inner width=”1/2″][vc_column_text]

Project description:

The client has on-premises storage infrastructure, where the files with the genetic analysis data are placed.

We have designed and implemented the AWS technology stack-based, BigData pipeline, which transfers raw data from on premises storage to the AWS S3 storage and later to the database deployed on AWS infrastructure. The pipeline includes CHECKSUM validation of each file and its insertion into the database through the EKS jobs. The pipeline also contains a retries mechanism and notification using the AWS SES (Simple Email Service).

[/vc_column_text][/vc_column_inner][vc_column_inner width=”1/2″][vc_column_text]

Technologies:

• Python, AWS Stack: S3, Lambda, EKS, ECR, Cloudwatch, SES.

[/vc_column_text][/vc_column_inner][/vc_row_inner]

---

[/vc_column][/vc_row]