Find Jobs
Hire Freelancers

Senior Data Engineer

₹75000-150000 INR

已关闭
已发布超过 2 年前

₹75000-150000 INR

货到付款
Senior Data Engineer Technical Skills Languages – Python, SQL, Java, HCL, HTML/CSS/Javascript, Bash Database Technology – Spark, SybaseIQ, DB/2, Snowflake, Redshift, Hive, Presto, Oracle PL/SQL Tools – AWS, Terraform, Kubernetes, Docker, Jupyter, Intellij, vim, Git, SVN, Apache, nginx, Splunk, SSH · Primarily should have worked on the Data Lake, a petabyte-scale Data Warehouse built for Goldman Sachs’ unique requirements. The lake is used across hundreds of teams for many time-sensitive critical applications. · Derived a variety of SLOs and health indicators for the lake. Successfully optimized the lake, bringing ingestion time down under 15 minutes for more than 90% of users. · Designed an event-driven near real-time SLO monitor for the lake that processes millions of events a minute. · Crafted terraform AWS configurations from scratch to deploy key lake components to the cloud. · Developed and maintained a Jupyter notebook ecosystem on Kubernetes to support the SRE team. · Wrote Jupyter notebooks to analyze telemetry metrics, develop insights, and establish SLOs. Notebooks typically pulled in data using SQL or Pyspark, and further processed in Pandas. Visualizations were done using matplotlib. · Designed an automation framework for Jupyter notebooks to schedule, cache, serve, and email them to clients. · Implemented and maintained Prometheus metrics for high-level monitoring of the lake. These metrics are pushed to Grafana for visualization and Pagerduty for alerting. · Developed on Facebook’s Hadoop system through Hive and Presto, using Facebook’s internal ETL framework. · Maintained solutions with third parties for ad data ingestion and delivery, including coordination of data definitions and validation checks during ETL process. · Created APIs using hack (PHP) for upload endpoints. · Developed dashboards for sales lift data normalized across third parties using Tableau and internal tools. · Maintained ETL processes to solve bugs, data quality issues, CPU and space optimization, and adding columns to tables, which were mainly core ad metrics data sets that had a wide impact across the company. · Developed Facebook status tables which was a dataset that exceeded 150TB and over 1.2 trillion rows, from Facebook’s graph structure and curated into an easily digestible hive table, used by research teams for insights, sentiment analysis, and machine learning applications.
项目 ID: 31572045

关于此项目

2提案
远程项目
活跃2 年前

想赚点钱吗?

在Freelancer上竞价的好处

设定您的预算和时间范围
为您的工作获得报酬
简要概述您的提案
免费注册和竞标工作
2威客以平均价₹112,500 INR来参与此工作竞价
用户头像
Hi, We at Tecogno Solutions are a team of Passionate Data Science and Full Stack professionals having more than five years of combined experience in multiple areas including Backend, Frontend, Machine learning (ML), Computer vision(CV) and Artificial Intelligence (AI). We have developed and delivered multiple similar Machine Learning projects. We have a strong command over Python, Flask, Django, Dialogflow, Rasa, OpenCV, Google Vision API, Tesseract, Spacy, PHP, MySQL, Software architecture, Tensorflow, Spacy, Twilio, Node.js, RESTful API, NLP, CNN, AWS, GCP, Google API's etc. Thus we are capable of fulfilling all your requirements and for further information please refer to our profile or visit our website. Looking forward to work with you. Thanks
₹150,000 INR 在31天之内
4.6 (3条评论)
5.1
5.1

关于客户

INDIA的国旗
Mysore, India
0.0
0
会员自9月 21, 2021起

客户认证

谢谢!我们已通过电子邮件向您发送了索取免费积分的链接。
发送电子邮件时出现问题。请再试一次。
已注册用户 发布工作总数
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
加载预览
授予地理位置权限。
您的登录会话已过期而且您已经登出,请再次登录。