apache spark using Pyspark ETL help
$30-50 USD
已取消
已发布大约 4 年前
$30-50 USD
货到付款
Basically I have an ETL with 2 updates and I want to write the same updates in Pyspark
table_a:
+---+-----------+-------+--------------+
|key|col_a | col_b | current_flag |
+---+-----------+-------+--------------+
|001| Value1 | T123 | Y |
|002| oth_val1 | T123 | N |
|003| oth_val2 | T123 | N |
|004| oth_val3 | T123 | N |
|005| Value2 | T123 | Y |
|006| oth_val4 | T789 | N |
|007| Value2 | T789 | Y |
|008| Value1 | T789 | N |
+---+-----------+-------+--------------+
UPDATE table_abc SET col_a = 'Value1'
WHERE col_b IN
(
SELECT col_b
FROM table_abc
WHERE col_a = 'Value1'
and current_flag = 'Y'
)
AND current_flag = 'N'
COMMIT;
+---+-----------+-------+--------------+
|key|col_a | col_b | current_flag |
+---+-----------+-------+--------------+
|001| Value1 | T123 | Y |
|002| Value1 | T123 | N | -- updated
|003| Value1 | T123 | N | -- updated
|004| Value1 | T123 | N | -- updated
|005| Value2 | T123 | Y |
|006| oth_val4 | T789 | N |
|007| Value2 | T789 | Y |
|008| Value1 | T789 | N |
+---+-----------+-------+--------------+
UPDATE table_abc SET col_a = 'Value2'
WHERE col_b IN
(
SELECT col_b
FROM table_abc
WHERE col_a = 'Value2'
and current_flag = 'Y'
)
AND current_flag = 'N'
COMMIT
+---+-----------+-------+--------------+
|key|col_a | col_b | current_flag |
+---+-----------+-------+--------------+
|001| Value1 | T123 | Y |
|002| Value1 | T123 | N |
|003| Value1 | T123 | N |
|004| Value1 | T123 | N |
|005| Value2 | T123 | Y |
|006| Value2 | T789 | N | -- updated
|007| Value2 | T789 | Y |
|008| Value2 | T789 | N | -- updated
+---+-----------+-------+--------------+
---------------------------------------------------------
#pyspark code to reproduce the updates
#initial dataframe is "table_a"
tval1 = [login to view URL](
col("col_a") == lit("Value1") & col("current_flag") == lit("Y")
)
t= [login to view URL]("t1").join(
[login to view URL]("tval1"), col("t1.col_b") == col("tval1.col_b"), "left-outer"
).select(
col("[login to view URL]"),
when( col("tval1.col_b").isNotNull(),
lit("Value1")
).otherwise(col("t1.col_a")).alias("col_a"),
col("t1.col_b"),
col("t1.current_flag")
)
#use data frame t from above
tval2 = [login to view URL](
col("col_a") == lit("Value2") & col("current_flag") == lit("Y")
)
t_new = [login to view URL]("t1").join(
[login to view URL]("tval2"), col("t1.col_b") == col("tval2.col_b"), "left-outer"
).select(
col("[login to view URL]"),
when( col("tval2.col_b").isNotNull(),
lit("Value2")
).otherwise(col("t1.col_a")).alias("col_a"),
col("t1.col_b"),
col("t1.current_flag")
)
but what really happens in Pyspark is this:
t_new:
+---+-----------+-------+--------------+
|key|col_a | col_b | current_flag |
+---+-----------+-------+--------------+
|001| Value1 | T123 | Y |
|002| Value2 | T123 | N |
|003| Value2 | T123 | N |
|004| Value2 | T123 | N |
|005| Value2 | T123 | Y |
|006| Value2 | T789 | N |
|007| Value2 | T789 | Y |
|008| Value2 | T789 | N |
+---+-----------+-------+--------------+
项目 ID: 25337503
关于此项目
23提案
远程项目
活跃4 年前
想赚点钱吗？
在Freelancer上竞价的好处
设定您的预算和时间范围
为您的工作获得报酬
简要概述您的提案
免费注册和竞标工作
23威客以平均价$82 USD来参与此工作竞价
5.1
5.1
5.0
5.0
4.2
4.2
3.4
3.4
3.2
3.2
3.0
3.0
2.8
2.8
3.0
3.0
1.8
1.8
1.2
1.2
1.1
1.1
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
关于客户
Bear, United States
28
付款方式已验证
会员自9月 15, 2005起
客户认证
这个客户的其他工作
$500-1000 USD
$95-100 USD
$30-50 USD
$-1-4 USD
$-1-4 USD
类似的工作
$25-50 AUD / hour
$30-250 USD
$30-250 USD
$30-250 USD
$10-30 USD
€250-750 EUR
₹37500-75000 INR
€8-30 EUR
$15-25 CAD / hour
$250-750 USD
$10-30 USD
$30-250 USD
$25-50 USD / hour
$250-750 USD
$30-250 USD
₹12500-37500 INR
₹750-1250 INR / hour
₹1500-12500 INR
$30-250 USD
$250-750 USD
谢谢！我们已通过电子邮件向您发送了索取免费积分的链接。
发送电子邮件时出现问题。请再试一次。
加载预览
授予地理位置权限。
您的登录会话已过期而且您已经登出，请再次登录。