{"id":6122,"date":"2022-10-04T10:54:26","date_gmt":"2022-10-04T10:54:26","guid":{"rendered":"https:\/\/bentego.com\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/"},"modified":"2025-10-20T16:09:25","modified_gmt":"2025-10-20T16:09:25","slug":"yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi","status":"publish","type":"post","link":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/","title":{"rendered":"Yarn, Airflow Dag Loglar\u0131n\u0131n, Hadoop Dosya \u0130\u015flemlerinin Hive Tablosuna Yaz\u0131lmas\u0131"},"content":{"rendered":"<p class=\"pw-post-body-paragraph zk zl yb sx b zm zn vg zo zp zq vj zr zs zt zu zv zw zx zy zz aba abb abc abd abe qc by\" data-selectable-paragraph=\"\">Veri m\u00fchendisli\u011fi bir \u00e7ok farkl\u0131 domainde ki veriyi; kontrol etme, depolama, manip\u00fcle etme gibi bir \u00e7ok i\u015flevselli\u011fe sahip olmay\u0131 gerektiren bir meslek dal\u0131d\u0131r. Bu \u00e7al\u0131\u015fmada b\u00fcy\u00fck operasyonlarda g\u00f6rev alan veri m\u00fchendislerinin operasyonun devaml\u0131l\u0131\u011f\u0131 ve \u00e7\u0131kabilecek sorunlardan sadece \u00fc\u00e7\u00fcn\u00fc otomatize etmesi ad\u0131na ger\u00e7ekle\u015ftirilmi\u015f bir hands-on projedir.<\/p>\n<p id=\"9530\" class=\"pw-post-body-paragraph zk zl yb sx b zm zn vg zo zp zq vj zr zs zt zu zv zw zx zy zz aba abb abc abd abe qc by\" data-selectable-paragraph=\"\">Peki bu sorunlar nelerdir?<\/p>\n<ul class=\"\">\n<li id=\"cd30\" class=\"abf abg yb sx b zm zn zp zq zs abh zw abi aba abj abe abk abl abm ea by\" data-selectable-paragraph=\"\">Bir prod ortam\u0131nda bir \u00e7ok geli\u015ftiricinin core kullan\u0131mlar\u0131n\u0131n kontrol alt\u0131nda tutulmas\u0131 gerekmektedir. Bu olmad\u0131\u011f\u0131 takdirde bir \u00e7ok ba\u015fka core t\u00fcketmesi gereken i\u015f aksayacakt\u0131r veya core \/ container oranlar\u0131n\u0131n bire yak\u0131n olmas\u0131 ile devam eden s\u00fcre\u00e7te verimsiz kaynak kullan\u0131mlar\u0131 ger\u00e7ekle\u015febilmektedir.<\/li>\n<li id=\"8972\" class=\"abf abg yb sx b zm abn zp abo zs abp zw abq aba abr abe abk abl abm ea by\" data-selectable-paragraph=\"\">Airflow DAG\u2019leri scheduler \u00e7al\u0131\u015fan ve birbirine \u00e7e\u015fitli sens\u00f6rler arac\u0131l\u0131\u011f\u0131 ile ba\u011fl\u0131 olarak birbirini trigger edebilmektedir. Bu da herhangi bir sens\u00f6rde olu\u015fan s\u0131k\u0131nt\u0131n\u0131n t\u00fcm dagleri domino etkisi ile etkilemesi demektir.<\/li>\n<li id=\"6b41\" class=\"abf abg yb sx b zm abn zp abo zs abp zw abq aba abr abe abk abl abm ea by\" data-selectable-paragraph=\"\">Her \u015firketin belirli bir disk kapasitesi bulunmaktad\u0131r. G\u00fcn\u00fcm\u00fcz de \u015firketlerin veri ak\u0131\u015flar\u0131 \u00e7ok astronomik boyutlara do\u011fru ilerlemektedir. Bu s\u00fcre\u00e7te hangi verilerin tutulmas\u0131 gerekti\u011fi hangi verilerin silinmesi gerekti\u011fi kontrol alt\u0131nda tutulmazsa \u015firket art\u0131k yeni veri giri\u015fleri sa\u011flayamacak ve operasyonelli\u011fi sekteye u\u011frayacakt\u0131r.<\/li>\n<\/ul>\n<p id=\"6ca7\" class=\"pw-post-body-paragraph zk zl yb sx b zm zn vg zo zp zq vj zr zs zt zu zv zw zx zy zz aba abb abc abd abe qc by\" data-selectable-paragraph=\"\">Yukar\u0131da olu\u015fan g\u00fcnl\u00fck operasyonlar\u0131n kontrol alt\u0131nda tutulmas\u0131 ve \u00e7e\u015fitli tablolara yaz\u0131larak takibi sa\u011flanmas\u0131 gerekmektedir gerekirse alarm mailleri olu\u015fturularak ilgili ki\u015filere bu mailin scheduler \u015fekilde ula\u015ft\u0131r\u0131lmas\u0131 elzemdir. Bu s\u00fcre\u00e7leri de y\u00f6neten ve kontrol alt\u0131nda tutulmas\u0131n\u0131 sa\u011flayacak hands-on bir proje ger\u00e7eklenmi\u015ftir. Bu \u00e7al\u0131\u015fmalar Airflow Dagine tak\u0131larak scheduler \u00e7al\u0131\u015fmas\u0131 sa\u011flanm\u0131\u015ft\u0131r.<\/p>\n<p id=\"e0f9\" class=\"pw-post-body-paragraph zk zl yb sx b zm zn vg zo zp zq vj zr zs zt zu zv zw zx zy zz aba abb abc abd abe qc by\" data-selectable-paragraph=\"\"><strong>\u00c7al\u0131\u015fmalar B\u00fcnyesinde kullan\u0131lan teknolojiler:<\/strong><\/p>\n<p id=\"9759\" class=\"pw-post-body-paragraph zk zl yb sx b zm zn vg zo zp zq vj zr zs zt zu zv zw zx zy zz aba abb abc abd abe qc by\" data-selectable-paragraph=\"\">Apache Hadoop, Apache Hive, YARN, Apache Airlfow ,SQLite, Bash, Python, SQL<\/p>\n<p data-selectable-paragraph=\"\"><strong>\u00c7al\u0131\u015fmalar:<\/strong><\/p>\n<ul>\n<li id=\"76b3\" class=\"abs abt yb bv kx abu abv abw sc abx aby abz sh si aca sj sm sn acb so sr ss acc st sw acd by\" style=\"text-align: left\">Yarn Loglar\u0131n\u0131n Hive Tablosuna Yaz\u0131lmas\u0131<\/li>\n<li class=\"abs abt yb bv kx abu abv abw sc abx aby abz sh si aca sj sm sn acb so sr ss acc st sw acd by\" style=\"text-align: left\">Airflow DAG Loglar\u0131n\u0131n Okunmas\u0131 Son Fail Tasklar\u0131n Tespiti Ve Task S\u00fcrelerinin \u00c7\u0131kar\u0131lmas\u0131<\/li>\n<li class=\"abs abt yb bv kx abu abv abw sc abx aby abz sh si aca sj sm sn acb so sr ss acc st sw acd by\" style=\"text-align: left\">Hadoop Dosyalar\u0131n\u0131n Bulunma S\u00fcrelerinin Hive Tablosuna Yaz\u0131lmas\u0131<\/li>\n<\/ul>\n<p id=\"76b3\" class=\"abs abt yb bv kx abu abv abw sc abx aby abz sh si aca sj sm sn acb so sr ss acc st sw acd by\"><strong>Yarn Loglar\u0131n\u0131n Hive Tablosuna Yaz\u0131lmas\u0131<\/strong><\/p>\n<p id=\"593a\" class=\"pw-post-body-paragraph zk zl yb sx b zm ace vg zo zp acf vj zr zs acg zu zv zw ach zy zz aba aci abc abd abe qc by\" data-selectable-paragraph=\"\">YARN, kaynak y\u00f6netimini uygulama zamanlamadan\/izlemeden ayr\u0131\u015ft\u0131rarak birden \u00e7ok programlama modelini (Apache Hadoop MapReduce bunlardan biri) destekler. YARN genel resource manager(RM), \u00e7al\u0131\u015fan d\u00fc\u011f\u00fcm\u00fc ba\u015f\u0131na Node Manager (NM) ve uygulama ba\u015f\u0131na Application Master(AM) kullan\u0131r. K\u0131sacas\u0131 YARN tam olarak kaynak y\u00f6netimini sa\u011flar.<\/p>\n<p id=\"ca21\" class=\"pw-post-body-paragraph zk zl yb sx b zm zn vg zo zp zq vj zr zs zt zu zv zw zx zy zz aba abb abc abd abe qc by\" data-selectable-paragraph=\"\">Bu s\u00fcre\u00e7te YARN verilerini alabilimemiz i\u00e7in gerekli pip k\u00fct\u00fcphanesi airflow venv ortam\u0131na install edilir.<\/p>\n<p data-selectable-paragraph=\"\">[fusion_syntax_highlighter theme=&#8221;&#8221; language=&#8221;x-sh&#8221; line_numbers=&#8221;&#8221; line_wrapping=&#8221;&#8221; copy_to_clipboard=&#8221;&#8221; copy_to_clipboard_text=&#8221;&#8221; hide_on_mobile=&#8221;small-visibility,medium-visibility,large-visibility&#8221; class=&#8221;&#8221; id=&#8221;&#8221; font_size=&#8221;&#8221; border_size=&#8221;&#8221; border_color=&#8221;&#8221; border_style=&#8221;&#8221; background_color=&#8221;&#8221; line_number_background_color=&#8221;&#8221; line_number_text_color=&#8221;&#8221; margin_top=&#8221;&#8221; margin_right=&#8221;&#8221; margin_bottom=&#8221;&#8221; margin_left=&#8221;&#8221;]cGlwIGluc3RhbCBoYWRvb3AteWFybi1yZXN0LWFwaQ==[\/fusion_syntax_highlighter]<\/p>\n<p data-selectable-paragraph=\"\">Ard\u0131ndan ilgili Rest API\u2019 den verinin \u00e7ekilebilmesi i\u00e7in gerekli script a\u015fa\u011f\u0131da verilmi\u015ftir.<\/p>\n<p data-selectable-paragraph=\"\">[fusion_syntax_highlighter theme=&#8221;&#8221; language=&#8221;x-sh&#8221; line_numbers=&#8221;&#8221; line_wrapping=&#8221;&#8221; copy_to_clipboard=&#8221;&#8221; copy_to_clipboard_text=&#8221;&#8221; hide_on_mobile=&#8221;small-visibility,medium-visibility,large-visibility&#8221; class=&#8221;&#8221; id=&#8221;&#8221; font_size=&#8221;&#8221; border_size=&#8221;&#8221; border_color=&#8221;&#8221; border_style=&#8221;&#8221; background_color=&#8221;&#8221; line_number_background_color=&#8221;&#8221; line_number_text_color=&#8221;&#8221; margin_top=&#8221;&#8221; margin_right=&#8221;&#8221; margin_bottom=&#8221;&#8221; margin_left=&#8221;&#8221;]ZGVmIHlhcm5fY29ubmVjdGlvbl9yZXN0X2FwaSgqKmt3YXJncyk6CgogICAgeWFybiA9IFlhcm4oImh0dHA6Ly9sb2NhbGhvc3Q6ODA4OCIsICdqc29uJykKCiAgICByZXNwb25zZV9vYmogPSB5YXJuLmNsdXN0ZXJfYXBwbGljYXRpb25zKHsibGltaXQiOjEwMH0pCgogICAgaWYgeWFybi5yZXNwb25zZV90eXBlID09ICdqc29uJzoKICAgICAgICBwcmludChqc29uLmR1bXBzKHJlc3BvbnNlX29iaiwgaW5kZW50PTQsIHNvcnRfa2V5cz1UcnVlKSkKICAgIGVsaWYgeWFybi5yZXNwb25zZV90eXBlID09ICd4bWwnOgogICAgICAgIHByaW50KHhtbC5kb20ubWluaWRvbS5wYXJzZVN0cmluZyhFVC50b3N0cmluZyhyZXNwb25zZV9vYmopKS50b3ByZXR0eXhtbCgpKQoKICAgIGt3YXJnc1sndGknXS54Y29tX3B1c2goa2V5ID0gJ2Zvcm1hdHRlZF9wcm9jZXNzZWRfZGF0YScsIHZhbHVlID0gcmVzcG9uc2Vfb2JqKQ==[\/fusion_syntax_highlighter]<\/p>\n<p id=\"1f8a\" class=\"pw-post-body-paragraph yt yu xk th b yv yw un yx yy yz uq za zb zc zd ze zf zg zh zi zj zk zl zm zn ql by\" data-selectable-paragraph=\"\">Gelen veri ise bir airflow Xcom\u2019una yollanm\u0131\u015ft\u0131r bu xcom\u2019dan da json verisi json-&gt;dataframe d\u00f6n\u00fc\u015f\u00fcm\u00fc sa\u011flan\u0131cak, task fonksiyonunda tutulacakt\u0131r(pulling) edilecektir.<\/p>\n<p id=\"7f21\" class=\"pw-post-body-paragraph yt yu xk th b yv yw un yx yy yz uq za zb zc zd ze zf zg zh zi zj zk zl zm zn ql by\" data-selectable-paragraph=\"\">Json verisi ilk \u00f6nce dataframe\u2019e \u00e7evrilerek pandas ile manip\u00fcle edilebilir hale getirilmi\u015f ve ayn\u0131 zamanda bir ini\u015f dosyas\u0131na(landing file) da &#8220;.csv&#8221; format\u0131nda indirilmi\u015ftir.<\/p>\n<p data-selectable-paragraph=\"\">[fusion_syntax_highlighter theme=&#8221;&#8221; language=&#8221;x-sh&#8221; line_numbers=&#8221;&#8221; line_wrapping=&#8221;&#8221; copy_to_clipboard=&#8221;&#8221; copy_to_clipboard_text=&#8221;&#8221; hide_on_mobile=&#8221;small-visibility,medium-visibility,large-visibility&#8221; class=&#8221;&#8221; id=&#8221;&#8221; font_size=&#8221;&#8221; border_size=&#8221;&#8221; border_color=&#8221;&#8221; border_style=&#8221;&#8221; background_color=&#8221;&#8221; line_number_background_color=&#8221;&#8221; line_number_text_color=&#8221;&#8221; margin_top=&#8221;&#8221; margin_right=&#8221;&#8221; margin_bottom=&#8221;&#8221; margin_left=&#8221;&#8221;]ZGVmIHlhcm5fanNvbl9yZXNwb25zZV9wYXJzZXIoKiprd2FyZ3MpOgoKICAgIHJlc3BvbnNlX29iamVjdF9qc29uID0ganNvbi5kdW1wcyhrd2FyZ3NbJ3RpJ10ueGNvbV9wdWxsKGtleSA9ICdmb3JtYXR0ZWRfcHJvY2Vzc2VkX2RhdGEnLCB0YXNrX2lkcyA9IFsneWFybl9sb2dfeGNvbV9wdXNoJ10pKQoKICAgIHJlc3BvbnNlX29iamVjdF9qc29uID0ganNvbi5sb2FkcyhyZXNwb25zZV9vYmplY3RfanNvbikKICAgIAogICAgcmVzcG9uc2Vfb2JqZWN0X2pzb24gPSByZXNwb25zZV9vYmplY3RfanNvblswXQogCiAgICByZXNwb25zZV9vYmplY3RfanNvbiA9IHJlc3BvbnNlX29iamVjdF9qc29uWydhcHBzJ10KCiAgICByZXNwb25zZV9vYmpfbGlzdCA9IFtdCiAgICAKICAgIGZvciBpIGluIHJlc3BvbnNlX29iamVjdF9qc29uLml0ZW1zKCk6CiAgICAgICAgcmVzcG9uc2Vfb2JqX2xpc3QuYXBwZW5kKFtpWzBdLGlbMV1dKQoKICAgIHJlc3BvbnNlX29ial9kZiA9IHBkLkRhdGFGcmFtZShyZXNwb25zZV9vYmpfbGlzdCwgY29sdW1ucyA9IFsnbm9kZXMnLCAnayddKQoKICAgIHJlc3BvbnNlX29ial9kZiA9IHJlc3BvbnNlX29ial9kZi5leHBsb2RlKCdrJykKCiAgICByZXNwb25zZV9vYmpfZGYgPSBwZC5qc29uX25vcm1hbGl6ZShqc29uLmxvYWRzKHJlc3BvbnNlX29ial9kZi50b19qc29uKG9yaWVudCA9ICdyZWNvcmRzJykpKQoKICAgIGRmX2ZpbHRlcnJlZF9jb2x1bW5zID0gcmVzcG9uc2Vfb2JqX2RmW1snbm9kZXMnLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnay5pZCcsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICdrLnVzZXInLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnay5uYW1lJywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ2sucXVldWUnLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnay5zdGF0ZScsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICdrLmZpbmFsU3RhdHVzJywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ2sucHJvZ3Jlc3MnLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICdrLnRyYWNraW5nVUknLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnay50cmFja2luZ1VybCcsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICdrLmRpYWdub3N0aWNzJywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ2suY2x1c3RlcklkJywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ2sucHJpb3JpdHknLCAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ2suc3RhcnRlZFRpbWUnLCAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ2suZmluaXNoZWRUaW1lJywgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICdrLmVsYXBzZWRUaW1lJywgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICdrLmFtQ29udGFpbmVyTG9ncycsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICdrLmFtSG9zdEh0dHBBZGRyZXNzJywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ2subWFzdGVyTm9kZUlkJywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ2suYWxsb2NhdGVkTUInLCAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ2suYWxsb2NhdGVkVkNvcmVzJywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ2sucnVubmluZ0NvbnRhaW5lcnMnIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIF1dIAoKICAgIGRmX2ZpbHRlcnJlZF9jb2x1bW5zLnRvX2Nzdihsb2NhbF9maWxlX3BhdGhfeWFybikKCiAgICByZXR1cm4gcHJpbnQocmVzcG9uc2Vfb2JqX2RmLmluZm8oKSk=[\/fusion_syntax_highlighter]<\/p>\n<p data-selectable-paragraph=\"\">\u0130lgili fonksiyonlar airflow dagine tak\u0131larak task olarak eklenmi\u015ftir.<\/p>\n<p data-selectable-paragraph=\"\">[fusion_syntax_highlighter theme=&#8221;&#8221; language=&#8221;x-sh&#8221; line_numbers=&#8221;&#8221; line_wrapping=&#8221;&#8221; copy_to_clipboard=&#8221;&#8221; copy_to_clipboard_text=&#8221;&#8221; hide_on_mobile=&#8221;small-visibility,medium-visibility,large-visibility&#8221; class=&#8221;&#8221; id=&#8221;&#8221; font_size=&#8221;&#8221; border_size=&#8221;&#8221; border_color=&#8221;&#8221; border_style=&#8221;&#8221; background_color=&#8221;&#8221; line_number_background_color=&#8221;&#8221; line_number_text_color=&#8221;&#8221; margin_top=&#8221;&#8221; margin_right=&#8221;&#8221; margin_bottom=&#8221;&#8221; margin_left=&#8221;&#8221;]Cnlhcm5fbG9nX3B1c2hfeGNvbSA9IFB5dGhvbk9wZXJhdG9yKAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0YXNrX2lkID0gJ3lhcm5fbG9nX3hjb21fcHVzaCcsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHByb3ZpZGVfY29udGV4dD1UcnVlLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBweXRob25fY2FsbGFibGU9IHlhcm5fY29ubmVjdGlvbl9yZXN0X2FwaSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGFnID0gZGFnCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICkKeWFybl9sb2dzX3BhcnNlciA9IFB5dGhvbk9wZXJhdG9yKAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRhc2tfaWQgPSAneWFybl9sb2dzX3BhcnNlcicsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHJvdmlkZV9jb250ZXh0PVRydWUsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHl0aG9uX2NhbGxhYmxlPSB5YXJuX2pzb25fcmVzcG9uc2VfcGFyc2VyLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhZyA9IGRhZwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICk=[\/fusion_syntax_highlighter]<\/p>\n<div id=\"malwarebytes-root\">Bu s\u00fcre\u00e7te hedeflenen \u015fema ise ini\u015f dosyas\u0131na(landing file) indirilen &#8220;.csv&#8221; format\u0131nda ki dosyalar\u0131n hadoop ortam\u0131na u\u00e7urulmas\u0131d\u0131r. A\u015fa\u011f\u0131da bulunan scriptte formatlama i\u015flemin de atanan de\u011fi\u015fkenler ise, airflow ayn\u0131 zamanda b\u00fcnyesinde de\u011fi\u015fken saklama \u00f6zelliklerine sahiptir. Bunu json\u2019lar arac\u0131l\u0131\u011f\u0131 ile yapmak m\u00fcmk\u00fcnd\u00fcr. Bu da projenin dinamik ve bir json keyine ba\u011fl\u0131 olarak farkl\u0131 parametreler, dosya yollar\u0131 gibi bir \u00e7ok farkl\u0131 de\u011fi\u015fkene json key\u2019i arac\u0131l\u0131\u011f\u0131 ile adapte olmas\u0131 sa\u011flanm\u0131\u015ft\u0131r.<\/div>\n<div><\/div>\n<div>[fusion_syntax_highlighter theme=&#8221;&#8221; language=&#8221;x-sh&#8221; line_numbers=&#8221;&#8221; line_wrapping=&#8221;&#8221; copy_to_clipboard=&#8221;&#8221; copy_to_clipboard_text=&#8221;&#8221; hide_on_mobile=&#8221;small-visibility,medium-visibility,large-visibility&#8221; class=&#8221;&#8221; id=&#8221;&#8221; font_size=&#8221;&#8221; border_size=&#8221;&#8221; border_color=&#8221;&#8221; border_style=&#8221;&#8221; background_color=&#8221;&#8221; line_number_background_color=&#8221;&#8221; line_number_text_color=&#8221;&#8221; margin_top=&#8221;&#8221; margin_right=&#8221;&#8221; margin_bottom=&#8221;&#8221; margin_left=&#8221;&#8221;]CmZseWluZ19maWxlX21hbmFnZXJfeWFybiA9ICJoZGZzIGRmcyAtcHV0IC1mIHtsb2NhbF9maWxlX3BhdGh9L3lhcm5fbG9ncy5jc3Yge2hkZnNfZmlsZV9wYXRofSIuZm9ybWF0KGxvY2FsX2ZpbGVfcGF0aCA9IExPQ0FMX0ZJTEVfUEFUSF9ZQVJOLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBoZGZzX2ZpbGVfcGF0aCA9IEhERlNfRklMRV9QQVRIKQpsb2dzX2ZseV90b19oZGZzX3lhcm4gPSBCYXNoT3BlcmF0b3IoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGFza19pZCA9ICdsb2dzX2ZseV90b19oZGZzJywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBwcm92aWRlX2NvbnRleHQgPSBUcnVlLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGJhc2hfY29tbWFuZCA9IGZseWluZ19maWxlX21hbmFnZXJfeWFybiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBkYWcgPSBkYWcKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAp[\/fusion_syntax_highlighter]<\/div>\n<div><\/div>\n<div>Hadoop\u2019a ilgili veri lokal \u00fczerinden yolland\u0131ktan sonra hive \u00fczerinde bir ddl schema olu\u015fturulup (ilgili database i\u00e7erisinde -log_resources-) ard\u0131ndan hive ile insert ve load i\u015flemi ger\u00e7ekle\u015ftirilir. Bir internal tablo olarak olu\u015fturulmu\u015ftur.<\/div>\n<div><\/div>\n<div>[fusion_syntax_highlighter theme=&#8221;&#8221; language=&#8221;x-sh&#8221; line_numbers=&#8221;&#8221; line_wrapping=&#8221;&#8221; copy_to_clipboard=&#8221;&#8221; copy_to_clipboard_text=&#8221;&#8221; hide_on_mobile=&#8221;small-visibility,medium-visibility,large-visibility&#8221; class=&#8221;&#8221; id=&#8221;&#8221; font_size=&#8221;&#8221; border_size=&#8221;&#8221; border_color=&#8221;&#8221; border_style=&#8221;&#8221; background_color=&#8221;&#8221; line_number_background_color=&#8221;&#8221; line_number_text_color=&#8221;&#8221; margin_top=&#8221;&#8221; margin_right=&#8221;&#8221; margin_bottom=&#8221;&#8221; margin_left=&#8221;&#8221;]CmhxbF9jcmVhdGVfdGFibGUgPSAiIiIKQ1JFQVRFIFRBQkxFIElGIE5PVCBFWElTVFMgbG9nc19yZXNvdXJjZXMueWFybl9sb2dzX3RhYmxlKAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbm9kZXMgc3RyaW5nLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGlkIHN0cmluZywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBgdXNlcmAgc3RyaW5nLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG5hbWUgc3RyaW5nLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHF1ZXVlIHN0cmluZywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzdGF0ZSBzdHJpbmcsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZmluYWxTdGF0dXMgc3RyaW5nLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHByb2dyZXNzIGZsb2F0LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdHJhY2tpbmdVSSBzdHJpbmcsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdHJhY2tpbmdVcmwgc3RyaW5nLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRpYWdub3N0aWNzIHN0cmluZywgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbHVzdGVySWQgaW50LCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHByaW9yaXR5IGludCwgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc3RhcnRlZFRpbWUgaW50LCAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmaW5pc2hlZFRpbWUgaW50LCAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBlbGFwc2VkVGltZSBpbnQsICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGFtQ29udGFpbmVyTG9ncyBzdHJpbmcsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYW1Ib3N0SHR0cEFkZHJlc3Mgc3RyaW5nLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1hc3Rlck5vZGVJZCBzdHJpbmcsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYWxsb2NhdGVkTUIgaW50LCAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhbGxvY2F0ZWRWQ29yZXMgaW50LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcnVubmluZ0NvbnRhaW5lcnMgaW50CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICApIHJvdyBmb3JtYXQgZGVsaW1pdGVkIGZpZWxkcyB0ZXJtaW5hdGVkIGJ5ICcsJyBzdG9yZWQgYXMgdGV4dGZpbGUKIiIiCgpocWxfaW5zZXJ0X2VsZW1lbnRzID0gImxvYWQgZGF0YSBpbnBhdGggJ3toZGZzX2xvY2FsX3BhdGh9L3lhcm5fbG9ncy5jc3YnIGludG8gdGFibGUgbG9nc19yZXNvdXJjZXMueWFybl9sb2dzX3RhYmxlOyIuZm9ybWF0KGhkZnNfbG9jYWxfcGF0aCA9IEhERlNfRklMRV9QQVRIKQ==[\/fusion_syntax_highlighter]<\/div>\n<div><\/div>\n<div>\u0130lgili string operasyonellikleri daglere tak\u0131larak(hive ve bash operatorleri arac\u0131l\u0131\u011f\u0131 ile) task \u015femas\u0131 sonu\u00e7land\u0131r\u0131lm\u0131\u015ft\u0131r.<\/div>\n<div><\/div>\n<div><\/div>\n<div>[fusion_syntax_highlighter theme=&#8221;&#8221; language=&#8221;x-sh&#8221; line_numbers=&#8221;&#8221; line_wrapping=&#8221;&#8221; copy_to_clipboard=&#8221;&#8221; copy_to_clipboard_text=&#8221;&#8221; hide_on_mobile=&#8221;small-visibility,medium-visibility,large-visibility&#8221; class=&#8221;&#8221; id=&#8221;&#8221; font_size=&#8221;&#8221; border_size=&#8221;&#8221; border_color=&#8221;&#8221; border_style=&#8221;&#8221; background_color=&#8221;&#8221; line_number_background_color=&#8221;&#8221; line_number_text_color=&#8221;&#8221; margin_top=&#8221;&#8221; margin_right=&#8221;&#8221; margin_bottom=&#8221;&#8221; margin_left=&#8221;&#8221;]bG9nc19mbHlfdG9faGRmc195YXJuID0gQmFzaE9wZXJhdG9yKAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRhc2tfaWQgPSAnbG9nc19mbHlfdG9faGRmcycsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHJvdmlkZV9jb250ZXh0ID0gVHJ1ZSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBiYXNoX2NvbW1hbmQgPSBmbHlpbmdfZmlsZV9tYW5hZ2VyX3lhcm4sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGFnID0gZGFnCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKQpjcmVhdGVfdGFibGVfc2NoZW1hID0gSGl2ZU9wZXJhdG9yKAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRhc2tfaWQgPSAnY3JlYXRlX3RhYmxlX3NjaGVtYScsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHJvdmlkZV9jb250ZXh0ID0gVHJ1ZSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBocWwgPSBocWxfY3JlYXRlX3RhYmxlLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhZyA9IGRhZwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICkKaW5zZXJ0X3RhYmxlX3NjaGVtYSA9IEhpdmVPcGVyYXRvcigKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0YXNrX2lkID0gJ2luc2VydF90YWJsZScsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHJvdmlkZV9jb250ZXh0PSBUcnVlLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhxbCA9IGhxbF9pbnNlcnRfZWxlbWVudHMsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhZyA9IGRhZwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICk=[\/fusion_syntax_highlighter]<\/div>\n<div><\/div>\n<div>\n<p id=\"107f\" class=\"abb abc xk bv kx abd abe abf sm abg abh abi sr ss abj st sw sx abk sy tb tc abl td tg abm by\"><strong>Airflow DAG Loglar\u0131n\u0131n Okunmas\u0131 Son Fail Tasklar\u0131n Tespiti Ve Task S\u00fcrelerinin \u00c7\u0131kar\u0131lmas\u0131<\/strong><\/p>\n<p id=\"de0e\" class=\"pw-post-body-paragraph yt yu xk th b yv abn un yx yy abo uq za zb abp zd ze zf abq zh zi zj abr zl zm zn ql by\" data-selectable-paragraph=\"\">Airflow DAG\u2019lerinin status durumlar\u0131, execution zamanlar\u0131 vb metrikler airflow metadb\u2019sinde depolan\u0131r. Bu metadb sayesinde ise DAG\u2019lerin genel durumlar\u0131 hakk\u0131nda \u00e7e\u015fitli d\u00f6k\u00fcmler, veya BI dashboardlar\u0131 da \u00e7\u0131kar\u0131labilmektedir. Bu \u00e7al\u0131\u015fmada ise k\u0131stas al\u0131nan iki temel metrik bulunmaktad\u0131r. Bunlardan birincisi fail olan daglerin tespit edilmesi ve tasklerin \u00e7al\u0131\u015fma s\u00fcreleri; tasklerin \u00e7al\u0131\u015fma s\u00fcrelerinin zaman zaman uzamas\u0131 anormal durumlar\u0131n erken fark edilmesi a\u00e7\u0131s\u0131nda \u00f6nemlidir.<\/p>\n<p data-selectable-paragraph=\"\">[fusion_syntax_highlighter theme=&#8221;&#8221; language=&#8221;x-sh&#8221; line_numbers=&#8221;&#8221; line_wrapping=&#8221;&#8221; copy_to_clipboard=&#8221;&#8221; copy_to_clipboard_text=&#8221;&#8221; hide_on_mobile=&#8221;small-visibility,medium-visibility,large-visibility&#8221; class=&#8221;&#8221; id=&#8221;&#8221; font_size=&#8221;&#8221; border_size=&#8221;&#8221; border_color=&#8221;&#8221; border_style=&#8221;&#8221; background_color=&#8221;&#8221; line_number_background_color=&#8221;&#8221; line_number_text_color=&#8221;&#8221; margin_top=&#8221;&#8221; margin_right=&#8221;&#8221; margin_bottom=&#8221;&#8221; margin_left=&#8221;&#8221;]CmRlZiBhaXJmbG93X2RhZ3NfbG9ncygqYXJncyk6CiAgICAKICAgIGNvbiA9IHNxbGl0ZTMuY29ubmVjdCgiL2hvbWUvdHJhaW4vYWlyZmxvdy9haXJmbG93LmRiIikKCiAgICAKICAgIAogICAgZmFpbGVkX3Rhc2tzX2xhc3RfdGltZSA9ICBwZC5yZWFkX3NxbCgnJydTRUxFQ1QgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0YXNrX2lkLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGFnX2lkLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZXhlY3V0aW9uX2RhdGUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBGUk9NIHRhc2tfaW5zdGFuY2UKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFdIRVJFIHN0YXRlID0gJ2ZhaWxlZCcgQU5EIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZXhlY3V0aW9uX2RhdGUgQkVUV0VFTiBkYXRldGltZSgnbm93JywgJ3N0YXJ0IG9mIGRheScpIEFORCBkYXRldGltZSgnbm93JywgJ2xvY2FsdGltZScpICcnJywgY29uID0gY29uKQogICAgCiAgICAgIAogICAgcnVubmluZ190aW1lc190YXNrcyA9ICBwZC5yZWFkX3NxbCgnJycgIFNFTEVDVCAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZXhlY3V0aW9uX2RhdGUsICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBNSU4oc3RhcnRfZGF0ZSkgQVMgc3RhcnQsICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBNQVgoZW5kX2RhdGUpIEFTIGVuZCwgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIE1BWChlbmRfZGF0ZSkgLSBNSU4oc3RhcnRfZGF0ZSkgQVMgZHVyYXRpb24KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgRlJPTSAgdGFza19pbnN0YW5jZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgV0hFUkUgIGRhZ19pZCA9ICdyZXNvdXJjZV9tYW5hZ2VyX2xvZ3MnIEFORCAgc3RhdGUgPSAnc3VjY2VzcycKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBHUk9VUCBCWSAgZXhlY3V0aW9uX2RhdGUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBPUkRFUiBCWSAgZXhlY3V0aW9uX2RhdGUgREVTQycnJywgY29uID0gY29uKQogICAKICAgIHJ1bm5pbmdfdGltZXNfdGFza3MudG9fY3N2KGxvY2FsX2ZpbGVfcGF0aF9kYWdzX3J1bm5pbmcpCiAgICBmYWlsZWRfdGFza3NfbGFzdF90aW1lLnRvX2Nzdihsb2NhbF9maWxlX3BhdGhfZmFpbHVyZV90YXNrcyk=[\/fusion_syntax_highlighter]<\/p>\n<\/div>\n<p data-selectable-paragraph=\"\">\u00c7al\u0131\u015fma b\u00fcnyesinde SQLite \u00fczerinde connection olu\u015fturulmu\u015f olup ilgili veriler Hadoop b\u00fcnyesine ini\u015f dosyas\u0131(landing file) \u00fczerinden ge\u00e7irilmi\u015ftir ilgili fonksiyon dag\u2019e tak\u0131lm\u0131\u015f ve hadoop\u2019a ilgili veri u\u00e7urularak task \u015femas\u0131 tamamlanm\u0131\u015ft\u0131r.<\/p>\n<p data-selectable-paragraph=\"\">[fusion_syntax_highlighter theme=&#8221;&#8221; language=&#8221;x-sh&#8221; line_numbers=&#8221;&#8221; line_wrapping=&#8221;&#8221; copy_to_clipboard=&#8221;&#8221; copy_to_clipboard_text=&#8221;&#8221; hide_on_mobile=&#8221;small-visibility,medium-visibility,large-visibility&#8221; class=&#8221;&#8221; id=&#8221;&#8221; font_size=&#8221;&#8221; border_size=&#8221;&#8221; border_color=&#8221;&#8221; border_style=&#8221;&#8221; background_color=&#8221;&#8221; line_number_background_color=&#8221;&#8221; line_number_text_color=&#8221;&#8221; margin_top=&#8221;&#8221; margin_right=&#8221;&#8221; margin_bottom=&#8221;&#8221; margin_left=&#8221;&#8221;]CmxvZ3NfZmx5X3RvX2hkZnNfZGFncyA9IEJhc2hPcGVyYXRvcigKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0YXNrX2lkID0gJ2xvZ3NfZmx5X3RvX2hkZnNfZGFncycsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHJvdmlkZV9jb250ZXh0ID0gVHJ1ZSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBiYXNoX2NvbW1hbmQgPSBmbHlpbmdfZmlsZV9tYW5hZ2VyX2RhZ3MsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGFnID0gZGFnCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKQpkYWdfdGFza3NfcnVuaW5nX3RpbWUgPSBQeXRob25PcGVyYXRvcigKICAgIAkJCQkJCQl0YXNrX2lkID0gJ3J1bm5pbmdfdGltZXNfZGFncycsCiAgICAJCQkJCQkJZGVwZW5kc19vbl9wYXN0PUZhbHNlLAogICAgCQkJCQkJCXB5dGhvbl9jYWxsYWJsZT1haXJmbG93X2RhZ3NfbG9ncywKICAgIAkJCQkJCQlkYWcgPSBkYWcKCQkJCQkJCQkp[\/fusion_syntax_highlighter]<\/p>\n<p id=\"c06e\" class=\"abb abc xk bv kx abd abe abf sm abg abh abi sr ss abj st sw sx abk sy tb tc abl td tg abm by\"><strong>Hadoop Dosyalar\u0131n\u0131n Bulunma S\u00fcrelerinin Hive Tablosuna Yaz\u0131lmas\u0131<\/strong><\/p>\n<p id=\"afaa\" class=\"pw-post-body-paragraph yt yu xk th b yv abn un yx yy abo uq za zb abp zd ze zf abq zh zi zj abr zl zm zn ql by\" data-selectable-paragraph=\"\">Bash script kullan\u0131larak hadoop\u2019da bulunan dosyalar\u0131n bulunma s\u00fcrelerini airflow dag\u2019i \u00fczerinden scheduler bir bi\u00e7imde hive tablosuna yazarak, hadoop dosya kontrol\u00fcn\u00fcn daha d\u00fczenli ve kontrol edilebilir boyuta getirilmesi ve \u00f6l\u00e7eklenebilir bir \u00e7al\u0131\u015fma d\u00fczlemi olu\u015fturulmas\u0131 ama\u00e7lanm\u0131\u015ft\u0131r. Bu s\u00fcre\u00e7te ilgili bash script \u201c.sh\u201d uzant\u0131s\u0131 ile yaz\u0131lm\u0131\u015f olup a\u015fa\u011f\u0131da verildi\u011fi gibi hadoop i\u00e7erisinde ki dosyalar\u0131 tarayarak i\u00e7lerinden 10 g\u00fcn ve daha fazla g\u00fcn bulunan dosyalar\u0131n airflow logunda \u00e7\u0131kar\u0131lmas\u0131, ve dosyalar\u0131n ya\u015fam s\u00fcrelerinin hive tablosuna yaz\u0131lmas\u0131 ile sonu\u00e7land\u0131r\u0131lm\u0131\u015ft\u0131r.<\/p>\n<p data-selectable-paragraph=\"\">[fusion_syntax_highlighter theme=&#8221;&#8221; language=&#8221;x-sh&#8221; line_numbers=&#8221;&#8221; line_wrapping=&#8221;&#8221; copy_to_clipboard=&#8221;&#8221; copy_to_clipboard_text=&#8221;&#8221; hide_on_mobile=&#8221;small-visibility,medium-visibility,large-visibility&#8221; class=&#8221;&#8221; id=&#8221;&#8221; font_size=&#8221;&#8221; border_size=&#8221;&#8221; border_color=&#8221;&#8221; border_style=&#8221;&#8221; background_color=&#8221;&#8221; line_number_background_color=&#8221;&#8221; line_number_text_color=&#8221;&#8221; margin_top=&#8221;&#8221; margin_right=&#8221;&#8221; margin_bottom=&#8221;&#8221; margin_left=&#8221;&#8221;]IyEvYmluL2Jhc2gKCm5vdz0kKGRhdGUgKyVzKQpoZGZzIGRmcyAtbHMgLVIgfCB3aGlsZSByZWFkIGY7IGRvCiBkaXJfZGF0ZT1gZWNobyAkZiB8IGF3ayAne3ByaW50ICQ2fSdgCiBkaWZmZXJlbmNlPSQoKCAoICRub3cgLSAkKGRhdGUgLWQgIiRkaXJfZGF0ZSIgKyVzKSApIC8gKDI0ICogNjAgKiA2MCApICkpCiBoaXZlIC1lICJpbnNlcnQgaW50byB0YWJsZSBsb2dzX3Jlc291cmNlcy5oZGZzX2ZpbGVfbGlmZWN5Y2xlIHZhbHVlcygnJHtmfScsICcke2RpZmZlcmVuY2V9Jyk7IgogaWYgWyAkZGlmZmVyZW5jZSAtZ3QgMTAgXTsgdGhlbgogICBlY2hvICRmOyBlY2hvICRkaWZmZXJlbmNlOwogZmkKZG9uZQ==[\/fusion_syntax_highlighter]<\/p>\n<p data-selectable-paragraph=\"\">Burada kullan\u0131lan date alma y\u00f6ntemi awk\u2019ler ile dosya ismi \u00fczerinden default al\u0131nm\u0131\u015ft\u0131r ancak direkt olarak <em><strong>hdfs fs -stat \u201c%Y\u201d {path_dir}<\/strong><\/em> \u00fczerinden <em><strong>hdfs dfs -ls -C(sadece dosyan\u0131n ismi al\u0131n\u0131r)<\/strong><\/em> \u015feklinde de al\u0131nabilirdi ancak burada hive tablosuna yaz\u0131lan verinin \u00f6zelli\u011fi silinecek verinin yetkilerinin de kimde oldu\u011funun g\u00f6sterilmesidir. Sonu\u00e7 olarak ilgili bash file\u2019da airflow dagine tak\u0131larak sonu\u00e7land\u0131r\u0131lm\u0131\u015ft\u0131r.<\/p>\n<p data-selectable-paragraph=\"\">Burada bash ve \u201c.sh\u201d dosya uzant\u0131s\u0131 sonras\u0131 bir bo\u015fluk b\u0131rakmak gerekmektedir.<\/p>\n<p data-selectable-paragraph=\"\"><em><strong> A\u00e7\u0131klama i\u00e7in<\/strong><\/em>: <a class=\"ay qc\" href=\"https:\/\/stackoverflow.com\/questions\/66052458\/how-to-run-a-shell-script-through-apache-airflow\" target=\"_blank\" rel=\"noopener ugc nofollow\">https:\/\/stackoverflow.com\/questions\/66052458\/how-to-run-a-shell-script-through-apache-airflow<\/a><\/p>\n<p data-selectable-paragraph=\"\">[fusion_syntax_highlighter theme=&#8221;&#8221; language=&#8221;x-sh&#8221; line_numbers=&#8221;&#8221; line_wrapping=&#8221;&#8221; copy_to_clipboard=&#8221;&#8221; copy_to_clipboard_text=&#8221;&#8221; hide_on_mobile=&#8221;small-visibility,medium-visibility,large-visibility&#8221; class=&#8221;&#8221; id=&#8221;&#8221; font_size=&#8221;&#8221; border_size=&#8221;&#8221; border_color=&#8221;&#8221; border_style=&#8221;&#8221; background_color=&#8221;&#8221; line_number_background_color=&#8221;&#8221; line_number_text_color=&#8221;&#8221; margin_top=&#8221;&#8221; margin_right=&#8221;&#8221; margin_bottom=&#8221;&#8221; margin_left=&#8221;&#8221;]ZmlsZV9saWZlX2N5Y2xlID0gQmFzaE9wZXJhdG9yKAogICAgdGFza19pZD0nYmFzaF9leGFtcGxlJywKICAgIGJhc2hfY29tbWFuZCA9ICdiYXNoIC9ob21lL3RyYWluL2FpcmZsb3cvYmFzaF9jb21tYW5kcy9oZGZzX3JlbW92ZV9maWxlX2J5X2RheS5zaCAnLAogICAgZGFnPWRhZykKCmZpbGVfbGlmZV9jeWNsZQ==[\/fusion_syntax_highlighter]<\/p>\n<p data-selectable-paragraph=\"\">Sonu\u00e7 olarak bir operasyonel s\u00fcreci kontrol alt\u0131nda tutmak ve bu s\u00fcre\u00e7lerden devaml\u0131 olarak bilgi almak ad\u0131na \u00e7\u0131kabilecek \u00fc\u00e7 adet problem i\u00e7in \u00e7e\u015fitli bilgi kaynaklar\u0131 yarat\u0131ld\u0131.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Veri m\u00fchendisli\u011fi bir \u00e7ok farkl\u0131 domainde ki veriyi; kontrol etme, depolama, manip\u00fcle etme gibi bir \u00e7ok i\u015flevselli\u011fe sahip olmay\u0131 gerektiren bir meslek [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":5926,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[87],"tags":[],"class_list":["post-6122","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Yarn, Airflow Dag Loglar\u0131n\u0131n, Hadoop Dosya \u0130\u015flemlerinin Hive Tablosuna Yaz\u0131lmas\u0131 - Bentego<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/\" \/>\n<meta property=\"og:locale\" content=\"tr_TR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Yarn, Airflow Dag Loglar\u0131n\u0131n, Hadoop Dosya \u0130\u015flemlerinin Hive Tablosuna Yaz\u0131lmas\u0131 - Bentego\" \/>\n<meta property=\"og:description\" content=\"Veri m\u00fchendisli\u011fi bir \u00e7ok farkl\u0131 domainde ki veriyi; kontrol etme, depolama, manip\u00fcle etme gibi bir \u00e7ok i\u015flevselli\u011fe sahip olmay\u0131 gerektiren bir meslek [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/\" \/>\n<meta property=\"og:site_name\" content=\"Bentego\" \/>\n<meta property=\"article:published_time\" content=\"2022-10-04T10:54:26+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-10-20T16:09:25+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-86__.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2400\" \/>\n\t<meta property=\"og:image:height\" content=\"1600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Bentego\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Yazan:\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bentego\" \/>\n\t<meta name=\"twitter:label2\" content=\"Tahmini okuma s\u00fcresi\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 dakika\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/\"},\"author\":{\"name\":\"Bentego\",\"@id\":\"https:\/\/bentego.com\/tr\/#\/schema\/person\/0348418b7b0cbca83fdd7a899d54821e\"},\"headline\":\"Yarn, Airflow Dag Loglar\u0131n\u0131n, Hadoop Dosya \u0130\u015flemlerinin Hive Tablosuna Yaz\u0131lmas\u0131\",\"datePublished\":\"2022-10-04T10:54:26+00:00\",\"dateModified\":\"2025-10-20T16:09:25+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/\"},\"wordCount\":2439,\"publisher\":{\"@id\":\"https:\/\/bentego.com\/tr\/#organization\"},\"image\":{\"@id\":\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-86__.png\",\"articleSection\":[\"Blog\"],\"inLanguage\":\"tr\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/\",\"url\":\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/\",\"name\":\"Yarn, Airflow Dag Loglar\u0131n\u0131n, Hadoop Dosya \u0130\u015flemlerinin Hive Tablosuna Yaz\u0131lmas\u0131 - Bentego\",\"isPartOf\":{\"@id\":\"https:\/\/bentego.com\/tr\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-86__.png\",\"datePublished\":\"2022-10-04T10:54:26+00:00\",\"dateModified\":\"2025-10-20T16:09:25+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#breadcrumb\"},\"inLanguage\":\"tr\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"tr\",\"@id\":\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#primaryimage\",\"url\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-86__.png\",\"contentUrl\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-86__.png\",\"width\":2400,\"height\":1600},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/bentego.com\/tr\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Yarn, Airflow Dag Loglar\u0131n\u0131n, Hadoop Dosya \u0130\u015flemlerinin Hive Tablosuna Yaz\u0131lmas\u0131\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/bentego.com\/tr\/#website\",\"url\":\"https:\/\/bentego.com\/tr\/\",\"name\":\"Bentego\",\"description\":\"Turning data into enterprise value\",\"publisher\":{\"@id\":\"https:\/\/bentego.com\/tr\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/bentego.com\/tr\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"tr\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/bentego.com\/tr\/#organization\",\"name\":\"Bentego\",\"url\":\"https:\/\/bentego.com\/tr\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"tr\",\"@id\":\"https:\/\/bentego.com\/tr\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/05\/logo-bentego.svg\",\"contentUrl\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/05\/logo-bentego.svg\",\"width\":433,\"height\":109,\"caption\":\"Bentego\"},\"image\":{\"@id\":\"https:\/\/bentego.com\/tr\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/bentego.com\/tr\/#\/schema\/person\/0348418b7b0cbca83fdd7a899d54821e\",\"name\":\"Bentego\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Yarn, Airflow Dag Loglar\u0131n\u0131n, Hadoop Dosya \u0130\u015flemlerinin Hive Tablosuna Yaz\u0131lmas\u0131 - Bentego","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/","og_locale":"tr_TR","og_type":"article","og_title":"Yarn, Airflow Dag Loglar\u0131n\u0131n, Hadoop Dosya \u0130\u015flemlerinin Hive Tablosuna Yaz\u0131lmas\u0131 - Bentego","og_description":"Veri m\u00fchendisli\u011fi bir \u00e7ok farkl\u0131 domainde ki veriyi; kontrol etme, depolama, manip\u00fcle etme gibi bir \u00e7ok i\u015flevselli\u011fe sahip olmay\u0131 gerektiren bir meslek [&hellip;]","og_url":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/","og_site_name":"Bentego","article_published_time":"2022-10-04T10:54:26+00:00","article_modified_time":"2025-10-20T16:09:25+00:00","og_image":[{"width":2400,"height":1600,"url":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-86__.png","type":"image\/png"}],"author":"Bentego","twitter_card":"summary_large_image","twitter_misc":{"Yazan:":"Bentego","Tahmini okuma s\u00fcresi":"5 dakika"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#article","isPartOf":{"@id":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/"},"author":{"name":"Bentego","@id":"https:\/\/bentego.com\/tr\/#\/schema\/person\/0348418b7b0cbca83fdd7a899d54821e"},"headline":"Yarn, Airflow Dag Loglar\u0131n\u0131n, Hadoop Dosya \u0130\u015flemlerinin Hive Tablosuna Yaz\u0131lmas\u0131","datePublished":"2022-10-04T10:54:26+00:00","dateModified":"2025-10-20T16:09:25+00:00","mainEntityOfPage":{"@id":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/"},"wordCount":2439,"publisher":{"@id":"https:\/\/bentego.com\/tr\/#organization"},"image":{"@id":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#primaryimage"},"thumbnailUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-86__.png","articleSection":["Blog"],"inLanguage":"tr"},{"@type":"WebPage","@id":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/","url":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/","name":"Yarn, Airflow Dag Loglar\u0131n\u0131n, Hadoop Dosya \u0130\u015flemlerinin Hive Tablosuna Yaz\u0131lmas\u0131 - Bentego","isPartOf":{"@id":"https:\/\/bentego.com\/tr\/#website"},"primaryImageOfPage":{"@id":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#primaryimage"},"image":{"@id":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#primaryimage"},"thumbnailUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-86__.png","datePublished":"2022-10-04T10:54:26+00:00","dateModified":"2025-10-20T16:09:25+00:00","breadcrumb":{"@id":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#breadcrumb"},"inLanguage":"tr","potentialAction":[{"@type":"ReadAction","target":["https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/"]}]},{"@type":"ImageObject","inLanguage":"tr","@id":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#primaryimage","url":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-86__.png","contentUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-86__.png","width":2400,"height":1600},{"@type":"BreadcrumbList","@id":"https:\/\/bentego.com\/tr\/yarn-airflow-dag-loglarinin-hadoop-dosya-islemlerinin-hive-tablosuna-yazilmasi\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/bentego.com\/tr\/"},{"@type":"ListItem","position":2,"name":"Yarn, Airflow Dag Loglar\u0131n\u0131n, Hadoop Dosya \u0130\u015flemlerinin Hive Tablosuna Yaz\u0131lmas\u0131"}]},{"@type":"WebSite","@id":"https:\/\/bentego.com\/tr\/#website","url":"https:\/\/bentego.com\/tr\/","name":"Bentego","description":"Turning data into enterprise value","publisher":{"@id":"https:\/\/bentego.com\/tr\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/bentego.com\/tr\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"tr"},{"@type":"Organization","@id":"https:\/\/bentego.com\/tr\/#organization","name":"Bentego","url":"https:\/\/bentego.com\/tr\/","logo":{"@type":"ImageObject","inLanguage":"tr","@id":"https:\/\/bentego.com\/tr\/#\/schema\/logo\/image\/","url":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/05\/logo-bentego.svg","contentUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/05\/logo-bentego.svg","width":433,"height":109,"caption":"Bentego"},"image":{"@id":"https:\/\/bentego.com\/tr\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/bentego.com\/tr\/#\/schema\/person\/0348418b7b0cbca83fdd7a899d54821e","name":"Bentego"}]}},"_links":{"self":[{"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/posts\/6122","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/comments?post=6122"}],"version-history":[{"count":1,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/posts\/6122\/revisions"}],"predecessor-version":[{"id":6246,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/posts\/6122\/revisions\/6246"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/media\/5926"}],"wp:attachment":[{"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/media?parent=6122"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/categories?post=6122"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/tags?post=6122"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}