{"id":6110,"date":"2023-05-30T11:04:47","date_gmt":"2023-05-30T11:04:47","guid":{"rendered":"https:\/\/bentego.com\/azure-uzerinde-data-pipeline-olusturma\/"},"modified":"2023-05-30T11:04:47","modified_gmt":"2023-05-30T11:04:47","slug":"azure-uzerinde-data-pipeline-olusturma","status":"publish","type":"post","link":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/","title":{"rendered":"Azure \u00fczerinde Data Pipeline Olu\u015fturma"},"content":{"rendered":"

Merhabalar bu yaz\u0131mda sizlere Bulut Servis Sa\u011flay\u0131c\u0131lar\u0131ndan olan Azure<\/strong> \u00fczerinde bir Data Pipeline<\/strong> olu\u015fturma s\u00fcrecini anlataca\u011f\u0131m. Gelin isterseniz ilk \u00f6nce projemizin mimarisine bakal\u0131m. <\/p>\n

<\/p>\n

Mimarimiz yukar\u0131da g\u00f6rm\u00fc\u015f oldu\u011funuz gibidir. Veri ak\u0131\u015f\u0131 anlat\u0131m\u0131na ge\u00e7meden \u00f6nce Azure<\/strong> \u00fczerinde kulland\u0131\u011f\u0131m\u0131z servisleri k\u0131saca bir tan\u0131yal\u0131m. <\/p>\n

Azure Resource Group \u2192 Microsoft Azure<\/strong> bulut platformunda kaynaklar\u0131n\u0131z\u0131 y\u00f6netmek i\u00e7in kullan\u0131lan bir yap\u0131d\u0131r. Azure<\/strong>‘da olu\u015fturdu\u011funuz kaynaklar, sanal makineler, depolama hesaplar\u0131n\u0131, a\u011f yap\u0131land\u0131rmalar\u0131n\u0131, uygulamalar\u0131 ve di\u011fer hizmetleri mant\u0131ksal bir grup i\u00e7inde toplaman\u0131z\u0131 sa\u011flar.<\/p>\n

Azure Data Factory \u2192 Microsoft Azure<\/strong> bulut platformunda veri entegrasyonu ve veri i\u015fleme i\u015flerini otomatikle\u015ftirmek i\u00e7in kullan\u0131lan bir hizmettir. Azure Data Factory<\/strong>, farkl\u0131 veri kaynaklar\u0131 aras\u0131nda veri hareketini<\/strong>, d\u00f6n\u00fc\u015f\u00fcm\u00fcn\u00fc<\/strong> ve i\u015flenmesini<\/strong> sa\u011flayarak veri entegrasyonunu<\/strong> kolayla\u015ft\u0131r\u0131r.<\/p>\n

Azure Synapse Analytics \u2192 Microsoft Azure<\/strong> bulut platformunda, verilerinizi analiz etmek, i\u015flemek ve raporlamak<\/strong> i\u00e7in kullanabilece\u011finiz g\u00fc\u00e7l\u00fc bir veri analiti\u011fi \u00e7\u00f6z\u00fcm\u00fcd\u00fcr.Azure Synapse Analytics<\/strong>, veri ambar\u0131, veri entegrasyonu, veri sorgulama ve b\u00fcy\u00fck veri analiti\u011fi<\/strong> gibi i\u015flemleri tek bir platformda<\/strong> birle\u015ftirir.<\/p>\n

Azure Data Lake \u2192 <\/strong>B\u00fcy\u00fck veri depolama ve analizi i\u00e7in kullan\u0131lan bir hizmettir. Azure<\/strong> bulut platformunda, yap\u0131land\u0131r\u0131lm\u0131\u015f<\/strong> ve yap\u0131land\u0131r\u0131lmam\u0131\u015f<\/strong> b\u00fcy\u00fck veri k\u00fctlelerini g\u00fcvenli ve \u00f6l\u00e7eklenebilir<\/strong> bir \u015fekilde depolamak<\/strong> i\u00e7in kullan\u0131l\u0131r.<\/p>\n

Azure SQL Database \u2192 Microsoft Azure<\/strong> bulut platformunda bar\u0131nd\u0131r\u0131lan ili\u015fkisel veritaban\u0131 hizmetidir. Geleneksel SQL Server<\/strong> veritaban\u0131n\u0131n bulut tabanl\u0131 bir s\u00fcr\u00fcm\u00fc<\/strong> olarak d\u00fc\u015f\u00fcn\u00fclebilir. <\/p>\n

Mimariye bakt\u0131\u011f\u0131m\u0131zda ilk k\u0131s\u0131mda On-Premise<\/strong> tarafta ger\u00e7ekle\u015fen i\u015flemlerimiz mevcut. Gelin isterseniz ilk olarak lokalimizde \u00e7al\u0131\u015fan python koduna ve \u00e7\u0131kt\u0131s\u0131na bir bakal\u0131m. <\/p>\n

\n

import<\/span> spotipy
\nfrom<\/span> spotipy.oauth2 import SpotifyClientCredentials
\n import<\/span> pandas as<\/span> pd
\nfrom<\/span> datetime import<\/span> date
\nimport<\/span> configparser<\/p>\n

parser = configparser.ConfigParser()
\nparser.read( “conf.ini”<\/span>)<\/p>\n

client_credentials_manager = SpotifyClientCredentials(
\nclient_id = parser.get(“<\/span>credentials“, “CLIENT_ID”<\/span>),
\n client_secret<\/span>= parser.get(“credentials “, “CLIENT_SECRET”<\/span>)
\n)<\/p>\n

sp = spotipy.Spotify(client_credentials_manager = client_credentials_manager)<\/p>\n

playlist_link = ‘https:\/\/open.spotify.com\/playlist\/37i9dQZF1EIWxkuBxGv3mp’<\/span><\/p>\n

playlist_URI = playlist_link.split(“\/”<\/span>)[-1<\/span>]
\ndata = sp.playlist_tracks(playlist_URI)
\ndate_str = str<\/span>(date.today())<\/p>\n

album_list = []<\/p>\n

for<\/span> row in<\/span> data[ “items”<\/span>]:
\nalbum_id = row[ “track”<\/span>][ “album”<\/span>][ “id”<\/span>]
\nalbum_name = row[ “track”<\/span>][ “album”<\/span>][ “name”<\/span>]
\nalbum_release_date = row[ “track”<\/span>][ “<\/span>album “<\/span>][ “release_date”]
\nalbum_total_tracks = row[ “track”<\/span>][ “album<\/span> “][ “total_tracks “<\/span>]
\nalbum_url = row[ “track”<\/span>][ “<\/span> album<\/span>“][ “external_urls”][ “spotify<\/span> “]
\nalbum_element = {
\n “album_id”<\/span>: album_id,
\n “isim”<\/span>: alb\u00fcm_ad\u0131,
\n “album_release_date”<\/span>: alb\u00fcm_release_date,
\n “album_total_tracks”<\/span>: album_total_tracks,
\n “album_url”<\/span>: album_url
\n}
\nalbum_list.append(album_element)<\/p>\n

artist_list = []<\/p>\n

for<\/span> row in<\/span> data[“items”<\/span>]:
\n for<\/span> key, value in<\/span> row.items():
\n if<\/span> key == “track”<\/span>:
\n for<\/span> sanat\u00e7\u0131 in<\/span> value[“sanat\u00e7\u0131lar”<\/span>]:
\nsanat\u00e7\u0131_dict = {
\n “artist_id”<\/span>: artist[“id”<\/span>],
\n “artist_name”<\/span>: artist[“name”<\/span>],
\n “external_url”<\/span>: artist[“href”<\/span>]
\n}
\nartist_list.append(artist_dict)<\/p>\n

song_list = []<\/p>\n

for<\/span> row in<\/span> data[ “items”<\/span>]:
\nsong_id = row[ “track”<\/span>][ “id”<\/span>]
\nsong_name = row[ “track”<\/span>][ “name”<\/span>]
\nsong_duration = row[ “track”<\/span>][ “duration_ms”<\/span>]
\nsong_url = row[ “track”<\/span>][ “external_urls”<\/span>][ “spotify<\/span> “]
\nsong_popularity = row[ “track”<\/span>][ “popularity”<\/span>]
\nsong_added = row[ “added_at”<\/span>]
\nalbum_id = row[ “track”<\/span>][ “album”<\/span>][ “id”<\/span>]
\nartist_id = row[ “track”<\/span>][ “album”<\/span>][ “artists”<\/span>][ 0<\/span>][ “id”<\/span>]
\nsong_element = {
\n “song_id”<\/span>: song_id,
\n “\u015fark\u0131_ad\u0131”<\/span>: \u015fark\u0131_ad\u0131,
\n “duration_ms”<\/span>: song_duration,
\n “url”<\/span>: song_url,
\n “pop\u00fclerlik”<\/span>: song_popularity,
\n “song_added”<\/span>: song_added,
\n “album_id”<\/span>: album_id,
\n “artist_id”<\/span>: artist_id
\n}
\nsong_list.append(song_element)<\/p>\n

album_df = pd.DataFrame.from_dict(album_list)
\nalbum_df = album_df.drop_duplicates(subset = [“album_id”<\/span>])<\/p>\n

artist_df = pd.DataFrame.from_dict(artist_list)
\nartist_df = artist_df.drop_duplicates(subset = [“artist_id”<\/span>])<\/p>\n

song_df = pd.DataFrame.from_dict(song_list)
\nalbum_df[“release_date<\/span>“] = pd.to_datetime(album_df[“album_release_date”<\/span>])
\nsong_df[“song_added<\/span>“] = pd.to_datetime(song_df[“song_added”<\/span>])<\/p>\n

song_df.to_parquet(f “data\\\\{date_str}<\/span><\/span>_song_ds.parquet”,engine=”fastparquet<\/span>“<\/span>)
\nalbum_df.to_parquet(f “data\\\\{date_str}<\/span><\/span>_album_ds.parquet”,engine=”fastparquet”)
\nartist_df.to_parquet(f “data\\\\{date_str}<\/span><\/span>_artist_ds.parquet“,engine=”fastparquet”<\/span>)<\/p><\/blockquote>\n

Conf.ini<\/strong><\/h3>\n
\n

[credentials]
\nCLIENT_ID = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
\nCLIENT_SECRET = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX<\/p>\n<\/blockquote>\n

<\/p>\n

G\u00f6rm\u00fc\u015f oldu\u011funuz Python<\/strong> kodu Spotify API<\/strong>‘na istekte bulunarak d\u00f6nen sonu\u00e7tan 3 farkl\u0131 veri seti elde ederek bunlar\u0131 parke <\/strong>format\u0131nda belirtilen dizine kaydediyor.<\/p>\n

Evet lokalimizdeki i\u015flemleri b\u00f6ylelikle bitiyoruz. \u015eimdi Azure <\/strong>\u00fczerindeki servislerimizi olu\u015fturarak i\u015flemlerimize devam edelim. <\/p>\n

Kaynak Grubu<\/strong> Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

Azure Veri G\u00f6l\u00fc<\/strong> Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

<\/p>\n

Azure Data Factory <\/strong>Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

Azure Synapse Analytics <\/strong>Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

<\/p>\n

Azure SQL Veritaban\u0131 <\/strong>Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

<\/p>\n

Gerekli servisleri olu\u015fturdu\u011fumuza g\u00f6re Azure Data Factory Studio <\/strong>ekran\u0131na giderek Manage<\/strong> sekmesinden Integration Runtimes(IR)<\/strong> k\u0131sm\u0131na gelerek lokalimizle ba\u011flant\u0131 sa\u011flayacak yeni bir tanesini olu\u015fturaca\u011f\u0131z.<\/p>\n

Self-Hosted IR<\/strong> Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

Se\u00e7enek 2<\/strong>‘deki link’e t\u0131klayarak indirme i\u015flemini ger\u00e7ekle\u015ftiriyoruz. \u0130\u015flem bittikten sonra \u00e7\u0131kacak ekrana Key1<\/strong>‘i kopyalayarak ilgili alana yap\u0131\u015ft\u0131r\u0131yoruz. <\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

Ba\u015far\u0131l\u0131 bir \u015fekilde lokalimiz ile ba\u011flant\u0131 sa\u011flayacak ve ilgili i\u015flemleri orada ger\u00e7ekle\u015ftirecek olan IR<\/strong>‘\u0131 olu\u015fturmu\u015f olduk.<\/p>\n

Data Factory<\/strong> \u00fczerinde olu\u015fturaca\u011f\u0131m\u0131z Pipeline da lokalimizde ki parke dosyalar\u0131n\u0131 kullanabilmek i\u00e7in Linked Service<\/strong> olu\u015fturaca\u011f\u0131z onu da a\u015fa\u011f\u0131daki ad\u0131mlar\u0131 takip ederek olu\u015fturabilirsiniz.<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

Olu\u015fturaca\u011f\u0131m\u0131z Data Pipeline i\u00e7erisinde bu LS(Linked Service)’i <\/strong>kullanarak lokal’deki her bir parke format\u0131 e\u011fer bug\u00fcn\u00fcn tarihi ile ba\u015fl\u0131yorsa onu alarak Azure Data Lake<\/strong>‘deki belirtti\u011fimiz dizine kopyalayacak.<\/p>\n

DP_Extract_OnpremToADL2:<\/strong><\/h3>\n

<\/p>\n

Yukar\u0131da g\u00f6rm\u00fc\u015f oldu\u011funuz Data Pipeline<\/strong>‘\u0131 a\u00e7\u0131klamak gerekirse yapt\u0131\u011f\u0131 i\u015f Get Metadata<\/strong>‘y\u0131 kullanarak lokal’deki her parke <\/strong>dosyas\u0131n\u0131n dosyas\u0131n\u0131 ismini ForEach Loop<\/strong>‘unda d\u00f6nd\u00fcrek e\u011fer bug\u00fcn\u00fcn tarihiyle ba\u015fl\u0131yorsa kopyalama i\u015flemini ger\u00e7ekle\u015ftirecek e\u011fer de\u011filse hi\u00e7bir i\u015flem yapmayacak.Bu kontrol\u00fc yapan Activity ‘If Condition’ <\/strong>ve ko\u015fulu yapan kod ise a\u015fa\u011f\u0131daki gibidir.<\/p>\n

<\/p>\n

<\/p>\n

Data Pipeline’\u0131 \u00e7al\u0131\u015ft\u0131rd\u0131\u011f\u0131m\u0131zda ba\u015far\u0131l\u0131 bir \u015fekilde \u00e7al\u0131\u015ft\u0131\u011f\u0131n\u0131 g\u00f6rmekteyiz. Azure Data Lake<\/strong> \u00fczerindeki belirtti\u011fimiz dizini kontrol ederek veriler ba\u015far\u0131l\u0131 bir \u015fekilde gelmi\u015f mi \u015fimdi ona bakal\u0131m.<\/p>\n

<\/p>\n

Lokalimizdeki veri setlerinin ba\u015far\u0131l\u0131 bir \u015fekilde kopyaland\u0131\u011f\u0131n\u0131 g\u00f6rmekteyiz.<\/p>\n

Bir sonraki a\u015famam\u0131z Azure Synapse Analytics<\/strong> \u00fczerinde Spark Pool<\/strong> olu\u015fturarak veri temizleme ve d\u00f6n\u00fc\u015ft\u00fcrme i\u015flemi uygulayaca\u011f\u0131z.<\/p>\n

Spark Pool<\/strong> Olu\u015fturma \u0130\u015flemi:<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

\n

from<\/span> notebookutils import<\/span> mssparkutils
\nfrom<\/span> pyspark.sql.functions import *
\nfrom<\/span> pyspark.sql.types import<\/span> *<\/p>\n

accountname= “bentegostg<\/span> ”
\nsourcecontainer= “raw”<\/span>
\nLinkedService= “bentego-synapse-ws-WorkspaceDefaultStorage”<\/span>
\nsourceFile_location= “ingest”<\/span>
\ndestinationcontainer=“rafine”<\/span><\/p>\n

spark.conf.set<\/span>(“spark.storage.synapse.linkedServiceName”<\/span>,LinkedService<\/span>)
\nspark.conf.set<\/span>(“fs.azure.account.oauth.provider.typ”<\/span>,”com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider”)
\n spark.<\/span>conf.set<\/span>(“spark.sql.parquet.writeLegacyFormat“, “true”<\/span>)<\/p>\n

SongCustomSchema = StructType([
\nStructField( “song_id”<\/span>, StringType(), True<\/span>),
\nStructField( “song_name”<\/span>, StringType(), True<\/span>),
\nStructField( “duration_ms”<\/span>, LongType(), True<\/span>),
\nStructField( “url”<\/span>, StringType(), True<\/span>),
\nStructField( “pop\u00fclerlik”<\/span>, LongType(), True<\/span>),
\nStructField( “song_added”<\/span>, StringType(), True<\/span>),
\nStructField( “album_id”<\/span>, StringType(), True<\/span>),
\nStructField( “artist_id”<\/span>, StringType(), True<\/span>)
\n]
\n)<\/p>\n

song_df = spark \\
\n.read \\
\n.schema(SongCustomSchema) \\
\n.parquet(f “abfss:\/\/{sourcecontainer}<\/span><\/span>@{accountname}.dfs.core.windows.net\/{sourceFile_location}<\/span><\/span>\/2023-05-29_song_ds.parquet”)<\/p>\n

song_df = song_df \\
\n.withColumn(“duration_ms<\/span>“,col(“duration_ms”<\/span>).cast(IntegerType())) \\
\n.withColumn(“pop\u00fclerlik<\/span>“,col(“pop\u00fclerlik”<\/span>).cast(IntegerType())) \\
\n.drop(col(“song_added”<\/span>))<\/p>\n

AlbumCustomSchema = StructType([
\nStructField( “album_id”<\/span>, StringType(), True<\/span>),
\nStructField( “name”<\/span>, StringType(), True<\/span>),
\nStructField( “album_release_date”<\/span>, StringType(), True<\/span>),
\nStructField( “album_total_tracks”<\/span>, LongType(), True<\/span>),
\nStructField( “album_url”<\/span>, StringType(), True<\/span>),
\nStructField( “release_date”<\/span>, StringType(), True<\/span>)
\n]
\n)<\/p>\n

album_df = spark \\
\n.read \\
\n.schema(AlbumCustomSchema) \\
\n.parquet(f “abfss:\/\/{sourcecontainer}<\/span><\/span>@{accountname}.dfs.core.windows.net\/{sourceFile_location}<\/span><\/span>\/2023-05-29_album_ds.parquet”)<\/p>\n

album_df = album_df \\
\n.withColumn(“album_total_tracks”<\/span>,col(“album_total_tracks”<\/span>).cast(IntegerType())) \\
\n.withColumn(“album_release_date”<\/span>,to_date(col(“album<\/span>_release_date“), “yyyy-MM-dd”<\/span>)) \\
\n.drop(“release_date”<\/span>)<\/p>\n

ArtistCustomSchema = StructType([
\nStructField(“artist_id”<\/span>, StringType(), True<\/span>),
\nStructField(“artist_name<\/span>“<\/span>, StringType(), True<\/span>),
\nStructField(“external_url”, StringType(), True<\/span>),
\n]
\n)<\/p>\n

artist_df = spark \\
\n.read \\
\n.schema(ArtistCustomSchema) \\
\n.parquet(f “abfss:\/\/{sourcecontainer}<\/span><\/span>@{accountname}.dfs.core.windows.net\/{sourceFile_location}<\/span><\/span>\/2023-05-29_artist_ds.parquet”)<\/p>\n

album_df.write \\
\n.mode(“overwrite”<\/span>) \\
\n.format<\/span>(“parquet<\/span>“<\/span>) \\
\n.option(“header<\/span>“<\/span>,”true”) \\
\n.option(“inferSchema<\/span>“, “true”) \\
\n.save(f “abfss:\/\/{destinationcontainer}<\/span><\/span>@{accountname}.dfs.core.windows.net\/album\/”<\/span>)<\/p>\n

artist_df.write \\
\n.mode(“overwrite”<\/span>) \\
\n.format<\/span>(“parquet<\/span>“<\/span>) \\
\n.option(“header<\/span>“<\/span>,”true”) \\
\n.option(“inferSchema<\/span>“, “true”) \\
\n.save(f “abfss:\/\/{destinationcontainer}<\/span><\/span>@{accountname}.dfs.core.windows.net\/artist\/”<\/span>)<\/p><\/blockquote>\n

Spark Pool<\/strong>‘u olu\u015fturduktan sonra Developer<\/strong> sekmesinden Notebook<\/strong> olu\u015fturarak yukar\u0131daki g\u00f6rm\u00fc\u015f oldu\u011funuz kodlar\u0131 yaz\u0131yoruz. Normal PySpark <\/strong>kodlar\u0131m\u0131z\u0131 yazarak verinin d\u00fczg\u00fcn ve temizlenmi\u015f halini yine Azure Data Lake \u00fczerindeki belirtti\u011fimiz dizinlere yolluyoruz. <\/p>\n

<\/p>\n

<\/p>\n

Kontrol etti\u011fimizde verilerin ba\u015far\u0131l\u0131 \u015fekilde dizinlere yaz\u0131ld\u0131\u011f\u0131n\u0131 g\u00f6rmekteyiz.<\/p>\n

Yukar\u0131da olu\u015fturdu\u011fumuz Notebook’u Data Factory <\/strong>\u00fczerinde olu\u015fturdu\u011fumuz Data Pipeline i\u00e7erisinden \u00e7a\u011f\u0131rmak i\u00e7in Azure<\/strong>Data Factory<\/strong>‘i servisine Synapse Administrator<\/strong> yetkisi vermemiz gerekli.<\/p>\n

<\/p>\n

\u015eimdi Azure Data Factory<\/strong> k\u0131sm\u0131na gidirek Synapse<\/strong> k\u0131sm\u0131nda olu\u015fturdu\u011fumuz Notebook’u ba\u011flanabilmesi i\u00e7in yeni bir Linked Service<\/strong> olu\u015ftural\u0131m.<\/p>\n

<\/p>\n

<\/p>\n

Ba\u015far\u0131l\u0131 \u015fekilde ba\u011flant\u0131m\u0131z\u0131 olu\u015fturduk. Data Pipeline olu\u015ftururak Notebook’u birde oradan \u00e7a\u011f\u0131ral\u0131m. <\/p>\n

<\/p>\n

<\/p>\n

Data Pipeline’\u0131 \u00e7al\u0131\u015ft\u0131rd\u0131\u011f\u0131m\u0131zda ba\u015far\u0131l\u0131 bir \u015fekilde \u00e7al\u0131\u015ft\u0131\u011f\u0131n\u0131 g\u00f6rmekteyiz.<\/p>\n

\u00c7al\u0131\u015fan Pyspark Job<\/strong>‘\u0131m\u0131z temizlenmi\u015f ve d\u00f6n\u00fc\u015ft\u00fcr\u00fclm\u00fc\u015f veriyi ba\u015fka bir dizine kaydetmi\u015f oldu. Bu veri setlerini \u015fimdi de Azure SQL Database<\/strong> k\u0131sm\u0131nda olu\u015fturdu\u011fumuz SpotifyDB’sinin alt\u0131nda tablolar olu\u015fturarak oraya aktaraca\u011f\u0131z. Data Factory<\/strong> servisinin Azure SQL Database<\/strong> servisine eri\u015febilmesi i\u00e7in a\u015fa\u011f\u0131daki ayar\u0131 yapmal\u0131s\u0131n\u0131z. <\/p>\n

<\/div>\n
<\/div>\n
<\/div>\n
Ba\u015far\u0131l\u0131 \u015fekilde ba\u011flant\u0131y\u0131 sa\u011flam\u0131\u015f olduk. \u015eimdi Azure SQL Veritaban\u0131<\/strong> aktar\u0131m i\u015flemini ger\u00e7ekle\u015ftirecek olan Data Pipeline’\u0131 olu\u015fturabiliriz. <\/div>\n
<\/div>\n
<\/div>\n
<\/div>\n
<\/div>\n
<\/div>\n
Olu\u015fturdu\u011fumuz Data Pipeline yukar\u0131daki gibidir ve \u00e7al\u0131\u015ft\u0131rd\u0131\u011f\u0131m\u0131zda da hata almadan i\u015flemi ger\u00e7ekle\u015ftirdi\u011fini g\u00f6rmekteyiz. Gelin bir de ilgili DB’miz alt\u0131nda tablolar olu\u015fmu\u015f mu ona bir bakal\u0131m. <\/div>\n
<\/div>\n
<\/div>\n
<\/div>\n

<\/p>\n

Evet istedi\u011fimiz tablolar ba\u015far\u0131l\u0131 bir \u015fekilde olu\u015fmu\u015f ve i\u00e7erisindeki veriler istedi\u011fimiz \u015fekilde ve tipte gelmi\u015f g\u00f6z\u00fck\u00fcyor.<\/div>\n

Umar\u0131m sizler i\u00e7in faydal\u0131 bir yaz\u0131 olmu\u015ftur. Herhangi bir sorunuz veya \u00f6neriniz var ise benimle ileti\u015fime ge\u00e7ebilirsiniz. Te\u015fekk\u00fcrler, g\u00f6r\u00fc\u015fmek \u00fczere \ud83d\udc4b <\/p>\n","protected":false},"excerpt":{"rendered":"

Merhabalar bu yaz\u0131mda sizlere Bulut Servis Sa\u011flay\u0131c\u0131lar\u0131ndan olan Azure \u00fczerinde bir Data Pipeline olu\u015fturma s\u00fcrecini anlataca\u011f\u0131m. Gelin isterseniz ilk \u00f6nce projemizin mimarisine […]<\/p>\n","protected":false},"author":1,"featured_media":5922,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[87],"tags":[],"class_list":["post-6110","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"yoast_head":"\nAzure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/\" \/>\n<meta property=\"og:locale\" content=\"tr_TR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Azure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego\" \/>\n<meta property=\"og:description\" content=\"Merhabalar bu yaz\u0131mda sizlere Bulut Servis Sa\u011flay\u0131c\u0131lar\u0131ndan olan Azure \u00fczerinde bir Data Pipeline olu\u015fturma s\u00fcrecini anlataca\u011f\u0131m. Gelin isterseniz ilk \u00f6nce projemizin mimarisine […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/\" \/>\n<meta property=\"og:site_name\" content=\"Bentego\" \/>\n<meta property=\"article:published_time\" content=\"2023-05-30T11:04:47+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2400\" \/>\n\t<meta property=\"og:image:height\" content=\"1600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Bentego\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Yazan:\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bentego\" \/>\n\t<meta name=\"twitter:label2\" content=\"Tahmini okuma s\u00fcresi\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 dakika\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/\"},\"author\":{\"name\":\"Bentego\",\"@id\":\"https:\/\/bentego.com\/tr\/#\/schema\/person\/e38e5868721eddd5493704ec7b5be37c\"},\"headline\":\"Azure \u00fczerinde Data Pipeline Olu\u015fturma\",\"datePublished\":\"2023-05-30T11:04:47+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/\"},\"wordCount\":1765,\"publisher\":{\"@id\":\"https:\/\/bentego.com\/tr\/#organization\"},\"image\":{\"@id\":\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png\",\"articleSection\":[\"Blog\"],\"inLanguage\":\"tr\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/\",\"url\":\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/\",\"name\":\"Azure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego\",\"isPartOf\":{\"@id\":\"https:\/\/bentego.com\/tr\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png\",\"datePublished\":\"2023-05-30T11:04:47+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#breadcrumb\"},\"inLanguage\":\"tr\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"tr\",\"@id\":\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage\",\"url\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png\",\"contentUrl\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png\",\"width\":2400,\"height\":1600},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/bentego.com\/tr\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Azure \u00fczerinde Data Pipeline Olu\u015fturma\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/bentego.com\/tr\/#website\",\"url\":\"https:\/\/bentego.com\/tr\/\",\"name\":\"Bentego\",\"description\":\"Turning data into enterprise value\",\"publisher\":{\"@id\":\"https:\/\/bentego.com\/tr\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/bentego.com\/tr\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"tr\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/bentego.com\/tr\/#organization\",\"name\":\"Bentego\",\"url\":\"https:\/\/bentego.com\/tr\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"tr\",\"@id\":\"https:\/\/bentego.com\/tr\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/05\/logo-bentego.svg\",\"contentUrl\":\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/05\/logo-bentego.svg\",\"width\":433,\"height\":109,\"caption\":\"Bentego\"},\"image\":{\"@id\":\"https:\/\/bentego.com\/tr\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/bentego.com\/tr\/#\/schema\/person\/e38e5868721eddd5493704ec7b5be37c\",\"name\":\"Bentego\",\"sameAs\":[\"http:\/\/bentego.com\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Azure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/","og_locale":"tr_TR","og_type":"article","og_title":"Azure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego","og_description":"Merhabalar bu yaz\u0131mda sizlere Bulut Servis Sa\u011flay\u0131c\u0131lar\u0131ndan olan Azure \u00fczerinde bir Data Pipeline olu\u015fturma s\u00fcrecini anlataca\u011f\u0131m. Gelin isterseniz ilk \u00f6nce projemizin mimarisine […]","og_url":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/","og_site_name":"Bentego","article_published_time":"2023-05-30T11:04:47+00:00","og_image":[{"width":2400,"height":1600,"url":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png","type":"image\/png"}],"author":"Bentego","twitter_card":"summary_large_image","twitter_misc":{"Yazan:":"Bentego","Tahmini okuma s\u00fcresi":"9 dakika"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#article","isPartOf":{"@id":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/"},"author":{"name":"Bentego","@id":"https:\/\/bentego.com\/tr\/#\/schema\/person\/e38e5868721eddd5493704ec7b5be37c"},"headline":"Azure \u00fczerinde Data Pipeline Olu\u015fturma","datePublished":"2023-05-30T11:04:47+00:00","mainEntityOfPage":{"@id":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/"},"wordCount":1765,"publisher":{"@id":"https:\/\/bentego.com\/tr\/#organization"},"image":{"@id":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage"},"thumbnailUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png","articleSection":["Blog"],"inLanguage":"tr"},{"@type":"WebPage","@id":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/","url":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/","name":"Azure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego","isPartOf":{"@id":"https:\/\/bentego.com\/tr\/#website"},"primaryImageOfPage":{"@id":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage"},"image":{"@id":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage"},"thumbnailUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png","datePublished":"2023-05-30T11:04:47+00:00","breadcrumb":{"@id":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#breadcrumb"},"inLanguage":"tr","potentialAction":[{"@type":"ReadAction","target":["https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/"]}]},{"@type":"ImageObject","inLanguage":"tr","@id":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage","url":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png","contentUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png","width":2400,"height":1600},{"@type":"BreadcrumbList","@id":"https:\/\/bentego.com\/tr\/azure-uzerinde-data-pipeline-olusturma\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/bentego.com\/tr\/"},{"@type":"ListItem","position":2,"name":"Azure \u00fczerinde Data Pipeline Olu\u015fturma"}]},{"@type":"WebSite","@id":"https:\/\/bentego.com\/tr\/#website","url":"https:\/\/bentego.com\/tr\/","name":"Bentego","description":"Turning data into enterprise value","publisher":{"@id":"https:\/\/bentego.com\/tr\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/bentego.com\/tr\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"tr"},{"@type":"Organization","@id":"https:\/\/bentego.com\/tr\/#organization","name":"Bentego","url":"https:\/\/bentego.com\/tr\/","logo":{"@type":"ImageObject","inLanguage":"tr","@id":"https:\/\/bentego.com\/tr\/#\/schema\/logo\/image\/","url":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/05\/logo-bentego.svg","contentUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/05\/logo-bentego.svg","width":433,"height":109,"caption":"Bentego"},"image":{"@id":"https:\/\/bentego.com\/tr\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/bentego.com\/tr\/#\/schema\/person\/e38e5868721eddd5493704ec7b5be37c","name":"Bentego","sameAs":["http:\/\/bentego.com"]}]}},"_links":{"self":[{"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/posts\/6110","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/comments?post=6110"}],"version-history":[{"count":0,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/posts\/6110\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/media\/5922"}],"wp:attachment":[{"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/media?parent=6110"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/categories?post=6110"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bentego.com\/tr\/wp-json\/wp\/v2\/tags?post=6110"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}