{"id":5196,"date":"2023-05-30T11:04:47","date_gmt":"2023-05-30T08:04:47","guid":{"rendered":"https:\/\/bentego.com\/?p=5196"},"modified":"2025-10-20T16:09:18","modified_gmt":"2025-10-20T16:09:18","slug":"azure-uzerinde-data-pipeline-olusturma","status":"publish","type":"post","link":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/","title":{"rendered":"Azure \u00fczerinde Data Pipeline Olu\u015fturma"},"content":{"rendered":"

Merhabalar bu yaz\u0131mda sizlere Bulut Servis Sa\u011flay\u0131c\u0131lar\u0131ndan olan\u00a0Azure<\/strong>\u00a0\u00fczerinde bir\u00a0Data Pipeline<\/strong>\u00a0olu\u015fturma s\u00fcrecini anlataca\u011f\u0131m. Gelin isterseniz ilk \u00f6nce projemizin mimarisine bakal\u0131m.<\/p>\n

<\/p>\n

Mimarimiz yukar\u0131da g\u00f6rm\u00fc\u015f oldu\u011funuz gibidir. Veri ak\u0131\u015f\u0131 anlat\u0131m\u0131na ge\u00e7meden \u00f6nce\u00a0Azure<\/strong>\u00a0\u00fczerinde kulland\u0131\u011f\u0131m\u0131z servisleri k\u0131saca bir tan\u0131yal\u0131m.<\/p>\n

Azure Resource Group \u2192 Microsoft Azure<\/strong>\u00a0bulut platformunda kaynaklar\u0131n\u0131z\u0131 y\u00f6netmek i\u00e7in kullan\u0131lan bir yap\u0131d\u0131r.\u00a0Azure<\/strong>\u2019da olu\u015fturdu\u011funuz kaynaklar\u0131, sanal makineleri, depolama hesaplar\u0131n\u0131, a\u011f yap\u0131land\u0131rmalar\u0131n\u0131, uygulamalar\u0131 ve di\u011fer hizmetleri mant\u0131ksal bir grup i\u00e7inde toplaman\u0131z\u0131 sa\u011flar.<\/p>\n

Azure Data Factory \u2192 Microsoft Azure<\/strong>\u00a0bulut platformunda veri entegrasyonu ve veri i\u015fleme i\u015flerini otomatikle\u015ftirmek i\u00e7in kullan\u0131lan bir hizmettir.\u00a0Azure Data Factory<\/strong>, farkl\u0131 veri kaynaklar\u0131 aras\u0131nda\u00a0veri hareketini<\/strong>,\u00a0d\u00f6n\u00fc\u015f\u00fcm\u00fcn\u00fc<\/strong>\u00a0ve\u00a0i\u015flenmesini<\/strong>\u00a0sa\u011flayarak\u00a0veri entegrasyonunu<\/strong>\u00a0kolayla\u015ft\u0131r\u0131r.<\/p>\n

Azure Synapse Analytics \u2192 Microsoft Azure<\/strong>\u00a0bulut platformunda,\u00a0verilerinizi analiz etmek, i\u015flemek ve raporlamak<\/strong>\u00a0i\u00e7in kullanabilece\u011finiz g\u00fc\u00e7l\u00fc bir veri analiti\u011fi \u00e7\u00f6z\u00fcm\u00fcd\u00fcr.Azure Synapse Analytics<\/strong>, veri ambar\u0131, veri entegrasyonu, veri sorgulama ve\u00a0b\u00fcy\u00fck veri analiti\u011fi<\/strong>\u00a0gibi i\u015flemleri\u00a0tek bir platformda<\/strong>\u00a0birle\u015ftirir.<\/p>\n

Azure Data Lake \u2192\u00a0<\/strong>B\u00fcy\u00fck veri depolama ve analizi i\u00e7in kullan\u0131lan bir hizmettir.\u00a0Azure<\/strong>\u00a0bulut platformunda,\u00a0yap\u0131land\u0131r\u0131lm\u0131\u015f<\/strong>\u00a0ve\u00a0yap\u0131land\u0131r\u0131lmam\u0131\u015f<\/strong>\u00a0b\u00fcy\u00fck veri k\u00fctlelerini\u00a0g\u00fcvenli ve \u00f6l\u00e7eklenebilir<\/strong>\u00a0bir \u015fekilde\u00a0depolamak<\/strong>\u00a0i\u00e7in kullan\u0131l\u0131r.<\/p>\n

Azure SQL Database \u2192 Microsoft Azure<\/strong>\u00a0bulut platformunda bar\u0131nd\u0131r\u0131lan ili\u015fkisel veritaban\u0131 hizmetidir. Geleneksel\u00a0SQL Server<\/strong>\u00a0veritaban\u0131n\u0131n\u00a0bulut tabanl\u0131 bir s\u00fcr\u00fcm\u00fc<\/strong>\u00a0olarak d\u00fc\u015f\u00fcn\u00fclebilir.<\/p>\n

Mimariye bakt\u0131\u011f\u0131m\u0131zda ilk k\u0131s\u0131mda\u00a0On-Premise<\/strong>\u00a0tarafta ger\u00e7ekle\u015fen i\u015flemlerimiz mevcut. Gelin isterseniz ilk olarak lokalimizde \u00e7al\u0131\u015fan python koduna ve \u00e7\u0131kt\u0131s\u0131na bir bakal\u0131m.<\/p>\n

\n

import<\/span> spotipy
\nfrom<\/span> spotipy.oauth2 import<\/span> SpotifyClientCredentials
\nimport<\/span> pandas as<\/span> pd
\nfrom<\/span> datetime import<\/span> date
\nimport<\/span> configparser<\/p>\n

parser = configparser.ConfigParser()
\nparser.read(“conf.ini”<\/span>)<\/p>\n

client_credentials_manager = SpotifyClientCredentials(
\nclient_id = parser.get(“credentials”<\/span>,“CLIENT_ID”<\/span>),
\nclient_secret = parser.get(“credentials”<\/span>, “CLIENT_SECRET”<\/span>)
\n)<\/p>\n

sp = spotipy.Spotify(client_credentials_manager = client_credentials_manager)<\/p>\n

playlist_link = ‘https:\/\/open.spotify.com\/playlist\/37i9dQZF1EIWxkuBxGv3mp’<\/span><\/p>\n

playlist_URI = playlist_link.split(“\/”<\/span>)[-1<\/span>]
\ndata = sp.playlist_tracks(playlist_URI)
\ndate_str = str<\/span>(date.today())<\/p>\n

album_list = []<\/p>\n

for<\/span> row in<\/span> data[“items”<\/span>]:
\nalbum_id = row[“track”<\/span>][“album”<\/span>][“id”<\/span>]
\nalbum_name = row[“track”<\/span>][“album”<\/span>][“name”<\/span>]
\nalbum_release_date = row[“track”<\/span>][“album”<\/span>][“release_date”<\/span>]
\nalbum_total_tracks = row[“track”<\/span>][“album”<\/span>][“total_tracks”<\/span>]
\nalbum_url = row[“track”<\/span>][“album”<\/span>][“external_urls”<\/span>][“spotify”<\/span>]
\nalbum_element = {
\n\u00a0 \u00a0 \u00a0 \u00a0“album_id”<\/span>: album_id,
\n\u00a0 \u00a0 \u00a0 \u00a0“name”<\/span>: album_name,
\n\u00a0 \u00a0 \u00a0 \u00a0“album_release_date”<\/span>: album_release_date,
\n\u00a0 \u00a0 \u00a0 \u00a0“album_total_tracks”<\/span>: album_total_tracks,
\n\u00a0 \u00a0 \u00a0 \u00a0“album_url”<\/span>: album_url
\n}
\nalbum_list.append(album_element)<\/p>\n

artist_list = []<\/p>\n

for<\/span> row in<\/span> data[“items”<\/span>]:
\n\u00a0 for<\/span> key, value in<\/span> row.items():
\n\u00a0 \u00a0 if<\/span> key == “track”<\/span>:
\n\u00a0 \u00a0 \u00a0 for<\/span> artist in<\/span> value[“artists”<\/span>]:
\nartist_dict = {
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0“artist_id”<\/span>: artist[“id”<\/span>],
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0“artist_name”<\/span>: artist[“name”<\/span>],
\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0“external_url”<\/span>: artist[“href”<\/span>]
\n}
\nartist_list.append(artist_dict)<\/p>\n

song_list = []<\/p>\n

for<\/span> row in<\/span> data[“items”<\/span>]:
\nsong_id = row[“track”<\/span>][“id”<\/span>]
\nsong_name = row[“track”<\/span>][“name”<\/span>]
\nsong_duration = row[“track”<\/span>][“duration_ms”<\/span>]
\nsong_url = row[“track”<\/span>][“external_urls”<\/span>][“spotify”<\/span>]
\nsong_popularity = row[“track”<\/span>][“popularity”<\/span>]
\nsong_added = row[“added_at”<\/span>]
\nalbum_id = row[“track”<\/span>][“album”<\/span>][“id”<\/span>]
\nartist_id = row[“track”<\/span>][“album”<\/span>][“artists”<\/span>][0<\/span>][“id”<\/span>]
\nsong_element = {
\n\u00a0 \u00a0 “song_id”<\/span>: song_id,
\n\u00a0 \u00a0 “song_name”<\/span>: song_name,
\n\u00a0 \u00a0 “duration_ms”<\/span>: song_duration,
\n\u00a0 \u00a0 “url”<\/span>: song_url,
\n\u00a0 \u00a0 “popularity”<\/span>: song_popularity,
\n\u00a0 \u00a0 “song_added”<\/span>: song_added,
\n\u00a0 \u00a0 “album_id”<\/span>: album_id,
\n\u00a0 \u00a0 “artist_id”<\/span>: artist_id
\n}
\nsong_list.append(song_element)<\/p>\n

album_df = pd.DataFrame.from_dict(album_list)
\nalbum_df = album_df.drop_duplicates(subset = [“album_id”<\/span>])<\/p>\n

artist_df = pd.DataFrame.from_dict(artist_list)
\nartist_df = artist_df.drop_duplicates(subset = [“artist_id”<\/span>])<\/p>\n

song_df = pd.DataFrame.from_dict(song_list)
\nalbum_df[“release_date”<\/span>] = pd.to_datetime(album_df[“album_release_date”<\/span>])
\nsong_df[“song_added”<\/span>] = pd.to_datetime(song_df[“song_added”<\/span>])<\/p>\n

song_df.to_parquet(f”data\\\\{date_str}<\/span>_song_ds.parquet”<\/span>,engine=“fastparquet”<\/span>)
\nalbum_df.to_parquet(f”data\\\\{date_str}<\/span>_album_ds.parquet”<\/span>,engine=“fastparquet”<\/span>)
\nartist_df.to_parquet(f”data\\\\{date_str}<\/span>_artist_ds.parquet”<\/span>,engine=“fastparquet”<\/span>)<\/p><\/blockquote>\n

Conf.ini<\/strong><\/h3>\n
\n

[credentials]
\nCLIENT_ID = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
\nCLIENT_SECRET = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX<\/p>\n<\/blockquote>\n

<\/p>\n

G\u00f6rm\u00fc\u015f oldu\u011funuz\u00a0Python<\/strong>\u00a0kodu\u00a0Spotify API<\/strong>\u2019na istekte bulunarak d\u00f6nen sonu\u00e7tan 3 farkl\u0131 veri seti elde ederek bunlar\u0131\u00a0parquet\u00a0<\/strong>format\u0131nda belirtilen dizine kaydediyor.<\/p>\n

Evet lokalimizdeki i\u015flemleri b\u00f6ylelikle bitiyoruz. \u015eimdi\u00a0Azure\u00a0<\/strong>\u00fczerindeki servislerimizi olu\u015fturarak i\u015flemlerimize devam edelim.<\/p>\n

Resource Group<\/strong>\u00a0Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

Azure Data Lake<\/strong>\u00a0Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

<\/p>\n

Azure Data Factory\u00a0<\/strong>Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

Azure Synapse Analytics\u00a0<\/strong>Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

<\/p>\n

Azure SQL Database\u00a0<\/strong>Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

<\/p>\n

Gerekli servisleri olu\u015fturdu\u011fumuza g\u00f6re\u00a0Azure Data Factory Studio\u00a0<\/strong>ekran\u0131na giderek\u00a0Manage<\/strong>\u00a0sekmesinden\u00a0Integration Runtimes(IR)<\/strong>\u00a0k\u0131sm\u0131na gelerek lokalimizle ba\u011flant\u0131 sa\u011flayacak yeni bir tanesini olu\u015fturaca\u011f\u0131z.<\/p>\n

Self-Hosted IR<\/strong>\u00a0Olu\u015fturma \u0130\u015flemi:<\/h3>\n

<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

Option 2<\/strong>‘deki link\u2019e t\u0131klayarak indirme i\u015flemini ger\u00e7ekle\u015ftiriyoruz. \u0130\u015flem bittikten sonra \u00e7\u0131kacak ekrana\u00a0Key1<\/strong>‘i kopyalayarak ilgili alana yap\u0131\u015ft\u0131r\u0131yoruz.<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

Ba\u015far\u0131l\u0131 bir \u015fekilde lokalimiz ile ba\u011flant\u0131 sa\u011flayacak ve ilgili i\u015flemleri orada ger\u00e7ekle\u015ftirecek olan\u00a0IR<\/strong>\u2019\u0131 olu\u015fturmu\u015f olduk.<\/p>\n

Data Factory<\/strong>\u00a0\u00fczerinde olu\u015fturaca\u011f\u0131m\u0131z Pipeline da lokalimizde ki parquet dosyalar\u0131n\u0131 kullanabilmek i\u00e7in\u00a0Linked Service<\/strong>\u00a0olu\u015fturaca\u011f\u0131z onu da a\u015fa\u011f\u0131daki ad\u0131mlar\u0131 takip ederek olu\u015fturabilirsiniz.<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

Olu\u015fturaca\u011f\u0131m\u0131z Data Pipeline i\u00e7erisinde bu\u00a0LS(Linked Service)\u2019i\u00a0<\/strong>kullanarak lokal\u2019deki her bir parquet format\u0131 e\u011fer bug\u00fcn\u00fcn tarihi ile ba\u015fl\u0131yorsa onu alarak\u00a0Azure Data Lake<\/strong>\u2019deki belirtti\u011fimiz dizine kopyalayacak.<\/p>\n

DP_Extract_OnpremToADL2:<\/strong><\/h3>\n

<\/p>\n

Yukar\u0131da g\u00f6rm\u00fc\u015f oldu\u011funuz\u00a0Data Pipelin<\/strong>e\u2019\u0131 a\u00e7\u0131klamak gerekirse yapt\u0131\u011f\u0131 i\u015f\u00a0Get Metadata<\/strong>\u2019y\u0131 kullanarak lokal\u2019deki her\u00a0parquet\u00a0<\/strong>dosyas\u0131n\u0131n dosya ismini\u00a0ForEach Loop<\/strong>\u2019unda d\u00f6nd\u00fcrek e\u011fer bug\u00fcn\u00fcn tarihiyle ba\u015fl\u0131yorsa kopyalama i\u015flemini ger\u00e7ekle\u015ftirecek e\u011fer de\u011filse hi\u00e7bir i\u015flem yapmayacak.Bu kontrol\u00fc yapan Activity\u00a0\u2018If Condition\u2019\u00a0<\/strong>ve ko\u015fulu yapan kod ise a\u015fa\u011f\u0131daki gibidir.<\/p>\n

<\/p>\n

<\/p>\n

Data Pipeline\u2019\u0131 \u00e7al\u0131\u015ft\u0131rd\u0131\u011f\u0131m\u0131zda ba\u015far\u0131l\u0131 bir \u015fekilde \u00e7al\u0131\u015ft\u0131\u011f\u0131n\u0131 g\u00f6rmekteyiz.\u00a0Azure Data Lake<\/strong>\u00a0\u00fczerindeki belirtti\u011fimiz dizini kontrol ederek veriler ba\u015far\u0131l\u0131 \u015fekilde gelmi\u015f mi \u015fimdi ona bakal\u0131m.<\/p>\n

<\/p>\n

Lokalimizdeki veri setlerinin ba\u015far\u0131l\u0131 \u015fekilde kopyaland\u0131\u011f\u0131 g\u00f6rmekteyiz.<\/p>\n

Bir sonraki a\u015famam\u0131z\u00a0Azure Synapse Analytics<\/strong>\u00a0\u00fczerinde\u00a0Spark Pool<\/strong>\u00a0olu\u015fturarak veri temizleme ve d\u00f6n\u00fc\u015ft\u00fcrme i\u015flemi uygulayaca\u011f\u0131z.<\/p>\n

Spark Pool<\/strong>\u00a0Olu\u015fturma \u0130\u015flemi:<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\n

\n

from<\/span> notebookutils import<\/span> mssparkutils
\nfrom<\/span> pyspark.sql.functions import<\/span> *
\nfrom<\/span> pyspark.sql.types import<\/span> *<\/p>\n

accountname=“bentegostg”<\/span>
\nsourcecontainer=“raw”<\/span>
\nLinkedService=“bentego-synapse-ws-WorkspaceDefaultStorage”<\/span>
\nsourceFile_location=“ingest”<\/span>
\ndestinationcontainer=“refined”<\/span><\/p>\n

spark.conf.set<\/span>(“spark.storage.synapse.linkedServiceName”<\/span>,LinkedService)
\nspark.conf.set<\/span>(“fs.azure.account.oauth.provider.typ”<\/span>,“com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider”<\/span>)
\nspark.conf.set<\/span>(“spark.sql.parquet.writeLegacyFormat”<\/span>,“true”<\/span>)<\/p>\n

SongCustomSchema = StructType([
\nStructField(“song_id”<\/span>, StringType(), True<\/span>),
\nStructField(“song_name”<\/span>, StringType(), True<\/span>),
\nStructField(“duration_ms”<\/span>, LongType(), True<\/span>),
\nStructField(“url”<\/span>, StringType(), True<\/span>),
\nStructField(“popularity”<\/span>, LongType(), True<\/span>),
\nStructField(“song_added”<\/span>, StringType(), True<\/span>),
\nStructField(“album_id”<\/span>, StringType(), True<\/span>),
\nStructField(“artist_id”<\/span>, StringType(), True<\/span>)
\n]
\n)<\/p>\n

song_df = spark \\
\n.read \\
\n.schema(SongCustomSchema) \\
\n.parquet(f”abfss:\/\/{sourcecontainer}<\/span>@{accountname}<\/span>.dfs.core.windows.net\/{sourceFile_location}<\/span>\/2023-05-29_song_ds.parquet”<\/span>)<\/p>\n

song_df = song_df \\
\n.withColumn(“duration_ms”<\/span>,col(“duration_ms”<\/span>).cast(IntegerType())) \\
\n.withColumn(“popularity”<\/span>,col(“popularity”<\/span>).cast(IntegerType())) \\
\n.drop(col(“song_added”<\/span>))<\/p>\n

AlbumCustomSchema = StructType([
\nStructField(“album_id”<\/span>, StringType(), True<\/span>),
\nStructField(“name”<\/span>, StringType(), True<\/span>),
\nStructField(“album_release_date”<\/span>, StringType(), True<\/span>),
\nStructField(“album_total_tracks”<\/span>, LongType(), True<\/span>),
\nStructField(“album_url”<\/span>, StringType(), True<\/span>),
\nStructField(“release_date”<\/span>, StringType(), True<\/span>)
\n]
\n)<\/p>\n

album_df = spark \\
\n.read \\
\n.schema(AlbumCustomSchema) \\
\n.parquet(f”abfss:\/\/{sourcecontainer}<\/span>@{accountname}<\/span>.dfs.core.windows.net\/{sourceFile_location}<\/span>\/2023-05-29_album_ds.parquet”<\/span>)<\/p>\n

album_df = album_df \\
\n.withColumn(“album_total_tracks”<\/span>,col(“album_total_tracks”<\/span>).cast(IntegerType())) \\
\n.withColumn(“album_release_date”<\/span>,to_date(col(“album_release_date”<\/span>),“yyyy-MM-dd”<\/span>)) \\
\n.drop(“release_date”<\/span>)<\/p>\n

ArtistCustomSchema = StructType([
\nStructField(“artist_id”<\/span>, StringType(), True<\/span>),
\nStructField(“artist_name”<\/span>, StringType(), True<\/span>),
\nStructField(“external_url”<\/span>, StringType(), True<\/span>),
\n]
\n)<\/p>\n

artist_df = spark \\
\n.read \\
\n.schema(ArtistCustomSchema) \\
\n.parquet(f”abfss:\/\/{sourcecontainer}<\/span>@{accountname}<\/span>.dfs.core.windows.net\/{sourceFile_location}<\/span>\/2023-05-29_artist_ds.parquet”<\/span>)<\/p>\n

album_df.write \\
\n.mode(“overwrite”<\/span>) \\
\n.format<\/span>(“parquet”<\/span>) \\
\n.option(“header”<\/span>,“true”<\/span>) \\
\n.option(“inferSchema”<\/span>,“true”<\/span>) \\
\n.save(f”abfss:\/\/{destinationcontainer}<\/span>@{accountname}<\/span>.dfs.core.windows.net\/album\/”<\/span>)<\/p>\n

artist_df.write \\
\n.mode(“overwrite”<\/span>) \\
\n.format<\/span>(“parquet”<\/span>) \\
\n.option(“header”<\/span>,“true”<\/span>) \\
\n.option(“inferSchema”<\/span>,“true”<\/span>) \\
\n.save(f”abfss:\/\/{destinationcontainer}<\/span>@{accountname}<\/span>.dfs.core.windows.net\/artist\/”<\/span>)<\/p><\/blockquote>\n

Spark Pool<\/strong>\u2019u olu\u015fturduktan sonra\u00a0Developer<\/strong>\u00a0sekmesinden\u00a0Notebook<\/strong>\u00a0olu\u015fturarak yukar\u0131daki g\u00f6rm\u00fc\u015f oldu\u011funuz kodlar\u0131 yaz\u0131yoruz. Normal\u00a0PySpark\u00a0<\/strong>kodlar\u0131m\u0131z\u0131 yazarak verinin d\u00fczg\u00fcn ve temizlenmi\u015f halini yine Azure Data Lake \u00fczerindeki belirti\u011fimiz dizinlere yolluyoruz.<\/p>\n

<\/p>\n

<\/p>\n

Kontrol etti\u011fimizde verilerin ba\u015far\u0131l\u0131 \u015fekilde dizinlere yaz\u0131ld\u0131\u011f\u0131n\u0131 g\u00f6rmekteyiz.<\/p>\n

Yukar\u0131da olu\u015fturdu\u011fumuz Notebook\u2019u\u00a0Data Factory\u00a0<\/strong>\u00fczerinde olu\u015fturdu\u011fumuz Data Pipeline i\u00e7erisinden \u00e7a\u011f\u0131rmak i\u00e7in\u00a0Azure Data Factory\u2019<\/strong>i servisine\u00a0Synapse Administrator<\/strong>\u00a0yetkisi vermemiz gerekli.<\/p>\n

<\/p>\n

\u015eimdi\u00a0Azure Data Factory<\/strong>\u00a0k\u0131sm\u0131na gidirek\u00a0Synapse<\/strong>\u00a0k\u0131sm\u0131nda olu\u015fturdu\u011fumuz Notebook\u2019u ba\u011flanabilmesi i\u00e7in yeni bir\u00a0Linked Service<\/strong>\u00a0olu\u015ftural\u0131m.<\/p>\n

<\/p>\n

<\/p>\n

Ba\u015far\u0131l\u0131 \u015fekilde ba\u011flant\u0131m\u0131z\u0131 olu\u015fturduk. Data Pipeline olu\u015ftururak Notebook\u2019u birde oradan \u00e7a\u011f\u0131ral\u0131m.<\/p>\n

<\/p>\n

<\/p>\n

Data Pipeline\u2019\u0131 \u00e7al\u0131\u015ft\u0131rd\u0131\u011f\u0131m\u0131zda ba\u015far\u0131l\u0131 bir \u015fekilde \u00e7al\u0131\u015ft\u0131\u011f\u0131n\u0131 g\u00f6rmekteyiz.<\/p>\n

\u00c7al\u0131\u015fan\u00a0Pyspark Job<\/strong>\u2019\u0131m\u0131z temizlenmi\u015f ve d\u00f6n\u00fc\u015ft\u00fcr\u00fclm\u00fc\u015f veriyi ba\u015fka bir dizine kaydetmi\u015f oldu. Bu veri setlerini \u015fimdi de\u00a0Azure SQL Database<\/strong>\u00a0k\u0131sm\u0131nda olu\u015fturdu\u011fumuz SpotifyDB\u2019sinin alt\u0131nda tablolar olu\u015fturarak oraya aktaraca\u011f\u0131z.\u00a0Data Factory<\/strong>\u00a0servisinin\u00a0Azure SQL Database<\/strong>\u00a0servisine eri\u015febilmesi i\u00e7in a\u015fa\u011f\u0131daki ayar\u0131 yapmal\u0131s\u0131n\u0131z.<\/p>\n

<\/div>\n
<\/div>\n
<\/div>\n
Ba\u015far\u0131l\u0131 \u015fekilde ba\u011flant\u0131y\u0131 sa\u011flam\u0131\u015f olduk. \u015eimdi Azure SQL Database<\/strong> aktar\u0131m i\u015flemini ger\u00e7ekle\u015ftirecek olan Data Pipeline\u2019\u0131 olu\u015fturabiliriz.<\/div>\n
<\/div>\n
<\/div>\n
<\/div>\n
<\/div>\n
<\/div>\n
Olu\u015fturdu\u011fumuz Data Pipeline yukar\u0131daki gibidir ve \u00e7al\u0131\u015ft\u0131rd\u0131\u011f\u0131m\u0131zda da hata almadan i\u015flemi ger\u00e7ekle\u015ftirdi\u011fini g\u00f6rmekteyiz. Gelin bir de ilgili DB\u2019miz alt\u0131nda tablolar olu\u015fmu\u015f mu ona bir bakal\u0131m.<\/div>\n
<\/div>\n
<\/div>\n
<\/div>\n

<\/p>\n

Evet istedi\u011fimiz tablolar ba\u015far\u0131l\u0131 bir \u015fekilde olu\u015fmu\u015f ve i\u00e7erisindeki veriler istedi\u011fimiz \u015fekilde ve tipte gelmi\u015f g\u00f6z\u00fck\u00fcyor.<\/div>\n

Umar\u0131m sizler i\u00e7in faydal\u0131 bir yaz\u0131 olmu\u015ftur. Herhangi bir sorunuz veya \u00f6neriniz var ise benimle ileti\u015fime ge\u00e7ebilirsiniz. Te\u015fekk\u00fcrler, g\u00f6r\u00fc\u015fmek \u00fczere \ud83d\udc4b<\/p>\n","protected":false},"excerpt":{"rendered":"

Merhabalar bu yaz\u0131mda sizlere Bulut Servis Sa\u011flay\u0131c\u0131lar\u0131ndan olan\u00a0Azure\u00a0\u00fczerinde bir\u00a0Data Pipeline\u00a0olu\u015fturma s\u00fcrecini anlataca\u011f\u0131m. Gelin isterseniz ilk \u00f6nce projemizin mimarisine bakal\u0131m. Mimarimiz yukar\u0131da g\u00f6rm\u00fc\u015f […]<\/p>\n","protected":false},"author":2,"featured_media":1010,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-5196","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"yoast_head":"\nAzure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Azure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego\" \/>\n<meta property=\"og:description\" content=\"Merhabalar bu yaz\u0131mda sizlere Bulut Servis Sa\u011flay\u0131c\u0131lar\u0131ndan olan\u00a0Azure\u00a0\u00fczerinde bir\u00a0Data Pipeline\u00a0olu\u015fturma s\u00fcrecini anlataca\u011f\u0131m. Gelin isterseniz ilk \u00f6nce projemizin mimarisine bakal\u0131m. Mimarimiz yukar\u0131da g\u00f6rm\u00fc\u015f […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/\" \/>\n<meta property=\"og:site_name\" content=\"Bentego\" \/>\n<meta property=\"article:published_time\" content=\"2023-05-30T08:04:47+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-10-20T16:09:18+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2400\" \/>\n\t<meta property=\"og:image:height\" content=\"1600\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Bentego\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bentego\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"12 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/\"},\"author\":{\"name\":\"Bentego\",\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/#\\\/schema\\\/person\\\/0348418b7b0cbca83fdd7a899d54821e\"},\"headline\":\"Azure \u00fczerinde Data Pipeline Olu\u015fturma\",\"datePublished\":\"2023-05-30T08:04:47+00:00\",\"dateModified\":\"2025-10-20T16:09:18+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/\"},\"wordCount\":1753,\"publisher\":{\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/bentego.com\\\/wp-content\\\/uploads\\\/2025\\\/06\\\/Frame-82__.png\",\"articleSection\":[\"Blog\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/\",\"url\":\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/\",\"name\":\"Azure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/bentego.com\\\/wp-content\\\/uploads\\\/2025\\\/06\\\/Frame-82__.png\",\"datePublished\":\"2023-05-30T08:04:47+00:00\",\"dateModified\":\"2025-10-20T16:09:18+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/#primaryimage\",\"url\":\"https:\\\/\\\/bentego.com\\\/wp-content\\\/uploads\\\/2025\\\/06\\\/Frame-82__.png\",\"contentUrl\":\"https:\\\/\\\/bentego.com\\\/wp-content\\\/uploads\\\/2025\\\/06\\\/Frame-82__.png\",\"width\":2400,\"height\":1600},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/azure-uzerinde-data-pipeline-olusturma\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/bentego.com\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Azure \u00fczerinde Data Pipeline Olu\u015fturma\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/bentego.com\\\/en\\\/\",\"name\":\"Bentego\",\"description\":\"Turning data into enterprise value\",\"publisher\":{\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/bentego.com\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/#organization\",\"name\":\"Bentego\",\"url\":\"https:\\\/\\\/bentego.com\\\/en\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/bentego.com\\\/wp-content\\\/uploads\\\/2025\\\/05\\\/logo-bentego.svg\",\"contentUrl\":\"https:\\\/\\\/bentego.com\\\/wp-content\\\/uploads\\\/2025\\\/05\\\/logo-bentego.svg\",\"width\":433,\"height\":109,\"caption\":\"Bentego\"},\"image\":{\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/bentego.com\\\/en\\\/#\\\/schema\\\/person\\\/0348418b7b0cbca83fdd7a899d54821e\",\"name\":\"Bentego\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Azure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/","og_locale":"en_US","og_type":"article","og_title":"Azure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego","og_description":"Merhabalar bu yaz\u0131mda sizlere Bulut Servis Sa\u011flay\u0131c\u0131lar\u0131ndan olan\u00a0Azure\u00a0\u00fczerinde bir\u00a0Data Pipeline\u00a0olu\u015fturma s\u00fcrecini anlataca\u011f\u0131m. Gelin isterseniz ilk \u00f6nce projemizin mimarisine bakal\u0131m. Mimarimiz yukar\u0131da g\u00f6rm\u00fc\u015f […]","og_url":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/","og_site_name":"Bentego","article_published_time":"2023-05-30T08:04:47+00:00","article_modified_time":"2025-10-20T16:09:18+00:00","og_image":[{"width":2400,"height":1600,"url":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png","type":"image\/png"}],"author":"Bentego","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Bentego","Est. reading time":"12 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/#article","isPartOf":{"@id":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/"},"author":{"name":"Bentego","@id":"https:\/\/bentego.com\/en\/#\/schema\/person\/0348418b7b0cbca83fdd7a899d54821e"},"headline":"Azure \u00fczerinde Data Pipeline Olu\u015fturma","datePublished":"2023-05-30T08:04:47+00:00","dateModified":"2025-10-20T16:09:18+00:00","mainEntityOfPage":{"@id":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/"},"wordCount":1753,"publisher":{"@id":"https:\/\/bentego.com\/en\/#organization"},"image":{"@id":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage"},"thumbnailUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png","articleSection":["Blog"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/","url":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/","name":"Azure \u00fczerinde Data Pipeline Olu\u015fturma - Bentego","isPartOf":{"@id":"https:\/\/bentego.com\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage"},"image":{"@id":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage"},"thumbnailUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png","datePublished":"2023-05-30T08:04:47+00:00","dateModified":"2025-10-20T16:09:18+00:00","breadcrumb":{"@id":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/#primaryimage","url":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png","contentUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/06\/Frame-82__.png","width":2400,"height":1600},{"@type":"BreadcrumbList","@id":"https:\/\/bentego.com\/en\/azure-uzerinde-data-pipeline-olusturma\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/bentego.com\/en\/"},{"@type":"ListItem","position":2,"name":"Azure \u00fczerinde Data Pipeline Olu\u015fturma"}]},{"@type":"WebSite","@id":"https:\/\/bentego.com\/en\/#website","url":"https:\/\/bentego.com\/en\/","name":"Bentego","description":"Turning data into enterprise value","publisher":{"@id":"https:\/\/bentego.com\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/bentego.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/bentego.com\/en\/#organization","name":"Bentego","url":"https:\/\/bentego.com\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/bentego.com\/en\/#\/schema\/logo\/image\/","url":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/05\/logo-bentego.svg","contentUrl":"https:\/\/bentego.com\/wp-content\/uploads\/2025\/05\/logo-bentego.svg","width":433,"height":109,"caption":"Bentego"},"image":{"@id":"https:\/\/bentego.com\/en\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/bentego.com\/en\/#\/schema\/person\/0348418b7b0cbca83fdd7a899d54821e","name":"Bentego"}]}},"_links":{"self":[{"href":"https:\/\/bentego.com\/en\/wp-json\/wp\/v2\/posts\/5196","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bentego.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bentego.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bentego.com\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/bentego.com\/en\/wp-json\/wp\/v2\/comments?post=5196"}],"version-history":[{"count":2,"href":"https:\/\/bentego.com\/en\/wp-json\/wp\/v2\/posts\/5196\/revisions"}],"predecessor-version":[{"id":5662,"href":"https:\/\/bentego.com\/en\/wp-json\/wp\/v2\/posts\/5196\/revisions\/5662"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/bentego.com\/en\/wp-json\/wp\/v2\/media\/1010"}],"wp:attachment":[{"href":"https:\/\/bentego.com\/en\/wp-json\/wp\/v2\/media?parent=5196"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bentego.com\/en\/wp-json\/wp\/v2\/categories?post=5196"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bentego.com\/en\/wp-json\/wp\/v2\/tags?post=5196"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}