Have you ever faced a challenge where records keep getting duplicated when you are inserting some new data into an existing table in Databricks? If yes, then this blog is for you. Let’s start with a simple use case: Inserting parquet data from one folder in Datalake to a Delta table using Databricks. Follow the... Continue Reading →
To be fit or to be thin. Which one will you prefer? Above question will divide people into two categories: Wondering how both are different. They already know about the difference but unable to practice it in their lifestyles. This Blog is for both categories of people. Let’s start with the basics, what do you... Continue Reading →
Copying files from SharePoint to Datalake or any other target location is one task you cannot ignore as a Data Engineer. Someday someone for sure is going to ask you to do that. So, how can you achieve that? Suppose we need to copy an excel file from SharePoint to Datalake Gen 2. You need... Continue Reading →
The path is filled with thorns of pain and suffering,I am still going on; believing your love will be its ending. Each step I take, I meet with countless hurdles and loathing,I am still going on; keeping aside the fear of failing. Don't know how, when or whether there will be any benefit, but I... Continue Reading →
Step 1: In order to read .xlsx file, you need to have the library openpyxl installed in the Databricks cluster. Steps to install library openpyxl to Databircks cluster: Step 1: Select the Databricks cluster where you want to install the library. Step 2: Click on Libraries. Step 3: Click on Install New. Step 4: Select PyPI. Step 5: Put openpyxl in the text box under Package... Continue Reading →
SQL stands for Structured Query Language. SQL is a database management language for relational databases. SQL lets you access and manipulate databases. SQL queries are not case sensitive. (SELECT = select) Division of SQL: Data Manipulation Language (DML) is used to add, update or delete data . Examples: INSERT, DELETE and UPDATE. Data Definition Language (DDL) is used... Continue Reading →
Fashion is other name of confidence and comfort. Whatever your wear, wear as you were born in it.
As a Data Engineer you may come across a project where you need to Decrypt the PGP Encrypted files in order to get the data and apply transformation logic on it
Requirement: The main requirement is to be able to assign and predict House Price in a given location based on age of house, distance from Airport and number of conveniences stores within 5 km radius.
How to read .xlsx file: Step 1: In order to read .xlsx file, you need to have the library com.crealytics:spark-excel_2.11:0.12.2 installed in the Databricks cluster. Steps to install library com.crealytics:spark-excel_2.11:0.12.2 to Databircks cluster: Step 1: Select the Databricks cluster where you want to install the library. Step 2: Click on Libraries. Step 3: Click on... Continue Reading →
Cloud Computing refers to the process of making computing services available to users over the internet.
Reinforcement Learning is an area of Machine Learning which is applied when you don’t have training dataset and want the machine to learn from its experience.
Ization 