A good Data Analyst should combine technical expertise, strong analytical skills, business acumen, and effective communication to deliver valuable insights and drive data-informed decision-making within an organization.
Azure SQL DB is a fully managed relational database service, Azure Dedicated SQL pool is a distributed analytics service for large datasets, and Elastic Pool is a resource allocation model for managing multiple SQL databases with varying workloads.
The field of data analytics is evolving rapidly, and new trends and technologies are emerging every month, below are the latest trends as of now: Augmented Analytics: Augmented analytics combines artificial intelligence (AI) and machine learning (ML) techniques with data analytics to automate data preparation, insight generation, and data visualization. It empowers non-technical users to... Continue Reading →
Microsoft Copilot is an AI-powered code completion tool developed by Microsoft in collaboration with OpenAI that uses machine learning to provide code suggestions and completions to developers as they write code. It is designed to help developers be more productive by automating some of the more repetitive tasks of coding. It has the potential to... Continue Reading →
AI has become the Buzzword of the 21st century, anyone who even has a little idea of what AI can do is either excited, nervous, or fearful of it, and rightly so.
Have you ever faced a challenge where records keep getting duplicated when you are inserting some new data into an existing table in Databricks? If yes, then this blog is for you. Let’s start with a simple use case: Inserting parquet data from one folder in Datalake to a Delta table using Databricks. Follow the... Continue Reading →
Copying files from SharePoint to Datalake or any other target location is one task you cannot ignore as a Data Engineer. Someday someone for sure is going to ask you to do that. So, how can you achieve that? Suppose we need to copy an excel file from SharePoint to Datalake Gen 2. You need... Continue Reading →
Step 1: In order to read .xlsx file, you need to have the library openpyxl installed in the Databricks cluster. Steps to install library openpyxl to Databircks cluster: Step 1: Select the Databricks cluster where you want to install the library. Step 2: Click on Libraries. Step 3: Click on Install New. Step 4: Select PyPI. Step 5: Put openpyxl in the text box under Package... Continue Reading →
SQL stands for Structured Query Language. SQL is a database management language for relational databases. SQL lets you access and manipulate databases. SQL queries are not case sensitive. (SELECT = select) Division of SQL: Data Manipulation Language (DML) is used to add, update or delete data . Examples: INSERT, DELETE and UPDATE. Data Definition Language (DDL) is used... Continue Reading →
As a Data Engineer you may come across a project where you need to Decrypt the PGP Encrypted files in order to get the data and apply transformation logic on it
How to read .xlsx file: Step 1: In order to read .xlsx file, you need to have the library com.crealytics:spark-excel_2.11:0.12.2 installed in the Databricks cluster. Steps to install library com.crealytics:spark-excel_2.11:0.12.2 to Databircks cluster: Step 1: Select the Databricks cluster where you want to install the library. Step 2: Click on Libraries. Step 3: Click on... Continue Reading →
Big Data refers to the data which is nearly impossible to process using a single computer as it is huge in size, generated frequently and is in many different formats.
Ization 