Use Python to combine multiple Excel files into one master spreadsheet

Sharing is caring!

Last Updated on April 9, 2023 by Jay

In this short tutorial, I’ll show you how to use Python to combine multiple Excel files into one master spreadsheet. Imagine that you have dozens of Excel files with the same data fields, and your job is to aggregate sheets from those files. Manually doing this job is super inefficient, and Python will save you a lot of time in the long run, so let’s all work smarter!

Note that this article talks about appending Excel files with the same format/data fields. Merging multiple dataset is a different task.

If you are new to Python, this series Integrate Python with Excel offers some tips on how to use Python to supercharge your Excel spreadsheets.

The workflow

To solve the problem, we’ll need to follow the below work flow:

1. Identify the files we need to combine
2. Get data from the file
3. Move data from step 2) to a master dataset (we will call it “dataframe”)
4. Report 2-3 for the number of files
5. Save the master dataset into an Excel spreadsheet

Import libraries

Alright, let’s see how to code the above work flow in Python. For this exercise, we’ll need to use two Python libraries: os and pandas. If you want to follow along, feel free to grab the source code and files used in this tutorial from here. Although you can combine as many Excel files as you wish, we’ll use three files to demonstrate the process.

If you need help with installing Python or libraries, here’s a guide on how to do that.

os library gives a way of using operating system dependent functionalities. Such as manipulating folder and file paths. We use this library to get all the Excel file names, including their paths.

pandas library is the gold standard for data analysis and manipulation. It is fast, powerful, and flexible. We use this library to load Excel data into Python, manipulate data, and recreate the master spreadsheet.

We’ll start by importing these two libraries. Then find the current working directory, as well as all the file names within the directory.

import os
import pandas as pd
cwd = os.path.abspath('') 
files = os.listdir(cwd) 

The variable cwd shows the path to the current working directory, and the variable files is a list of all the file names within the current working directory. Notice there are non-Excel files, and we don’t want to open those, so we’ll handle that soon.

Combine multiple Excel files into one spreadsheet

Next, we create an empty dataframe df for storing the data for master spreadsheet. We loop through all the files within the current working directory, but only process the Excel files whose name ends with “.xlsx”. This is done by this line of code
if file.endswith('.xlsx'):

pd.read_excel() will read Excel data into Python and store it as a pandas DataFrame object. Be aware that this method reads only the first tab/sheet of the Excel file by default. If your Excel file contains more than 1 sheet, continue reading to the next section.

df.append() will append/combine data from one file to another. Think about copying a block of data from one Excel file and pasting it into another. Instead of opening up Excel, data is stored inside your computer’s memory.

df = pd.DataFrame()
for file in files:
     if file.endswith('.xlsx'):
         df = df.append(pd.read_excel(file), ignore_index=True) 
df.head()

The above code does the following:

1. Loop through all the files in the current working directory, determine if a file is Excel by checking the file name ends with “.xlsx”.
2. If yes, read the file content (data), and append/add it to the master dataframe variable called df.
3. Save the master dataframe into an Excel spreadsheet.

We can examine the master dataframe by checking df.head(), which shows the first 5 rows of the data.

Seems good! Just another quick check to make sure we have loaded everything in the DataFrame. df.shape will show us the dimension (36 rows, 5 columns) of the data:

Everything looks good, so let’s output the data back into Excel. The last line df.to_excel() will do that.

Combine multiple sheets from the same Excel file

I talked about the two techniques to read multiple sheets from the same Excel file, so I won’t repeat it. However, I’ll walk through an example here with a slightly different setting.

We have 2 files each contains a number of sheets. We don’t know how many sheets are in each file, but we know the format is the same for all sheets. Our goal is to aggregate all sheets into one spreadsheet (and one file).

The workflow is similar:

1. Get all Excel files
2. Loop through the Excel files
3. For each file, loop through all sheets
4. Read each sheet into a dataframe, then combine all dataframes together.

df_total = pd.DataFrame()
for file in files:  # loop through Excel files
    if file.endswith('.xlsx'):
        excel_file = pd.ExcelFile(file)
        sheets = excel_file.sheet_names
        for sheet in sheets: # loop through sheets inside an Excel file
            df = excel_file.parse(sheet_name = sheet)
            df_total = df_total.append(df)
df_total.to_excel('combined_file.xlsx')

Putting it all together

Below is the full code put together. 10 lines of code will help you combine all your Excel files or sheets into one master spreadsheet. Enjoy!

import os
import pandas as pd
cwd = os.path.abspath('') 
files = os.listdir(cwd)  

## Method 1 gets the first sheet of a given file
df = pd.DataFrame()
for file in files:
    if file.endswith('.xlsx'):
        df = df.append(pd.read_excel(file), ignore_index=True) 
df.head() 
df.to_excel('total_sales.xlsx')



## Method 2 gets all sheets of a given file
df_total = pd.DataFrame()
for file in files:                         # loop through Excel files
    if file.endswith('.xlsx'):
        excel_file = pd.ExcelFile(file)
        sheets = excel_file.sheet_names
        for sheet in sheets:               # loop through sheets inside an Excel file
            df = excel_file.parse(sheet_name = sheet)
            df_total = df_total.append(df)
df_total.to_excel('combined_file.xlsx')