Get our FREE VBA eBook of the 30 most useful Excel VBA macros.

Automate Excel so that you can save time and stop doing the jobs a trained monkey could do.

Claim your free eBook


Power Query: How to Combine files based on a list

Power Query has a really powerful feature that lets us combine all the files in a folder. However, sometimes the files are not contained in a single folder. Sometimes they are spread out in multiple places. How can we deal with that? That’s what we are looking at in this post: how to combine files based on a list.

Download the example file: Click the link below to download the example file used for this post:

Watch the video

Watch the video on YouTube

Source Data Table

First, let’s start by making a Table of all the files to combine, including the sheets or tables in those workbooks we want to use.

Source Files list for Power Query to connect to

In this example, I will use worksheets, but you could use Tables or named ranges.

You will notice the worksheets in each workbook have different names; this is often a tricky Power Query situation, but this is no problem for this method.

Load the Table, which for the example is called SourceFiles, into Power Query.

Source Files List in Power Query

Connect to and transform a single workbook

We now need to get the code to connect to each workbook. I’m assuming you’re not an M-code expert, so we will work through this without writing any code.

In Power Query, right-click on the Queries pane and click New Queries > From File > From Workbook.

Connect to a new workbook

If your file is a different file type, use the relevant connector for that file.

Navigate to one of the files in the list and click Import.

In the Navigator window, select the relevant source data, then click OK.

Power Query Navigator window select file then OK

Next, we need to make any necessary transformations for that single workbook.

For the example:

  • Removed the top 4 rows
  • Promoted the headers
  • Changed the data type
  • Filtered to exclude null values from the Account Code column

Here is the final table for a single workbook of the example data.

Example changes in single workbook

Reuse the transformation steps

In the single workbook query, click View > Advanced Editor.

View - Advanced Editor

Copy all the code, then click Done to close the window.

Copy the Power Query M code

Go back to the query which lists all the files.

Click Add Column > Custom Column

Add column - custom column

In the Custom Column dialog box, give the column a name (I’ve used Data) and paste the code we copied earlier into the Custom Column formula box.

Paste the code into Advanced Editor

We just need to make a few small adjustments to the code.

  • Where the file path is hardcoded, change it for the column name containing the file path.
  • Where the sheet name is hard coded, change it for the column name containing the sheet name.

For our example. The code changes from this:

Source = Excel.Workbook(File.Contents("C:\Examples\Data\2022\P12\P12 2022 Trial Balance.xlsx"), null, true),
Dec_Sheet = Source{[Item="Dec",Kind="Sheet"]}[Data],[insert code]

To this:

Source = Excel.Workbook(File.Contents([File Path]), null, true),
Dec_Sheet = Source{[Item=[Sheet Name],Kind="Sheet"]}[Data],

Click OK to close the Custom Column dialog box.

The Power Query preview window now looks like this:

New Data column with all source files - expand to combine files

Expand the data to combine files

For the sake of simplicity, we will expand the columns using the Expand icon. If you could have new data at any point, you should expand the columns dynamically. There is a blog post about that method here: How to expand columns dynamically in Power Query

Click the expand icon. We don’t want the original column name as prefix, so uncheck that, then click OK.

Expand the data

BOOM!

That’s it. You’re done!

We can now delete the single workbook query, as it serves no purpose.

If we add more rows to the SourceFiles Table, the query will incorporate those files too.

Making additional transformations later

At some point, the source data may change, and you may want to change the query. So, what can we do?

  • Create a new blank query – right-click on the Queries pane, click New Query > Other Sources > Blank Query.
  • Copy and paste the code from the Custom column step into the new query.
  • Change the [File Path] and [Sheet Name] columns back to hard-coded values which reference a specific file.
  • This changes the new query back to a standard query.
  • Make any amendments you wish to that query.
  • Finally, Copy and paste the code back into the Custom Column.

Conclusion

We’ve seen that we can easily combine all the files in a list using Power Query. It takes a few minor amendments to the automatically generated code, but nothing we can’t handle. Now, it doesn’t matter where the files are, or what the sheets are called; we can easily combine them into a single data source.

Related Posts:


Headshot Round

About the author

Hey, I’m Mark, and I run Excel Off The Grid.

My parents tell me that at the age of 7 I declared I was going to become a qualified accountant. I was either psychic or had no imagination, as that is exactly what happened. However, it wasn't until I was 35 that my journey really began.

In 2015, I started a new job, for which I was regularly working after 10pm. As a result, I rarely saw my children during the week. So, I started searching for the secrets to automating Excel. I discovered that by building a small number of simple tools, I could combine them together in different ways to automate nearly all my regular tasks. This meant I could work less hours (and I got pay raises!). Today, I teach these techniques to other professionals in our training program so they too can spend less time at work (and more time with their children and doing the things they love).


Do you need help adapting this post to your needs?

I'm guessing the examples in this post don't exactly match your situation. We all use Excel differently, so it's impossible to write a post that will meet everybody's needs. By taking the time to understand the techniques and principles in this post (and elsewhere on this site), you should be able to adapt it to your needs.

But, if you're still struggling you should:

  1. Read other blogs, or watch YouTube videos on the same topic. You will benefit much more by discovering your own solutions.
  2. Ask the 'Excel Ninja' in your office. It's amazing what things other people know.
  3. Ask a question in a forum like Mr Excel, or the Microsoft Answers Community. Remember, the people on these forums are generally giving their time for free. So take care to craft your question, make sure it's clear and concise.  List all the things you've tried, and provide screenshots, code segments and example workbooks.
  4. Use Excel Rescue, who are my consultancy partner. They help by providing solutions to smaller Excel problems.

What next?
Don't go yet, there is plenty more to learn on Excel Off The Grid.  Check out the latest posts:

4 thoughts on “Power Query: How to Combine files based on a list

  1. Peter Gale says:

    It’s very inventive of you to come up with this solution, but wouldn’t it be better to create a function for the transformation process?

    • Excel Off The Grid says:

      Yes, I could have done. But I don’t think it’s a matter of better or worse.

      Creating a custom function provides a mechanism for passing the variables into the let/in statement. It would still be the same underlying M inside the let/in. Therefore, it would have make little to no difference, apart from how the variables are managed.

Leave a Reply

Your email address will not be published. Required fields are marked *