SQL Server Video Automating Flat File Imports using PowerShell

By: Tim Smith | Updated: 2014-06-18 | Comments (6) | Related: > PowerShell

Problem / Solution

SQL Server DBAs and Developers often get flat files that they need to import into SQL Server for queries and data manipulation. You could easily use SSIS to do imports or use the import wizard within SSMS, but this takes time to setup and if you have multiple files it can become a time consuming task. In this video tip, Tim Smith shows how you can use PowerShell to dynamically read the flat file, dynamically create a SQL Server table and then import the data into SQL Server with an easy to use PowerShell function. With this function you can pass parameters to the function and use this to automate the creation and import of flat files. Take a look at this video tip and also read the companion tip to get a step by step explanation and access to the code.

Key Learning Items

Using PowerShell variables
Using SQLConnection from PowerShell
Using SQLCommand from PowerShell
Running SQL queries and stored procedures from PowerShell
Dynamically building a table and inserting data into SQL Server from PowerShell
Creating a PowerShell function

Read Companion Tip

Click Here

About the author

Tim Smith works as a DBA and developer and also teaches Automating ETL on Udemy.

This author pledges the content of this article is based on professional experience and not AI generated.

View all my tips

Article Last Updated: 2014-06-18

Comments For This Article

Saturday, April 25, 2020 - 2:55:34 AM - mike

Hi terrific script thanks.

I was hoping that this would allow me to import my CSV.

My data includes quote qualified text fields where those fields include carriage returns and commas.

Incredibly there does not appear to be a true CSV import format that can be used with BULK. That being the case BULK determines a new row or new field midway through my " qualified text field.

Monday, May 7, 2018 - 6:32:54 PM - Ron Abbott

Great topic

I got the code working fine the other day, but then tried to add code to drop the table first so that I can recreate it each time. Now the code doesnt work anymore!. I have deleted the extra code I added, going back to what I thought had worked.

Here is my code with the errors (running from Powershell_ise

PS C:\Windows\system32> Function CreateRestOwners($location, $file, $extension)

{

$full = $location + $file + $extension

$all = Get-Content $full

$columns = Get-Content $full | Select -First 1

$columns = $columns.Replace(" ","")

$columns = $columns.Replace("`t","] VARCHAR(200), [")

$table = "CREATE TABLE " + $file + "([" + $columns + "] VARCHAR(100))

$connection = New-Object System.Data.SqlClient.SqlConnection

$buildTable = New-Object System.Data.SqlClient.SqlCommand

$insertData = New-Object System.Data.SqlClient.SqlCommand

$connection.ConnectionString = "Data Source=LPCONVVM-ADMIN\SQLDEVELOPMENT;Database=Rest01;integrated security=true"

$buildTable.CommandText = $table

$buildTable.Connection = $connection

$x = 0

$insertData.CommandText = "EXECUTE stp_CommaBulkInsert @1,@2"

$insertData.Parameters.Add("@1", $full)

$insertData.Parameters.Add("@2", $file)

$insertData.Connection = $connection

$connection.Open()

$buildTable.ExecuteNonQuery()

$connection.Close()

$buildTable.CommandText = $table

$x = 1

if ($x = 1)

{

$connection.Open()

$insertData.ExecuteNonQuery()

$connection.Close()

}

Unexpected token 'data' in expression or statement.

At line:12 char:33

Unexpected token 'Source=LPCONVVM-ADMIN\SQLDEVELOPMENT' in expression or statement.

At line:12 char:38

Unrecognized token in source text.

At line:16 char:59

Missing expression after ',' in pipeline element.

At line:16 char:58

Unexpected token '2' in expression or statement.

At line:16 char:59

Unexpected token '

$insertData.Parameters.Add(' in expression or statement.

At line:16 char:61

Unexpected token '1' in expression or statement.

At line:17 char:29

Unexpected token ', $full)

$insertData.Parameters.Add(' in expression or statement.

At line:17 char:31

Unexpected token '2' in expression or statement.

At line:18 char:29

The string starting:

At line:18 char:31

+ $insertData.Parameters.Add("@2 <<<< ", $file)

is missing the terminator: ".

At line:32 char:1

Missing closing '}' in statement block.

At line:32 char:1

Missing function body in function declaration.

At line:32 char:1

Monday, February 16, 2015 - 2:19:13 PM - Greg Robidoux

H Jack, take a look at this tip for the scripts: http://www.mssqltips.com/sqlservertip/3208/automating-flat-file-sql-server-imports-with-powershell/

Monday, February 16, 2015 - 1:59:51 PM - Jack Owens

Great demo

Could you please let us know where can I download your demo ".ps1" file from?

Is your file shared by any chance?

Wednesday, February 11, 2015 - 5:25:08 PM - JeremyH	Back To Top (36210)
Nice little video Tim. About 4.5 minutes in, where you mention how SQL Server doesn't recognize the new tables as existing yet (because the session was opened before the tables were created), you can refresh that session by pressing ctrl + shift + 'r'. Just a little SSMS tip.

Wednesday, January 7, 2015 - 5:32:20 PM - Ted

Great vid and info Tim! Thanks for sharing.

Ted Higgins