Month of PowerShell: Merging Two Files (Understanding ForEach)

#monthofpowershell

Let's say we have two files: firstnames.txt and lastnames.txt. You need to merge the two files together to create a list of email addresses for a target organization as part of a password spray attack. Maybe you want to use MSOLSpray (orig) and FireProx to discover login names.

This is a task for the foreach statement (or so I thought).

Let's look at the files. I've created firstnames.txt and lastnames.txt from the Name Census website data (first names, last names):

PS C:\temp> Get-Content .\firstnames.txt
andrea
barbara
bari
biddy
david
edward
elizabeth
heather
james
jennifer
john
jon
latisha
linda
maddie
mary
michael
patricia
robert
william
PS C:\temp> Get-Content .\lastnames.txt
allen
brown
davis
garcia
gray
harris
hernandez
johnson
jones
keely
kembrey
lopez
lulham
marfell
martinez
merckle
miller
rodriguez
smith
williams

Next, I'll read the files into variables $firstnames and $lastnames:

PS C:\temp> $firstnames = Get-Content .\firstnames.txt
PS C:\temp> $lastnames = Get-Content .\lastnames.txt
PS C:\temp>

Using two foreach loops, we can iterate on the $firstnames and $lastnames lists, creating the variables $first and $last for each name in both loops. Then we use Write-Host with variable expansion to produce the email address First.Last@falsimentis.com:

PS C:\temp> foreach ($first in $firstnames) { foreach ($last in $lastnames) { Write-Host "$first.$last@falsimentis.com" } }
andrea.allen@falsimentis.com
andrea.brown@falsimentis.com
andrea.davis@falsimentis.com
andrea.garcia@falsimentis.com
andrea.gray@falsimentis.com
andrea.harris@falsimentis.com
andrea.hernandez@falsimentis.com
andrea.johnson@falsimentis.com
andrea.jones@falsimentis.com
andrea.keely@falsimentis.com
andrea.kembrey@falsimentis.com
andrea.lopez@falsimentis.com
andrea.lulham@falsimentis.com
andrea.marfell@falsimentis.com
andrea.martinez@falsimentis.com
andrea.merckle@falsimentis.com
andrea.miller@falsimentis.com
andrea.rodriguez@falsimentis.com
andrea.smith@falsimentis.com
andrea.williams@falsimentis.com
barbara.allen@falsimentis.com
barbara.brown@falsimentis.com
barbara.davis@falsimentis.com
...

Fantastic! Intuitive, straightforward code to solve a problem. Problem solving with PowerShell! Success!

Except for one last part: this writes the output to the screen, but that's not what we want. We want the output to go to a file. So, let's add to the pipeline with Out-File to save the email addresses to a file.

Here's where it all started to go downhill.

This doesn't work:

PS C:\temp> foreach ($first in $firstnames) { foreach ($last in $lastnames) { Write-Host "$first.$last@falsimentis.com" } } | Out-File "falsimentis-email-guesses.txt"
At line:1 char:113
+ ... $lastnames) { Write-Host "$first.$last@falsimentis.com" } } | Out-Fil ...
+                                                                 ~
An empty pipe element is not allowed.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : EmptyPipeElement

The problem is this: ForEach doesn't output to the pipeline when it is used as a statement. But, ForEach does output to the pipeline when it is used as an alias for the ForEach-Object cmdlet.

If you're confused about that, you're not alone.

Let's take a look at how this PowerShell loop construct can have significantly different use cases. First, foreach is a PowerShell statement, meaning it can be used like this:

foreach ($currentobject in $listofobjects) {
    Write-Host $currentobject
}

In this use case, foreach does not output to the pipeline, preventing us from using it with later cmdlets, like Out-File. Here, foreach is a PowerShell statement; it is not the ForEach alias for ForEach-Object. Using ForEach-Object or the ForEach alias would require an object to precede it, like this:

$listofobjects | ForEach { 
    Write-Host $_
}

Here's the bottom line: When you use foreach as a statement, you can't leverage the output in the pipeline normally; when you use ForEach as an alias for ForEach-Object in the pipeline, you can.

Confounded by this, I asked some colleagues to come up with solutions to the challenge of merging two files and writing the output to a new file.

Solution 1: Mixed ForEach

This is the solution from the amazing Jon Gorenflo:

PS C:\temp> $firstnames = Get-Content .\firstnames.txt
PS C:\temp> $lastnames = Get-Content .\lastnames.txt
PS C:\temp> $firstnames | ForEach { foreach ($lastname in $lastnames) { "$_.$lastname@falsimentis.com" } } | Out-File "falsimentis-email-guesses.txt"
PS C:\temp> gci .\falsimentis-email-guesses.txt


    Directory: C:\temp


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         6/21/2022  11:28 AM          24962 falsimentis-email-guesses.txt


PS C:\temp> Get-Content .\falsimentis-email-guesses.txt | Select-Object -First 3
andrea.allen@falsimentis.com
andrea.brown@falsimentis.com
andrea.davis@falsimentis.com

Jon's solution is interesting because it uses a mix of ForEach as an alias for the ForEach-Object cmdlet and foreach as a statement (respectively; I've capitalized them to match in the preceding example). From what I understand, because the pipeline is created using $firstnames | ForEach, then subsequent foreach statements will continue to output to the pipeline.

Wired	Tired
It's ninja to use both variations of ForEach in one command.	It's syntactically awkward to use different syntax for ForEach. Using a mix of $_ and $lastname variables seems inconsistent.

Solution 2: Mixed ForEach

This is the solution from my #MOPS collaborator Mick Douglas:

PS C:\temp> $firstnames = Get-Content .\firstnames.txt
PS C:\temp> $lastnames = Get-Content .\lastnames.txt
PS C:\temp> foreach ($first in $firstnames) { foreach ($last in $lastnames) { "$first.$last@falsimentis.com" | Add-Content -Path ".\falsimentis-email-guesses.txt" } }
PS C:\temp> Get-Content .\falsimentis-email-guesses.txt | Select-Object -First 3
andrea.allen@falsimentis.com
andrea.brown@falsimentis.com
andrea.davis@falsimentis.com

NOTE: I've changed Mick's original solution to use variable expansion in the string for consistency with the other solutions; Mick originally wrote this using string concatenation.

Mick's solution uses the foreach statement consistently for both loops. If we were to add the Out-File pipeline at the end of the 2nd block, we'd get the An empty pipe element is not allowed error message. However, Mick uses the pipeline within the 2nd foreach block, adding one email address at a time using Add-Content. This works because we're not trying to pipeline the foreach statement, we're just using the pipeline for the string variable expansion for "$first.$last@falsimentis.com" and multiple Add-Content file writes.

Wired	Tired
Consistent use of foreach. Intuitive, and easy to read.	Each loop iteration writes to the file using Add-Content; this is OK for small files, but inefficient for large data sets since it is a lot of small writes instead of a small number of big writes.

Solution 3: Subexpressions

This is the solution I came up with:

PS C:\temp> $firstnames = Get-Content .\firstnames.txt
PS C:\temp> $lastnames = Get-Content .\lastnames.txt
PS C:\temp> $(foreach ($first in $firstnames) { foreach ($last in $lastnames) { "$first.$last@falsimentis.com" } }) | Out-File "falsimentis-email-guesses.txt"
PS C:\temp> Get-Content .\falsimentis-email-guesses.txt | Select-Object -First 3
andrea.allen@falsimentis.com
andrea.brown@falsimentis.com
andrea.davis@falsimentis.com

My initial attempt looks similar, where the email addresses are formed and we use the pipeline to write the results to a file. Here though, the foreach loops are embedded in a subexpression using PowerShell's $() operator.

By enclosing the original PowerShell in the subexpression operator, we get an object of type System.Array as the output:

PS C:\temp> $(foreach ($first in $firstnames) { foreach ($last in $lastnames) { "$first.$last@falsimentis.com" } }).GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Object[]                                 System.Array

The array is a collection of strings for each email address generated in the 2nd foreach block. Since it's an array, we can use the pipeline and Out-File to write it to a file.

Wired	Tired
Uses the pipeline in a consistent manner with common pipeline use. Reduces the number of write operations (faster).	The closing }}) syntax makes me wonder if I should just write Async JavaScript and get it over with.

Closing Thoughts

PowerShell's immediate differentiator from other scripting languages is the power of the pipeline. However, this isn't always as consistently accessible as we might like, and it requires a deeper understanding of PowerShell functionality to leverage well. Knowing the differences between the foreach statement and the ForEach alias for ForEach-Object is important, and helps us to know why a simple statement doesn't work the way you might expect.

Special thanks to Kirk Munro for his articles Essential PowerShell: Understanding foreach and Essential PowerShell: Understanding foreach (Addendum) that were instrumental in helping me understand the nuances of foreach.

-Joshua Wright

Return to Getting Started With PowerShell

---

Joshua Wright is the author of SANS SEC504: Hacker Tools, Techniques, and Incident Handling, a faculty fellow for the SANS Institute, and a senior technical director at Counter Hack.