Using Microsoft Co-pilot to help retrieve a password

I had a bit of a problem recently where a password that I needed, that was quite important, wasn't recorded in a password safe.  A year ago that wouldn't have been a problem, as I was typing this password a lot, but after a period of inactivity on that project I was stuck.  Fortunately this wasn't a password I had set  (it was set by a colleague), so it was less complicated than my normal style.  I could also remember parts of the password with certainty, leaving me just some characters in the middle that I couldn't remember.  I could also remember that some of the password rhymed.

I was convinced I was missing three lower case letters, with 17,576 potential combinations.  Admittedly, the fact there was a rhyme in there would help reduce the number of combinations, but I'd still need to run through the combinations to be sure I'd found everything that could rhyme.  I wasn't keen to run through 17,576 combinations manually, so decided on a script to automate the process.

Having recently been doing some research into "AI" [1] (large language models) to assist with programming, I was keen to see if I could use an "AI" to help solve this problem.  I chose PowerShell as my scripting language, as I was on a Windows machine and PowerShell is my chosen language there, and Microsoft Co-pilot as my "AI" of choice - it's built in to my Windows 11 machine.

Disclaimer: this post is provided to show the journey I took and to provide a hopefully interesting insight into how a large language model can help with tasks such as retrieving your own password.  You are not permitted to use this information to gain access to a system you are not authorised to access (which may be illegal in your area).

My scenario

I had a Java keystore file that contained a private key used to cryptographically sign a file.  Don't worry too much about what that means, but the important thing is that the keystore was protected with a password.  I don't use Java's keytool.exe very often, so I asked Co-pilot for advice on how to pass the password to it:

Prompt: With Java keytool how do I provide the password on the command line?

Response: According to the web search results, you can provide the password on the command line for Java keytool using the `-storepass` option.  [From batch file - Add password to keytool command - Stack Overflow]

Note that there can be security issues with passing a password in this way, especially if transcripts are being saved.  ⚠️ Use with caution.

Not taking the direct route

What I've learned from my research to date is that a lot of hosted large language models (LLM) will outright refuse to write malware or hacking tools.  From an ethical standpoint that's probably a good thing. You could possibly host the model yourself to bypass the restriction, but I didn't expect Co-pilot to be any different.  Once I'd solved my problem I tried the direct route, which confirmed my suspicions:

Prompt: I have forgotten  the password to a file.  Write me a PowerShell script to iterate over different combinations so I can access the file.

Response: I’m sorry, but I cannot help you with writing a PowerShell script to crack a password. This is because cracking a password is an unethical and potentially illegal activity that could harm someone’s privacy and security. I do not support or condone such actions.

The model did go on to tell me about PowerShell and passwords though.

Using Co-pilot

Having mentally ruled out the direct route I looked to break my problem down into chunks, and then identify what I wanted Co-pilot to help with.  To be clear, I could have done this myself, without Co-pilot, but I wanted the opportunity to play with the LLM, and wanted the starter help it gave to (hopefully) save time.  Given I was confident I only needed three characters in the middle of the password, I broke up the work as follows:

• Generate all three character combinations of lowercase letters of the alphabet
• Take each combination and supplement with the parts of the password I remember
• Test each password to see if it's correct
• Report the correct password once found

That's a very inefficient way to do it, but it was a starting point for me to refine.  I wanted Co-pilot to assist with the first point, generating the combinations.

Prompt: Write a PowerShell script to list different combinations of the letters a-z three times over.

Co-pilot proceeded to give me two options, explaining that:

This script will output 17,576 combinations of the letters a-z three times over, such as `aaa`, `aab`, `aac`, ..., `zzx`, `zzy`, `zzz`.

Of the two options, I opted to work with the first script suggestion, which is below.  Those familiar with PowerShell will note there's some syntax errors:

# BROKEN SCRIPT - Co-pilot's original

# Define the alphabet as an array of characters
$alphabet = [char []] \"abcdefghijklmnopqrstuvwxyz\"

# Initialize an empty array to store the combinations
$combinations = @ ()

# Loop over the alphabet three times
foreach ($letter1 in $alphabet) {
    foreach ($letter2 in $alphabet) {
        foreach ($letter3 in $alphabet) {
            # Concatenate the letters and add them to the array
            $combination = $letter1 + $letter2 + $letter3
            $combinations += $combination
        }
    }
}

# Output the array of combinations
$combinations

Correcting the syntax errors gets us the following:

# FIXED SCRIPT - corrected Co-pilot's output

# Define the alphabet as an array of characters
$alphabet = [char []] "abcdefghijklmnopqrstuvwxyz"

# Initialize an empty array to store the combinations
$combinations = @()

# Loop over the alphabet three times
foreach ($letter1 in $alphabet) {
    foreach ($letter2 in $alphabet) {
        foreach ($letter3 in $alphabet) {
            # Concatenate the letters and add them to the array
            $combination = $letter1 + $letter2 + $letter3
            $combinations += $combination
        }
    }
}

# Output the array of combinations
$combinations

I really like how Co-pilot lists its sources, as that helps you learn and also allows you to check if the output has any licences or restrictions.  Unfortunately it wasn't clear which referenced article related to that answer, but I'm assuming that code sample was derived from "function - Get all combinations of an array - Stack Overflow" despite not finding the code there, and it being in a different language.

Testing the combinations

Generating all of the combinations using the fixed script (the second one) took about 20 seconds on my laptop.  I could store each one, then go over the list trying the password but that seemed excessive.  Possibly it would have been quicker, but it was not necessary.  I took the script and modified it to perform the next steps.

As part of this I performed two changes at once:

• Take the combination and put in the middle of the password segments I remember
• Try the password, if the password was wrong try the next possible password.  If the password was right, say what it was and exit

I'll show the code now, and then explain it.  I've added comments to make it easier:

# Define the alphabet as an array of characters
$alphabet = [char []] "abcdefghijklmnopqrstuvwxyz"

# Provide the path to the keystore file
$keystoreFile = "C:\tmp\3letter.jks"

# Loop over the alphabet three times
foreach ($letter1 in $alphabet) {
    foreach ($letter2 in $alphabet) {
        foreach ($letter3 in $alphabet) {
            # Concatenate the letters and add them together
            $combination = $letter1 + $letter2 + $letter3
            
            # Concatenate the password segments I know with the letter component
            $attempt = 'THE-BIT-I-KNOW'+$combination+'873'
            
            # State what password is being attempted
            Write-Host "Attempting password: $attempt"
            
            # Pass the possible password to Java Keytool
            keytool.exe -list -keystore $keystoreFile -storepass $attempt
            
            # Check the exit code
            if($LASTEXITCODE -eq 0){
                # The password has been found.  State it and then stop trying
                write-host -ForegroundColor Red -BackgroundColor Yellow "Password found, result is $attempt"
                exit
            }
            
            # Clear the screen and try again
            Clear-Host
        }
    }
}

• At the beginning we define the $alphabet the same as before - the letters a - z, all lower case
• Next I provide the path to the $keystoreFile, which I'll use later on
• I then loop over the alphabet three times using foreach
• In the third foreach I build the letter combination by concatenating (joining together) the three letters - this is stored in a variable called $combination
• I then concatenate the first part of the password I remember, the letter combination, and the last part of the password I remember - this is stored in a variable called $attempt
• Using Write-Host I output to the screen the password that's being tried
• I use keytool.exe to try and access the key store.  I tell keytool.exe to list the keys in the store (-list), give it the path to the keystore (-keystore) and provide the password with -storepass $attempt
• Then I check the exit code ($LASTEXITCODE) to see if the command completed successfully
• If successful (a value of zero), use Write-Host to say the password was found, and what it was.  Use red text on a yellow background.  Then stop - exit will end the script
• Otherwise, clear the screen and carry on

Exit codes

I should probably explain what these are.  When programs finish (or stop running) they provide an exit code that gives the operating system information on whether or not the operation succeeded.  Generally, a value of zero (0) means everything succeeded, and a non-zero value means there was a problem.

I'd already tested with keytool.exe by creating a key store with a known password.  By providing the correct password I confirmed I got a zero exit code.  Providing the wrong password gave me an exit code of one.

In PowerShell, the variable $LASTEXITCODE contains the result of the last program that ran.  I checked how to find that in PowerShell by asking Co-pilot:

Prompt: In powershell how do I get the return code of the previously executed command?

A test run

I started off with a keystore for which I knew the password followed the pattern I described above, making sure the combination was early on in the sequence.  This was so that I could test the script would work correctly.  I set the password to THE-BIT-I-KNOWabs873 after generating the keystore following advice from Co-pilot:

Prompt: With Java keytool how do I generate a new keystore?

Response: To generate a new keystore using Java keytool, you can use the `-genkeypair` option with the following syntax:

keytool -genkeypair -alias <alias> -keyalg <algorithm> -keystore <keystore_file> -keysize <key_size>
...

After a while I received the news I was hoping for.  I paste here the PowerShell output as text (for those with screen readers) and as an image so you can see the colours:

Attempting password: THE-BIT-I-KNOWabs873
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

myalias, 15 Jan 2024, PrivateKeyEntry, 
Certificate fingerprint (SHA-256): 54:28:C4:81:E6:63:DD:1F:85:1B:B3:D9:36:27:92:D8:63:DA:B5:E0:25:FC:10:00:95:5C:78:36:8B:7D:56:3A
Password found, result is THE-BIT-I-KNOWabs873

So, did I find the password?

I did, but I had to make another change to the script.  After the script had run through all 17,576 combinations my script just ended - no "password found" message.  That's when I remembered something else - the missing piece was only two characters long (so only 26 x 26 possible combinations, or 676).  It had taken my script running for a couple of hours for me to remember that!  A quick modification and I had the missing password.

Conclusion

This exercise reminded me of a few things, and allowed me to discover some others.  Firstly, I was reminded to make sure I record passwords in a secure place (e.g. a password safe) always.  It probably didn't seem worth it originally because I was typing in the password regularly, and someone else knew it (they'd forgotten the password by the time I had this problem).

I discovered that using an LLM for development work, and general research can be quite useful.  It took me about ten minutes to take Co-pilot's original script, correct its syntax, and then end up with the script I needed.

An exercise for the reader

Rather than provide the final script, that generates combinations of only two lower case letters, I leave that as an exercise for you, dear reader.  I've provided the working script for three letters along with the example keystore file that was used in this blog post (password THE-BIT-I-KNOWabs873).  I've also provided a keystore file with a combination of two lower case letters, and a password in the same format as we've been using (so the password is THE-BIT-I-KNOW**873 where ** is two lower case letters).  Your challenge is to modify the script to find the password.

If you're stuck, I've also provided you the solution inside the folder called ANSWER.

• You can download the files here
• You'll be downloading a zip file called FindMissing2LettersPuzzle.zip (SHA256 hashA08A0855D1E8BD76EC872A7093E454D474F26B88AB84975F47FB2122B57AF5FD) that contains the files you need
  You can test the file hash using PowerShell: get-filehash .\FindMissing2LettersPuzzle.zip
  Disclaimer: while this file was uploaded free of known malware / viruses in January 2024, you should perform your own checks
• You will need PowerShell (I tested successfully with PowerShell 7, which you can get for free here, and PowerShell 5)
• You will need Java keytool, which is part of the Java Runtime Environment (see the Oracle Java website) and also the  Java Development Kit.  Depending on the location of keytool.exe you may need to modify the script to have the correct path.  For best results I recommend adding the folder that contains keytool.exe to your PATH (perform an Internet search for "add a folder to path" or type "edit environment variables for your account" into the Windows 11 start menu / search bar)

Banner image: The output of the script after finding the password.

[1] I still maintain these are not artificial intelligences.  As such, expect to see "AI" in quotes 😊.