Optimizing Wordlists with Masks

Note: Password data mentioned in this article was obtained through public resources to improve overall password security posture. Information shared in public breaches helps improve security recommendations.

Introduction #

To crack hashes, practitioners use large wordlists containing likely password candidates. They can then use them with different attack types, such as rules, to try and recover the plaintext values.

Not surprisingly, the best wordlists come from actual passwords as the human element in setting passwords tends to permeate through to create predictable patterns that are often targeted. For example, people are far less likely to add numbers to the beginning than to the end of a password. While people still set passwords starting with digits, it is less statistically common than at the end.

In this post, I will introduce my methodology for creating new password-cracking wordlists and benchmark them against other popular ones.

Extract, Transform, and Load #

I dumped all the cracked hashes on my password archive server to get started. We will work with ~958m (million) passwords for this test. To get the best results possible, I wanted to filter out any bad patterns before getting started:

# dump passwords
$ wc -l plain-passwords.lst
958104442 plain-passwords.lst

# remove low quality items
$ cat plain-passwords.lst | grep -vE 'http:\/\/https:\/\/|\@\.com|\@\.ru|\@\.cn|\@\.org|\@.*\.net|<tr>|<div>|<a href|<p>|<img src|\$HEX\[|fbobh_|\@mail|\@msn|\@aol|\@yahoo|\@gmail|\@hotmail' | grep -v '[^[:print:]]' > prepped-passwords.lst

$ wc -l prepped-passwords.lst
924073743 prepped-passwords.lst

This dropped the total count to ~924m which is quite a lot, but because we are making wordlists, any quality filtering will go a long way.

What’s Behind The Mask? #

One of my favorite strategies for creating wordlists is to use common password masks to filter wordlists for candidates. This way we can avoid seemingly “random” passwords and keep the best quality candidates together.

To do this, I used a tool called Maskcat which can turn plaintext passwords into hashcat masks with additional metadata such as the complexity and length. Additionally, masks that occurred less than three times were removed.

# make password masks
$ cat prepped-passwords.lst | maskcat > mask-passwords.lst

$ wc -l mask-passwords.lst
920747753 mask-passwords.lst

# aggregate masks
$ cat mask-passwords.lst | sort -T ./ | uniq -c | sort -T ./ -rn > sorted-mask-passwords.lst

$ head -n 5 sorted-all-masks.txt
83399314 ?l?l?l?l?l?l?l?l:8:1
15711147 ?l?l?l?l?l?l?l?l?l?l:10:1
14774094 ?d?d?d?d?d?d?d?d:8:1
14189820 ?d?d?d?d?d?d?d?d?d?d:10:1
13469846 ?l?l?l?l?l?l?l?l?l:9:1

Doing some math on the results, if we took just the top 5,000 masks, we would cover around 86.8% of the plaintext passwords. This is exciting because we can gain additional speed and performance by leaning out the wordlist to the most likely candidates.

Next up, let us collect all of the dumped passwords and any previously made wordlists to ensure we have complete coverage:

# get all the word sources
$ wc -l wordlist1.lst
363572796 wordlist1.lst

$ wc -l wordlist2.lst
445966458 wordlist2.lst

$ wc -l dumped-passwords.lst
924073743 dumped-passwords.lst

# file sizes
$ ll | grep lst
3.8G -rwxrwxrwx 1 jw jw 3.8G Jul 11 21:01 wordlist1.lst
5.1G -rwxrwxrwx 1 jw jw 5.1G Jul 11 21:11 wordlist2.lst
9.7G -rwxrwxrwx 1 jw jw 9.7G Jul  5 21:46 dumped-passwords.lst

We need to get the top 5,000 password masks from sorted-all-masks.txt, which will cover most of the database’s passwords. We also have metadata from maskcat that we can use to make even more specific wordlists.

After removing the item count from sorted-all-masks.txt, we can use a regex to filter the masks looking for items that are greater than or equal to eight (8) characters and between three (3) and four (4) complexity.

# top 5k masks overall
$ head top-5k-masks.txt
?l?l?l?l?l?l?l?l
?l?l?l?l?l?l?l?l?l?l
?d?d?d?d?d?d?d?d
?d?d?d?d?d?d?d?d?d?d
?l?l?l?l?l?l?l?l?l
?l?l?l?l?l?l?d?d
?l?l?l?l?l?l?l?l?d?d
?l?l?l?l?l?l?l
?l?l?l?l?l?l?d?d?d?d
?l?l?l?l?d?d?d?d

# masks from top 5k that meet above requirements
$ $ cat clean-sorted-3to4-complexity-mask-passwords.lst | grep -vE ':3:3$|:4:3$|:4:4$|:5:3$|:5:4$|:6:3$|:6:4$|:7:3$|:7:4$' > clean-sorted-3to4-complexity-ge8-len-mask-passwords.lst

$ head top-5k-3to4ge8-masks.txt
?u?l?l?l?l?l?d?d
?u?l?l?l?l?l?d?d?d?d
?u?l?l?l?l?d?d?d?d
?u?l?l?l?d?d?d?d
?u?l?l?l?l?l?l?d?d
?u?l?l?l?l?l?l?l?d?d
?s?l?l?l?d?d?d?d?d?d
?u?l?l?l?l?l?d?d?d
?u?l?l?l?l?d?d?d
?u?l?l?l?l?l?l?d

Now we take the wordlists and push them through maskcat to match entries that match the most popular masks. This will help slim down wordlists to the most probable entries.

# making a wordlist
$ cat wordlist1.lst | maskcat match top-5k-masks.txt > top5kmaskswords.lst
$ cat wordlist1.lst | maskcat match top-5k-masks.txt > top5kmaskswords-2.lst
$ cat dumped-passwords.lst | maskcat match top-5k-masks.txt > top5kmaskswords-3.lst

# making a complex wordlist
$ cat wordlist1.lst | maskcat match top-5k-3to4ge8-masks.txt > top5kmaskswords-3to4ge8.lst
$ cat wordlist1.lst | maskcat match top-5k-3to4ge8-masks.txt > top5kmaskswords-3to4ge8-2.lst
$ cat dumped-passwords.lst | maskcat match top-5k-3to4ge8-masks.txt > top5kmaskswords-3to4ge8-3.lst

# sample of a list
$ head top5kmaskswords-3to4ge8-3.lst
$01april
$01august
$01August
$01autumn
$01Autumn
$01december
$01february
$01january
$01march
$01november

After getting all the results, we can combine the files and remove duplicate values. After everything finished, we took around ~16GB (size of everything, deduped) down to ~12GB which is around 75% of the original size.

# final sizes
$ wc -l top-5k-masks-3to4ge8.lst
155719008 top-5k-masks-3to4ge8.lst

$ wc -l top-5k-masks.lst
1187881713 top-5k-masks.lst

$ ll
1.7G -rwxrwxrwx 1 jw jw 1.7G Jul 13 22:06 top-5k-masks-3to4ge8.lst
12G -rwxrwxrwx 1 jw jw  12G Jul 13 23:19 top-5k-masks.lst

Now there is a choice to make. The options are:

• Leave wordlists as they are, with duplicates between each other
• De-duplicate wordlists between each other

Both have advantages. By leaving the wordlists as they are, you can be sure that you have great coverage at the risk of running duplicates. If you opt to remove duplicates, you will remove the risk of running duplicates but may miss out on coverage unless you run both.

For this test, we will remove duplicates between the two with rli.bin. We will also take all the unmatched items to their own third wordlist to preserve the data. We will take the top 5k masks and reduce them by the top 5k masks with complexity. This way, the smaller complexity list retains its size, and the larger list is reduced.

# syntax
$ rli.bin -h
usage: rli.bin infile outfile removefiles...

# if the files are too large try splitting
split -n 2 file.lst

# start by removing entries from the top 5k masks passwords
$ rli.bin top-5k-masks.lst reduced-top-5k-masks.lst top-5k-masks-3to4ge8.lst

# then remove matched entries from the unmatched entries
$ rli.bin remainder.lst reduced-remainder.lst reduced-top-5k-masks.lst top-5k-masks-3to4ge8.lst

Lets check out the sizes of the final wordlists:

$ ll
3.5G -rwxrwxrwx 1 jw jw 3.5G Jul 14 14:28 final-remainder.lst
1.7G -rwxrwxrwx 1 jw jw 1.7G Jul 13 22:06 final-top-5k-mask-3to4ge8-passwords.lst
 11G -rwxrwxrwx 1 jw jw  11G Jul 14 13:19 final-top-5k-mask-passwords.lst

$ wc -l final*
  218014370 final-remainder.lst
  155719008 final-top-5k-mask-3to4ge8-passwords.lst
 1057925799 final-top-5k-mask-passwords.lst
 1431659177 total

We removed ~3.5GB of unlikely candidates into their own wordlist and shaved off ~2.4GB in duplicate values, leaving three newly optimized wordlists.

The Results #

To best measure effectiveness, we will split the wordlists into a few different sizes using the same methods above:

To break down the differences:

• no-ending: no special filtering, just the top x masks
• c8: filtered for complexity and length greater than or equal to eight (8)
• c8l: same as c8 but everything in lowercase
• nd: same as no-ending but skipping masks that are 100% digits
• leftovers: list containing all of the non-matched items

For testing, we will be using the same  document from the Rules Article created by PenguinKeeper and others to get a standard process and benchmark. A table with complete information will be provided at the bottom of the article.

One thing I want to note here is that because we are using a massive password dump, there is a likely chance that the more extensive lists directly contain the test set in them. Take results with a grain of salt.

The following summarizes the results:

Application #

Overall, the results show this strategy is an effective way of creating new wordlists using cracked passwords. I would implement it into any process where you want to develop wordlists targeting specific password policy requirements, create optimized wordlists, and create wordlists to approach password hash lists with no known plains.

Appendix: Full Results Table #