Password auditing with mixed AMD and Nvidia GPUs
The time has arrived to upgrade the GPU of your desktop PC. The price, performance and other features of the latest generation of AMD GPUs caught your interest, so you decided to take the leap and join Team Red. What to do now with your old Nvidia GPU? Do you sell it or give it away to a relative? There's a third option. Keep it and run both GPUs in parallel! If you have a motherboard and CPU powerful enough, and also a good PSU, you can take advantage of both GPUs to perform password audits during incident response engagements. I couldn't find much information about leveraging mixed GPU types to perform such assessments, so I decided to put it to the test.
During an incident response, it may be necessary to audit passwords to determine if any passwords have been compromised or weak passwords are in use. This can be done in a number of ways, including:
Checking for the use of default or easily guessable passwords: It is important to ensure that all accounts are using strong, unique passwords that are not based on easily guessable information (e.g., "password123").
Checking password strength: Weak passwords can be easily cracked using brute-force attacks or dictionary attacks. It is important to ensure that all passwords are strong and not susceptible to these types of attacks.
Auditing password reuse: It is important to ensure that users are not reusing the same password for multiple accounts, as this increases the risk of compromise if one password is discovered.
Checking for password expiration policies: It is a good security practice to set password expiration policies to ensure that passwords are regularly changed and that users are not using the same password for extended periods of time.
Auditing password storage: It is important to ensure that passwords are stored securely, such as by using hashing and salting techniques.
No matter what your scenario, having more GPU power is always better. You need to crunch through as many passwords hashes per second as possible. This is essential to speed up your assessment, and be able to provide a quick feedback into your IR report.
My system configuration for this test
Motherboard: MSI MPG Z690 Carbon WIFI
CPU: 12th Gen Intel Core i9-12900KS
RAM: 64GB DDR5
GPU1: NVIDIA GeForce RTX 2070 SUPER
GPU2: AMD Radeon RX 7900 XT
OS: Windows 10 Pro
PSU: Corsair HX1500i (1500W)
No overclocking was applied during this test to any of the GPUs. It is important to keep in mind that running 2 GPUs in parallel has a significant impact in terms of power consumption. A powerful PSU is required for such configuration.
Password auditing with hashcat
Before you can get started with hashcat, you need to download the CUDA Toolkit drivers from Nvidia's developer website. My AMD GPU was nicely detected by hashcat, so there was no need to download additional drivers. I had the standard Adrenaline Edition, Version 22.12.1 at the time of this test.
You can then download the latest hashcat from its GitHub repository.
I first executed hashcat's benchmark, to get a quick view on the performance of the tool leveraging both GPUs. For this I used the command below:
PS> .\hashcat.exe -b The tool first reports how many GPUs were detected using the CUDA and OpenCL drivers.
The tool then starts executing each test for the different types of password hashes that it supports. Below you can find the specific results for my system configuration.
-------------------
* Hash-Mode 0 (MD5)
-------------------
Speed.#1.........: 36917.1 MH/s (35.36ms) @ Accel:2048 Loops:512 Thr:32 Vec:8
Speed.#2.........: 59592.9 MH/s (22.97ms) @ Accel:1024 Loops:1024 Thr:32 Vec:1
Speed.#*.........: 96510.1 MH/s
----------------------
* Hash-Mode 100 (SHA1)
----------------------
Speed.#1.........: 11600.9 MH/s (57.01ms) @ Accel:64 Loops:1024 Thr:256 Vec:1
Speed.#2.........: 26161.5 MH/s (53.02ms) @ Accel:128 Loops:1024 Thr:256 Vec:1
Speed.#*.........: 37762.4 MH/s
---------------------------
* Hash-Mode 1400 (SHA2-256)
---------------------------
Speed.#1.........: 5059.4 MH/s (65.46ms) @ Accel:16 Loops:1024 Thr:512 Vec:1
Speed.#2.........: 11103.2 MH/s (62.47ms) @ Accel:128 Loops:512 Thr:256 Vec:1
Speed.#*.........: 16162.6 MH/s
---------------------------
* Hash-Mode 1700 (SHA2-512)
---------------------------
Speed.#1.........: 1673.7 MH/s (49.30ms) @ Accel:8 Loops:1024 Thr:256 Vec:1
Speed.#2.........: 2937.2 MH/s (59.08ms) @ Accel:256 Loops:256 Thr:64 Vec:1
Speed.#*.........: 4610.9 MH/s
-------------------------------------------------------------
* Hash-Mode 22000 (WPA-PBKDF2-PMKID+EAPOL) [Iterations: 4095]
-------------------------------------------------------------
Speed.#1.........: 583.8 kH/s (68.37ms) @ Accel:32 Loops:256 Thr:512 Vec:1
Speed.#2.........: 1252.6 kH/s (67.61ms) @ Accel:128 Loops:512 Thr:128 Vec:1
Speed.#*.........: 1836.4 kH/s
-----------------------
* Hash-Mode 1000 (NTLM)
-----------------------
Speed.#1.........: 66774.1 MH/s (38.58ms) @ Accel:256 Loops:1024 Thr:256 Vec:8
Speed.#2.........: 94695.5 MH/s (14.08ms) @ Accel:1024 Loops:1024 Thr:32 Vec:1
Speed.#*.........: 161.5 GH/s
---------------------
* Hash-Mode 3000 (LM)
---------------------
Speed.#1.........: 34008.9 MH/s (38.52ms) @ Accel:256 Loops:1024 Thr:128 Vec:1
Speed.#2.........: 63591.6 MH/s (21.00ms) @ Accel:1024 Loops:1024 Thr:32 Vec:1
Speed.#*.........: 97600.5 MH/s
--------------------------------------------
* Hash-Mode 5500 (NetNTLMv1 / NetNTLMv1+ESS)
--------------------------------------------
Speed.#1.........: 35327.5 MH/s (73.11ms) @ Accel:512 Loops:512 Thr:256 Vec:2
Speed.#2.........: 65133.3 MH/s (20.97ms) @ Accel:1024 Loops:1024 Thr:32 Vec:1
Speed.#*.........: 100.5 GH/s
----------------------------
* Hash-Mode 5600 (NetNTLMv2)
----------------------------
Speed.#1.........: 2653.4 MH/s (62.50ms) @ Accel:16 Loops:1024 Thr:256 Vec:1
Speed.#2.........: 4318.1 MH/s (80.78ms) @ Accel:64 Loops:512 Thr:256 Vec:1
Speed.#*.........: 6971.5 MH/s
--------------------------------------------------------
* Hash-Mode 1500 (descrypt, DES (Unix), Traditional DES)
--------------------------------------------------------
Speed.#1.........: 1387.0 MH/s (59.75ms) @ Accel:64 Loops:1024 Thr:32 Vec:1
Speed.#2.........: 2270.4 MH/s (76.47ms) @ Accel:128 Loops:1024 Thr:32 Vec:1
Speed.#*.........: 3657.4 MH/s
------------------------------------------------------------------------------
* Hash-Mode 500 (md5crypt, MD5 (Unix), Cisco-IOS $1$ (MD5)) [Iterations: 1000]
------------------------------------------------------------------------------
Speed.#1.........: 12121.3 kH/s (79.28ms) @ Accel:32 Loops:1000 Thr:1024 Vec:1
Speed.#2.........: 16734.3 kH/s (67.08ms) @ Accel:128 Loops:1000 Thr:256 Vec:1
Speed.#*.........: 28855.6 kH/s
----------------------------------------------------------------
* Hash-Mode 3200 (bcrypt $2*$, Blowfish (Unix)) [Iterations: 32]
----------------------------------------------------------------
Speed.#1.........: 30850 H/s (63.04ms) @ Accel:128 Loops:32 Thr:16 Vec:1
Speed.#2.........: 61759 H/s (61.66ms) @ Accel:256 Loops:32 Thr:16 Vec:1
Speed.#*.........: 92609 H/s
--------------------------------------------------------------------
* Hash-Mode 1800 (sha512crypt $6$, SHA512 (Unix)) [Iterations: 5000]
--------------------------------------------------------------------
Speed.#1.........: 263.0 kH/s (48.67ms) @ Accel:8192 Loops:256 Thr:32 Vec:1
Speed.#2.........: 449.8 kH/s (57.04ms) @ Accel:2048 Loops:512 Thr:128 Vec:1
Speed.#*.........: 712.8 kH/s
--------------------------------------------------------
* Hash-Mode 7500 (Kerberos 5, etype 23, AS-REQ Pre-Auth)
--------------------------------------------------------
Speed.#1.........: 641.1 MH/s (64.62ms) @ Accel:256 Loops:128 Thr:32 Vec:1
Speed.#2.........: 1386.1 MH/s (62.61ms) @ Accel:512 Loops:128 Thr:32 Vec:1
Speed.#*.........: 2027.2 MH/s
-------------------------------------------------
* Hash-Mode 13100 (Kerberos 5, etype 23, TGS-REP)
-------------------------------------------------
Speed.#1.........: 614.7 MH/s (67.42ms) @ Accel:256 Loops:128 Thr:32 Vec:1
Speed.#2.........: 1330.1 MH/s (65.35ms) @ Accel:512 Loops:128 Thr:32 Vec:1
Speed.#*.........: 1944.8 MH/s
---------------------------------------------------------------
* Hash-Mode 15300 (DPAPI masterkey file v1) [Iterations: 23999]
---------------------------------------------------------------
Speed.#1.........: 101.0 kH/s (68.36ms) @ Accel:16 Loops:512 Thr:512 Vec:1
Speed.#2.........: 222.0 kH/s (65.60ms) @ Accel:1024 Loops:256 Thr:32 Vec:1
Speed.#*.........: 322.9 kH/s
---------------------------------------------------------------
* Hash-Mode 15900 (DPAPI masterkey file v2) [Iterations: 12899]
---------------------------------------------------------------
Speed.#1.........: 57100 H/s (54.36ms) @ Accel:64 Loops:512 Thr:32 Vec:1
Speed.#2.........: 96573 H/s (67.80ms) @ Accel:64 Loops:512 Thr:64 Vec:1
Speed.#*.........: 153.7 kH/s
------------------------------------------------------------------
* Hash-Mode 7100 (macOS v10.8+ (PBKDF2-SHA512)) [Iterations: 1023]
------------------------------------------------------------------
Speed.#1.........: 660.0 kH/s (37.41ms) @ Accel:32 Loops:511 Thr:64 Vec:1
Speed.#2.........: 1125.5 kH/s (59.48ms) @ Accel:256 Loops:255 Thr:32 Vec:1
Speed.#*.........: 1785.5 kH/s
---------------------------------------------
* Hash-Mode 11600 (7-Zip) [Iterations: 16384]
---------------------------------------------
Speed.#1.........: 573.3 kH/s (63.02ms) @ Accel:128 Loops:4096 Thr:32 Vec:1
Speed.#2.........: 1160.9 kH/s (65.55ms) @ Accel:64 Loops:4096 Thr:128 Vec:1
Speed.#*.........: 1734.3 kH/s
------------------------------------------------
* Hash-Mode 12500 (RAR3-hp) [Iterations: 262144]
------------------------------------------------
Speed.#1.........: 74374 H/s (67.74ms) @ Accel:4 Loops:16384 Thr:512 Vec:1
Speed.#2.........: 171.8 kH/s (61.83ms) @ Accel:16 Loops:16384 Thr:256 Vec:1
Speed.#*.........: 246.2 kH/s
--------------------------------------------
* Hash-Mode 13000 (RAR5) [Iterations: 32799]
--------------------------------------------
Speed.#1.........: 62883 H/s (80.60ms) @ Accel:128 Loops:256 Thr:128 Vec:1
Speed.#2.........: 133.0 kH/s (80.30ms) @ Accel:128 Loops:512 Thr:128 Vec:1
Speed.#*.........: 195.8 kH/s
-----------------------------------------------------------------------
* Hash-Mode 6211 (TrueCrypt RIPEMD160 + XTS 512 bit) [Iterations: 1999]
-----------------------------------------------------------------------
Speed.#1.........: 443.9 kH/s (83.47ms) @ Accel:32 Loops:128 Thr:512 Vec:1
Speed.#2.........: 772.0 kH/s (54.01ms) @ Accel:32 Loops:256 Thr:256 Vec:1
Speed.#*.........: 1216.0 kH/s
-----------------------------------------------------------------------------------
* Hash-Mode 13400 (KeePass 1 (AES/Twofish) and KeePass 2 (AES)) [Iterations: 24569]
-----------------------------------------------------------------------------------
Speed.#1.........: 37133 H/s (91.49ms) @ Accel:16 Loops:512 Thr:256 Vec:1
Speed.#2.........: 127.1 kH/s (55.84ms) @ Accel:512 Loops:128 Thr:64 Vec:1
Speed.#*.........: 164.2 kH/s
----------------------------------------------------------------
* Hash-Mode 6800 (LastPass + LastPass sniffed) [Iterations: 499]
----------------------------------------------------------------
Speed.#1.........: 3666.6 kH/s (62.65ms) @ Accel:128 Loops:124 Thr:256 Vec:1
Speed.#2.........: 7624.5 kH/s (57.02ms) @ Accel:1024 Loops:249 Thr:32 Vec:1
Speed.#*.........: 11291.1 kH/s
--------------------------------------------------------------------
* Hash-Mode 11300 (Bitcoin/Litecoin wallet.dat) [Iterations: 200459]
--------------------------------------------------------------------
Speed.#1.........: 7636 H/s (86.95ms) @ Accel:4096 Loops:128 Thr:256 Vec:1
Speed.#2.........: 13347 H/s (49.90ms) @ Accel:8192 Loops:512 Thr:32 Vec:1
Speed.#*.........: 20982 H/sThe performance compared to bechmarks for other GPUs shows that the overall performance of my configuration is well above a single Nvidia RTX 3090, but as expected not as powerful as the Nvidia RTX 4090. There were no hashcat benchmarks publicly available for the Nvidia RTX 4080 at the time of writing this article.
In order to simulate a real-life scenario, I attempted to perform a password guess on a Linux sha512crypt hash using the well-known rockyou.txt password list that comes bundled with the popular penetration testing distro Kali. This list contains 14,344,391 unique passwords.
The command below was used to execute hashcat:
PS> .\hashcat.exe -m 1800 -a 0 .\hashes.txt ..\wordlists\rockyou.txtFirst the tool prints some information about the GPUs that will be used, optimizer options and details about the password list provided.
A summary of the results is provided after the tool finishes the assessment.
As shown at the bottom of the screenshot. It took hashcat less than 2 minutes to assess the provided hash against more than 14 million passwords. Not a bad result considering the complexity of this type of password hash.
Final thoughts
Our limited testing indicates that mixing Nvidia and AMD GPUs for password auditing seem to work well, at least when a recent version of the tool hashcat is used. Other tools leverage different drivers, so the results can differ significantly.
