Crypto in Malware / Reverse Engineering

Tip

Jifunze na fanya mazoezi ya AWS Hacking:HackTricks Training AWS Red Team Expert (ARTE)
Jifunze na fanya mazoezi ya GCP Hacking: HackTricks Training GCP Red Team Expert (GRTE) Jifunze na fanya mazoezi ya Azure Hacking: HackTricks Training Azure Red Team Expert (AzRTE)

Support HackTricks

This subsection helps when you see crypto/compression inside binaries and want to recognize it quickly.

Identifying cryptographic / compression algorithms

Technique-first heuristics

  • Lots of shifts/rotates, XORs, and 32-bit arithmetic in tight loops.
  • Lookup tables (S-boxes) in .data or generated at runtime.
  • Repeating loops of 0x100 iterations hinting RC4.

Windows crypto/compression APIs

CryptDeriveKey / CryptCreateHash

If these are used, the second parameter is an ALG_ID:

Table: https://learn.microsoft.com/en-us/windows/win32/seccrypto/alg-id

RtlCompressBuffer / RtlDecompressBuffer

Often indicates built-in Windows compression (LZNT1, XPRESS, etc).

Constants & tables

Sometimes you can fingerprint a hash/cipher by searching constants (or the first dword of tables) online.

AES tables example:

RC4 recognition notes

RC4 is often recognizable by:

  • Two loops of 256 iterations (init + KSA)
  • Then a PRGA loop using % 256 and XORing a keystream with data

Unpacking binaries

Technique

Packers transform a binary so static analysis is misleading (junk code, encrypted sections, runtime unpacking). The goal is to catch the moment it:

  • allocates/decrypts real code in memory
  • marks it executable
  • jumps into it

Identifying packed binaries

  • Lack of strings (or only packer strings)
  • Many strings without xrefs (commercial packers)
  • Use packer-ID tools:
  • PEiD
  • Exeinfo PE

Basic recommendations

  • Start analysis from the bottom and work upward; unpackers often jump late.
  • Look for JMP/CALL reg patterns or stack tricks (push addr; retn).
  • Breakpoint on VirtualAlloc/VirtualProtect and track RWX regions.
  • A sudden strings explosion after a jump often indicates you reached unpacked code.
  • Dump memory and fix headers with tools like PE-bear.

Tip

Jifunze na fanya mazoezi ya AWS Hacking:HackTricks Training AWS Red Team Expert (ARTE)
Jifunze na fanya mazoezi ya GCP Hacking: HackTricks Training GCP Red Team Expert (GRTE) Jifunze na fanya mazoezi ya Azure Hacking: HackTricks Training Azure Red Team Expert (AzRTE)

Support HackTricks