El libro

Introduction

This article is used to keep notes and summaries of the book “Encrypting and Decrypting a Software”.
The content will be continuously updated as I read through the book.

Reflection

This book covers many topics and has almost 1,000 pages.

The book includes the following topics:

• Software cracking (patching and bypassing)
• Mathematical cryptography (with formulas and expressions)
• Software vulnerabilities (stack buffer overflow and heap buffer overflow)
• Software packing, unpacking, and packer stubs
• Reverse engineering

Most of the topics focus on Windows rather than Linux.

Personally, I do not recommend this book for beginners in reverse engineering. Although the topics are broad, the book does not explore them in great depth. While the book introduces a wide range of topics, some of them could benefit from more detailed explanations, which may make the book less accessible to beginners.

There are several typos in the book, such as misusing the terms byte and bit, which was quite confusing.

The biggest drawback is that the example source code is very difficult to obtain for foreign readers. It would be excellent if the authors uploaded their source code to GitHub, similar to the author of Reverse Engineering Core.

On the other hand, since the book covers many topics, it can be used as a reference or dictionary. In addition, it includes some topics that are rarely discussed in other books.

Chapter 1 - Fundamental Knowledge

1.3 - Windows OS

Win32 API

• Kernel (kernel32.dll): Processes, threads, memory, file IO, etc.
• User (user32.dll): Mouse click, keyboard input, window, menu, etc.
• GDI (gdi32.dll): Display text.
• advapi32.dll
• comctl32.dll
• comdlg32.dll
• shell32.dll
• netapi32.dll

For Win32 API functions, “A” stands for ANSI. “W” stands for Widechars(Unicode).

int MessageBox(
    HWND hWnd,
    LPCTSTR lpText,
    LPCTSTR lpCaption,
    UINT uType
);

WOW64

WOW64 stands for Windows on Windows 64-bit.

Windows Message Mechanism

SendMessage()

LRESULT SendMessage(
    HWND hWnd,
    UINT Msg,
    WPARAM wParam,
    LPARAM lParam
);

WM_COMMAND

WM_COMMAND
    wNotifyCode = HIWORD(wParam);
    wID = LOWORD(wParam);
    hwndCtl = (HWND)lParam;

WM_GETTEXT

WM_GETTEXT
    wParam = (WPARAM)cchTextMax;
    lParam = (LPARAM)lpszText;

WM_QUIT

WM_QUIT
    nExitCode = (int)wParam;

WM_LBUTTONDOWN

WM_LBUTTONDOWN
    fwKeys = wParam;
    xPos = LOWORD(lParam);
    yPos = HIWORD(lParam);

Virtual Memory

Chapter 2 - Dynamic Analysis

2.1 - OllyDbg

BreakPoint

1. INT 3

   004013A5 CC D0404000

   OllyDbg replaces the original instruction to 0xCC. However, the original instruction in displayed.

2. Hardware BreakPoint
   Hardware break points are implemented with DRx registers. The addresses of break points are stored into DR0 ~ DR3, while DR7 stores the status. Thus, the maximum number of hardware break points are is 4.

   

   

   

   

   

   

3. Memory BreakPoint

   1. On Access

   2. On Write

      

4. Set break-on-access

   

5. Message BreakPoint

   1. View -> Windows

   2. Message breakpoint on ClassProc

      

      

6. Conditional BreakPoint

   

   Shift+F2

7. Conditional Log

2.3 - MDebug

2.4 - WinDbg

Chapter 3 - Static Analysis

3.2 - Disassembly Engine

1. ODDisasm
2. BeaEngine
3. Udis86
4. Capstone
5. AsmJit
6. Keystone

3.3 - Static Disassembly

IDA

Hexadecimal Tools

• HexWorkship
• WinHex
• Hiew

Chapter 4 - Reverse Engineering Techniques

4.1 - Reverse Engineering on x32 Platform

Calling Convention

Example1:

test1(p1, p2, p3);

__cdecl

push p3     ; right to left
push p2
push p1
call test1
add esp,0C  ; stack cleanup

pascal

push p1     ; left to right
push p2
push p3
call test1  ; stack cleanup in the function

stdcall

push p3     ; right to left
push p2
push p1
call test1  ; stack cleanup in the function

---

Example2:

test2(p1, p2);

stdcall

push p2                         ; Parameter 2
push p1                         ; Parameter 1
call test2
{
    push ebp                    ; Original EBP pointer
    mov ebp, esp                ; New EBP pointer
    mov eax, dword ptr[ebp+0C]  ; Parameter 2
    mov ebx, dword ptr[ebp+08]  ; Parameter 1
    sub esp, 8                  ; Reserve space for local variables

    ...

    add esp, 8                  ; Release space of local variables
    pop ebp                     ; Original EBP pointer
    ret 8                       ; return (Equal to ret # add esp, 8)
                                ; 2 (Parameter) x 4h = 8
}

• enter: push ebp # mov ebp,esp # sub esp, xxx
• leave: add esp, xxx # pop ebp

We can rewrite the script above

enter xxxx, 0   ; Reserver space xxxx for local variable

...

leave
ret 8

---

Enable Maximum Speed in VC6.0

push p2
push p1
call test2
{
    mov eax, dword ptr [esp+04]     ; Call parameter 1
    mov ecx, dword ptr [esp+08]     ; Call parameter 2

    ...

    ret 8                           ; return
}

Virtual Function

Chapter 5 - Software Protection Demonstration

5.1 - Serial Number and KeyGen

1. $$\text{Serial Number} = F(\text{Username})$$
2. $$\text{Username} = F^{-1}(\text{Serial Number})$$
3. $$F_1(\text{Username}) = F_2(\text{Serial Number})$$
4. $$Some value = F(\text{Username}, \text{Serial Number})$$

Exploiting Serial Number Protection Mechanism

Commonly used API functions for Serial Number Protection:

1. The following functions are commonly used for input:
   • GetWindowTextA(W)
   • GetDlgItemTextA(W)
   • GetDlgItemInt
   • hmencpy (Windows 9x/Me only)
2. The floowing functions are commonly used for messagebox:
   • MessageBoxA(W)
   • MessageBoxExA(W)
   • ShowWindow
   • MessageBoxIndirectA(W)
   • CreateDialogParamA(W)
   • CreateDialogIndirectParamA(W)
   • DialogBoxParamA(W)
   • DialogBoxIndirectParamA(W)
3. Other functions:
   1. Serial number might be stored in registry:
      • RegQueryValueExA(W)
   2. Serial number might be stored in an INI file:
      • GetPrivateProfileStringA(W)
      • GetPrivateProfileIntA(W)
      • GetProfileStringA(W)
      • GetProfileIntA(W)
   3. Serial number might be stored in other file:
      • CreateFileA(W)
      • _lopen()

KeyFile Protection

API Function	Description
FindFirstFileA
CreateFileA, _lopen
GetFileSize, GetFileSizeEx
GetFileAttributesA, GetFileAttributesExA
SetFilePointer, SetFilePointerEx	Mov
ReadFile	Obtain file content.

5.7 - Disk Protection

5.8 - Single Instance

Chapter 6 - Cryptography Algorithm

6.1 - Hashing

MD5

MD5 stands for Message Digest Algorithm 5.

$$A = 01234567h\\ B = 89abcdefh\\ C = fedcba98h\\ D = 76543210h\\$$ $$\begin{align*} F(X,Y,Z) &= (X\&Y) | ((\sim X)\&Z)\\ G(X,Y,Z) &= (X\&Z) | (Y\&(\sim Z))\\ H(X,Y,Z) &= X\text{\textasciicircum}Y\text{\textasciicircum}Z\\ I(X,Y,Z) &= Y\text{\textasciicircum}(X|(\sim Z)) \end{align*}$$

SHA

SHA stands for Secure Hash Algorithm.

• SHA-1
• SHA-256
• SHA-384
• SHA-512

6.2 - Symmetric Encryption

• DES (Data Encryption Standard)
• IDEA (International Data Encryption Algorithm)
• AES (Advanced Encryption Standard)
• BlowFish
• TwoFish

RC4

RC4 stands for Rivest Cipher 4.

KSA (Key-Scheduling Algorithm)

RC4-KSA(Key)
    for i := 0 to 255 do
        S[i] := i
    end for

    j := 0
    for i := 0 to 255 do
        j := (j + S[i] + Key[i mod keylength]) mod 256
        swap S[i] and S[j]
    end for

    return S

PRGA (Pseudo-Random Generation Algorithm)

RC4-PRGA(S)
    i := 0
    j := 0

    while true do
        i := (i + 1) mod 256
        j := (j + S[i]) mod 256
        swap S[i] and S[j]
        t := (S[i] + S[j]) mod 256
        K := S[t]
        output K
    end while

TEA

TEA stands for Tiny Encryption Algorithm.

TEA_Encrypt(v[2], k[4])
{
    uint32 v0 = v[0], v1 = v[1];
    uint32 sum = 0;
    uint32 delta = 0x9E3779B9;

    for (i = 0; i < 32; i++)
    {
        sum += delta;
        v0 += ((v1 << 4) + k[0]) ^ (v1 + sum) ^ ((v1 >> 5) + k[1]);
        v1 += ((v0 << 4) + k[2]) ^ (v0 + sum) ^ ((v0 >> 5) + k[3]);
    }

    v[0] = v0;
    v[1] = v1;
}

TEA_Decrypt(v[2], k[4])
{
    uint32 v0 = v[0], v1 = v[1];
    uint32 delta = 0x9E3779B9;
    uint32 sum = delta * 32;

    for (i = 0; i < 32; i++)
    {
        v1 -= ((v0 << 4) + k[2]) ^ (v0 + sum) ^ ((v0 >> 5) + k[3]);
        v0 -= ((v1 << 4) + k[0]) ^ (v1 + sum) ^ ((v1 >> 5) + k[1]);
        sum -= delta;
    }

    v[0] = v0;
    v[1] = v1;
}

Where:

$$\begin{align*} delta &= \frac{\sqrt{5} - 1}{2} \times 2^{32}\\ &= (\sqrt{5} - 1) \times 2^{31} \end{align*}$$

IDEA

BlowFish

AES

Chapter 7 - Windows Kernel Fundamentals

7.1 - Kernel Fundamentals

Source: https://www.linkedin.com/pulse/windows-os-rings-role-event-monitoring-jacob-stickney-thv5c

7.2 - Important Kernel Data Structure

7.3 - Fundamentals of Kernel Debugging

Chapter 8 - Windows Exception Handling

8.1 - Fundamentals of Exception Handling

VOID RaiseException(
  [in] DWORD           dwExceptionCode,
  [in] DWORD           dwExceptionFlags,
  [in] DWORD           nNumberOfArguments,
  [in] const ULONG_PTR *lpArguments
);

//Source: https://learn.microsoft.com/en-us/windows/win32/api/errhandlingapi/nf-errhandlingapi-raiseexception

When an interrupt or exception occurs, the CPU looks up the corresponding handler in the IDT.

The IDT (Interrupt Descriptor Table) is a linear table stored in memory. It consists of 256 entries. Each entry is 8 bytes long on x86 (32-bit) platforms and 16 bytes long on x64 (64-bit) platforms.

The base address and size of the IDT are described by the IDTR (IDT Register). The IDTR is 48 bits long on x86 systems. Although the IDTR can be accessed using the SIDT and LIDT instructions, the LIDT instruction can only be executed with Ring 0 privilege.

8.2 - SEH

Chapter 9 - Win32 API Debugging

Chapter 14 - Vulnerability Techniques

14.1 - Software Vulnerability

Stack Buffer Overflow

Heap Buffer Overflow

#include <stdio.h>
#include <Windows.h>

int main(int argc, char *argv[]) {
    char str[] = "\nHello123456789213456789\n";
    char *a, *b, *c;
    long *hHeap;
    hHeap = (long *)HeapCreate(0x00040000, 0, 0);
    printf("\n(+) Creating a heap at: 0x00%xh\n", hHeap);
    printf("(+) Allocating chunk A\n");

    a = (char *)HeapAlloc(hHeap, HEAP_ZERO_MEMORY, 0x10);
    printf("(+) Allocating chunk B\n");
    b = (char *)HeapAlloc(hHeap, HEAP_ZERO_MEMORY, 0x10);
    printf("(+) Chunk A=0x00%x\n(+) Chunk=0x00%x\n", a, b);
    printf("(+) Freeing chunk B to the lookaside\n");
    HeapFree(hHeap, 0, b);
    printf("(+) Now overflow chunk A:\n");

    printf("%x\n", str);
    printf(str);
    memcpy(a, "XXXXXXXXXXXXXXXXAAAABBBB\x64\xff\x12\x00", 28);
    printf("(+) Allocating chunk B\n");
    b = (char *)HeapAlloc(hHeap, HEAP_ZERO_MEMORY, 0x10);
    printf("(+) Allocating chunk C\n");
    c = (char *)HeapAlloc(hHeap, HEAP_ZERO_MEMROY, 0x10);
    printf("(+) Chunk A=0x00%x\n(+) Chunk B = 0x00%x\n(+) Chunk C=0x00%x\n", a, b, c);
    strcpy(c, "AAAAAAAAAAAA\n");
    printf(str);

    return 0;
}

14.2 - Shellcode

CHapter 15 - Software Unpacking

15.2 - Packer

• UPX
• ASPack

15.3 - Packer Stub

• ASProtect
• Armadillo
• EXECryptor
• Themida

THANKS FOR READING