Home Writeup

# Some pretext

Me, Chandra and Sidharth recently placed 3rd in a Reverse Engineering Hackathon/CTF conducted by IIT Madras and IIITDM Kancheepuram. This was very unique as it wasn't like any of the other RE focused CTFs we've played before and the style was really different.

The entire thing began with a CTF which had all categories you would expect in a normal CTF (RE, Pwn, Web, Forensics). Us being low level guys, managed to get down RE and Pwn, and took a bit of time with Forensics. Sadly Web was too guessy for us ๐Ÿ˜” (blind + skill issue). They froze the scoreboard like 30 mins into the CTF but I figure we qualified since they sent us a mail for the next round.

After the prelims CTF, I think the top 25-30 teams were picked for the next round. The next round was a RE specific CTF. That one was super easy except for one challenge which was broken, I think. They pushed a fix for it and then we solved that too. So far it was looking pretty normal and we weren't sure what the "hackathon" part of this was.

# What is a hackathon tho?

For context, a Hackathon is a software engineering/development oriented competition wherein teams are given certain topics/themes and guidelines to follow, and they have X amount of time to build a working product or a prototype, which they can then showcase to the judges and based on the judgement received they would get credited a certain number of points.

# Defense phase?

So after the 2 prelim rounds of CTFs, 20-25 teams were picked for the Hackathon (aka "Defense") phase. This is where the uniqueness of the entire thing began. We were given a document that outlined what we would need to do and the specifications we were allowed to use. The entire specification is here.

TL;DR

  • We are given a main.c file, with 3 sensitive values: key (a 16-byte AES Key), egg_params (a 5x6 array) and a function compute_gf()
  • Each team is given the same core binary with the above 3 things changed
  • You "defend" your binary by obfuscating it such that the 3 sensitive values are hard to recover
  • You "attack" other binaries by reversing and getting those 3 values, and submit them as flags

In the defense phase, we were given criterion to follow with which we were to obfuscate our binaries. These were pretty basic, like:

  • No using any off the shelf obfuscators like UPX, CobaltStrike, etc.
  • Code must be written by members of our team, and no one else
  • The obfuscated binary must be functionally the same as the original, etc.

Okay.. enough yap
This blog is more to focus on one of the challenges made by some folks at InfosecIITR.

They made an LLVM based obfuscator, and while our challenge was something similar, it followed a different style from theirs. We went for a control flow flattening approach - while they went for more of a instruction overlap approach. It was an easy challenge, but I just wanted to showcase some cool things IDAPython is capable of.

# The Challenge

The binary itself had a behaviour exactly the same as the others in the competition - give it plaintext as a command line argument, and it would encrypt it and give it back to you.

1
2
3
4
5
6
7
8
apollo@apollo:/mnt/f/Work/CTFs/binary-clash-25/attack-phase/3$ ./3_VMwhere
Invalid Usage!!
Usage: ./encrypt <plain_text>
apollo@apollo:/mnt/f/Work/CTFs/binary-clash-25/attack-phase/3$ ./3_VMwhere hello
Plaintext :: 68 65 6c 6c 6f 00 00 00 00 00 00 00 00 00 00 00
Ciphertext:: a0 6b 44 12 fc bd d2 a0 0c 71 d5 4b ba d6 84 94
Egg 0 : 0x28
Global Flag: 0x77

Now, looking at it in IDA, I wouldn't say it was the prettiest binary I've ever seen.
If you look for the main function in the normal Linux ELF method (look for start , then find the first argument passed to the function - libc_start_main - which is explicitly called by start), you will first off see that IDA marks it as a location and not a subroutine - meaning that something is off already.

looking at start in IDA Pro

And at the location:
disassembly of main

That definitely does not look right.

The jmp at 0x40B234 definitely looks suspicious, why is it jumping to the address of its own basic block + 5?

You can press u to undefine code at a given position in IDA, and c to redefine it as code. Let us undefine code starting from 0x40B228 , and defone code starting at 0x40B228 + 5 -> 0x40B22D , and see where that takes us.

hidden jump

Looks like the code there is being interpreted as data, we can fix that easily (hit u at that address and then hit c ). Doing so, gives us this

hidden jump but fr this time

Well well well, what do we have here? A hidden jmp instruction.

Analysing the code at that location gives us similar looking stuff

2nd obfuscation place

Applying the same stuff again, gives us the same results, except an instruction before the same old block again

jmp which leads to actual stuff

Again, with this jmp leading to a similar looking block, i.e:

  1. mov a random 8-byte value into rax
  2. xor eax, eax
  3. jmp into a location which is 5 bytes ahead of the basic block it belongs to, which will jump to a new location which may/may not have some code there, before it enters the same block again.

At this point in the competition the pattern was clear to me, and I started figuring out a way to extract just the instructions we need (between the real jmp and before the next obfuscated block). So I settled on writing down an IDAPython script for this, since I work mainly with IDA Pro anyways.

My idea was this:

  1. Look for instructions matching the format mentioned above
  2. If the instruction is found, then find the 5th byte offset from the start of the block, and extract the address of the real jmp from on there
  3. At the new jmp address, keep reading instructions until you meet the next instruction matching the block

Rinse. Repeat.

I wanted to take this as a chance to check out how good IDA's Python API was, and basically take it on a test drive.

Now I had to write an IDA Script to this (๐Ÿ—ฟ), ahem, so we shall do that next

The IDA Python documentation isn't exactly the best (if someone actually does find something, please hit me up), so I kind of had to rely mostly on grep.app to see examples of stuff similar to what I want to do.

Some time of dilly dallying, and I made a rough script to fix up the main function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
import ida_ua, idautils, idc, idaapi, ida_bytes, ida_funcs, ida_hexrays, ida_name, ida_idp, ida_segment
from ida_ua import insn_t

from collections import deque

def get_register_map():
    reg_map = {}
    proc = ida_idp.ph.regnames

    for idx, reg_name in enumerate(proc):
        if reg_name:
            reg_map[idx] = reg_name.lower()

    return reg_map

def get_instruction_at(address):
    insn = idaapi.insn_t()
    length = ida_ua.decode_insn(insn, address)

    if length == 0:
        print(f"0x{address:x}: No valid instruction found")
        return None

    mnemonic = insn.get_canon_mnem()

    disasm = idc.generate_disasm_line(address, 0)
    clean_disasm = idaapi.tag_remove(disasm)

    if " " in clean_disasm:
        operand_part = clean_disasm.split(" ", 1)[1]
    else:
        operand_part = ""

    bytes_data = ida_bytes.get_bytes(address, length)
    hex_bytes = " ".join(f"{b:02x}" for b in bytes_data)

    hex_bytes = bytes.fromhex(hex_bytes)
    return address, len(hex_bytes), mnemonic, operand_part

def is_messed_instruction(call_loc):
    """
    instructions of the type:
    mov     rax, 48FFFF08EB0B6754h
    xor eax, eax
    """
    global reg_map
    cur_insn = ida_ua.insn_t()
    insn_len = ida_ua.decode_insn(cur_insn, call_loc)

    if insn_len != 10:
        return False

    # check if it's a mov to rax with a large immediate
    try:
        if cur_insn.get_canon_mnem() != 'mov' or reg_map[cur_insn.Op1.reg] != 'rax':
            return False

        hex_value = hex(cur_insn.Op2.value)[2:]
        if len(bytes.fromhex(hex_value)) != 8:
            return False
    except (KeyError, ValueError):
        return False

    # check the second instruction (xor eax, eax)
    next_insn = ida_ua.insn_t()
    next_insn_len = ida_ua.decode_insn(next_insn, call_loc+10)

    if next_insn_len != 2:
        return False

    try:
        if next_insn.get_canon_mnem() != 'xor' or reg_map[next_insn.Op1.reg] != 'rax' or reg_map[next_insn.Op2.reg] != 'rax':
            return False
    except KeyError:
        return False

    # now we know for sure it is a messed instruction
    # calculate the jump address
    jmp_insn = ida_ua.insn_t()
    jmp_insn_len = ida_ua.decode_insn(jmp_insn, call_loc+12)

    if jmp_insn_len == 0 or jmp_insn.get_canon_mnem() != 'jmp':
        return False

    # jmp address - return as an integer, not a hex string
    jmp_addr = jmp_insn.Op1.addr
    print(f"jmp_addr = {hex(jmp_addr)}")
    return True, jmp_addr, jmp_addr - call_loc

# initialize register map
reg_map = get_register_map()
for i in range(8):
    if i in reg_map:
        reg_map[i] = 'r' + reg_map[i]

main_start = 0x407EB0
curr_addr = main_start
end_addr = 0x40AA40


while curr_addr < end_addr:
    result = is_messed_instruction(curr_addr)

    if result:
        is_messed, jmp_addr, length = result

        print(f"Found messed instruction at {hex(curr_addr)}, jumping to {hex(jmp_addr)}")

        # NOP out the bogus instruction 
        for i in range(5):
            idc.patch_byte(curr_addr + i, 0x90)

        # skip the next 5 bytes
        curr_addr += 5

        # force create an instruction at the jump address in idb
        idc.create_insn(jmp_addr)

        # decode the instruction at the jump address so we can get the new jump address (chain basically)
        jmp_insn = ida_ua.insn_t()
        jmp_insn_len = ida_ua.decode_insn(jmp_insn, jmp_addr)

        if jmp_insn_len > 0 and jmp_insn.get_canon_mnem() == 'jmp':
            jmp_chain_addr = jmp_insn.Op1.addr
            print(f"jmp_chain_addr = {hex(jmp_chain_addr)}")

            # NOP out the bogus instruction
            range_len = jmp_chain_addr - jmp_addr
            for i in range(range_len):
                idc.patch_byte(jmp_addr + i, 0x90)
        else:
            print(f"No valid jump instruction at {hex(jmp_addr)}")
    else:
        print(f"No messed instruction found at {hex(curr_addr)}")

Soon after this, I wanted to fix up the entire binary - whichever functions were being called that is. So I went with a DFS approach.
From main, make a list of all the functions being called from inside it. Subsequently, add those to the "deobfuscation" queue. Likewise, keep going until there are no functions left.

Also one annoying thing is inside IDA's API, all the instructions and registers are stored using codes instead of their actual names. I found that the method I've used in my script works for most cases.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
import ida_ua, idautils, idc, idaapi, ida_bytes, ida_funcs, ida_hexrays, ida_name, ida_idp, ida_segment
from ida_ua import insn_t
from collections import deque

def get_register_map():
    reg_map = {}
    proc = ida_idp.ph.regnames

    for idx, reg_name in enumerate(proc):
        if reg_name:
            reg_map[idx] = reg_name.lower()

    return reg_map

def get_instruction_at(address):
    insn = idaapi.insn_t()
    length = ida_ua.decode_insn(insn, address)

    if length == 0:
        print(f"0x{address:x}: No valid instruction found")
        return None

    mnemonic = insn.get_canon_mnem()

    disasm = idc.generate_disasm_line(address, 0)
    clean_disasm = idaapi.tag_remove(disasm)

    if " " in clean_disasm:
        operand_part = clean_disasm.split(" ", 1)[1]
    else:
        operand_part = ""

    bytes_data = ida_bytes.get_bytes(address, length)
    hex_bytes = " ".join(f"{b:02x}" for b in bytes_data)

    hex_bytes = bytes.fromhex(hex_bytes)
    return address, len(hex_bytes), mnemonic, operand_part

def is_messed_instruction(call_loc):
    """
    instructions of the type:
    mov     rax, 48FFFF08EB0B6754h
    xor eax, eax
    """
    global reg_map
    cur_insn = ida_ua.insn_t()
    insn_len = ida_ua.decode_insn(cur_insn, call_loc)

    if insn_len != 10:
        return False

    # check if it's a mov to rax with a large immediate
    try:
        if cur_insn.get_canon_mnem() != 'mov' or reg_map[cur_insn.Op1.reg] != 'rax':
            return False

        hex_value = hex(cur_insn.Op2.value)[2:]
        if len(bytes.fromhex(hex_value)) != 8:
            return False
    except (KeyError, ValueError):
        return False

    # check the second instruction (xor eax, eax)
    next_insn = ida_ua.insn_t()
    next_insn_len = ida_ua.decode_insn(next_insn, call_loc+10)

    if next_insn_len != 2:
        return False

    try:
        if next_insn.get_canon_mnem() != 'xor' or reg_map[next_insn.Op1.reg] != 'rax' or reg_map[next_insn.Op2.reg] != 'rax':
            return False
    except KeyError:
        return False

    # now we know for sure it is a messed instruction
    # calculate the jump address
    jmp_insn = ida_ua.insn_t()
    jmp_insn_len = ida_ua.decode_insn(jmp_insn, call_loc+12)

    if jmp_insn_len == 0 or jmp_insn.get_canon_mnem() != 'jmp':
        return False

    # jmp address - return as an integer, not a hex string
    jmp_addr = jmp_insn.Op1.addr
    print(f"jmp_addr = {hex(jmp_addr)}")
    return True, jmp_addr, jmp_addr - call_loc

def deobfuscate_function(start_addr, end_addr=None):
    """
    deobfuscate a function from start_addr to end_addr.
    if end_addr is not provided, use the function's end address.
    """
    print(f"Deobfuscating function at {hex(start_addr)}")
    
    # if end_addr is not provided, try to get it from IDA
    if end_addr is None:
        func = ida_funcs.get_func(start_addr)
        if func:
            end_addr = func.end_ea
        else:
            print(f"Warning: Could not determine end address for function at {hex(start_addr)}")
            # try to find the next function and use its start as our end
            next_func = ida_funcs.get_next_func(start_addr)
            if next_func:
                end_addr = next_func.start_ea
            else:
                print(f"Error: Cannot determine end address for function at {hex(start_addr)}")
                return
    
    print(f"function bounds: {hex(start_addr)} - {hex(end_addr)}")
    
    # queue for functions to deobfuscate (DFS)
    functions_to_process = []
    
    curr_addr = start_addr
    while curr_addr < end_addr:
        # check if it's a "messed" instruction
        result = is_messed_instruction(curr_addr)
        
        if result:
            is_messed, jmp_addr, length = result
            
            print(f"Found messed instruction at {hex(curr_addr)}, jumping to {hex(jmp_addr)}")
            
            # NOP out the bogus instruction 
            for i in range(5):
                idc.patch_byte(curr_addr + i, 0x90)
            
            # skip the next 5 bytes
            curr_addr += 5
            
            # force create an instruction at the jump address in IDB
            idc.create_insn(jmp_addr)
            
            # decode the instruction at the jump address so we can get the new jump address (chain basically)
            jmp_insn = ida_ua.insn_t()
            jmp_insn_len = ida_ua.decode_insn(jmp_insn, jmp_addr)
            
            if jmp_insn_len > 0 and jmp_insn.get_canon_mnem() == 'jmp':
                jmp_chain_addr = jmp_insn.Op1.addr
                print(f"jmp_chain_addr = {hex(jmp_chain_addr)}")
                
                # NOP out the bogus instruction
                range_len = jmp_chain_addr - jmp_addr
                for i in range(range_len):
                    idc.patch_byte(jmp_addr + i, 0x90)
            else:
                print(f"No valid jump instruction at {hex(jmp_addr)}")
        else:
            # check if it's a call instruction
            insn = ida_ua.insn_t()
            insn_len = ida_ua.decode_insn(insn, curr_addr)
            
            if insn_len > 0 and insn.get_canon_mnem() == 'call':
                # get the callee
                if insn.Op1.type == ida_ua.o_near:
                    call_target = insn.Op1.addr
                    print(f"Found call at {hex(curr_addr)} to {hex(call_target)}")
                    
                    # add the target to our list of functions to process
                    if call_target not in [f[0] for f in functions_to_process]:
                        # check if it's a valid function
                        func = ida_funcs.get_func(call_target)
                        if func:
                            functions_to_process.append((call_target, func.end_ea))
                        else:
                            # try and  create a function
                            if ida_funcs.add_func(call_target):
                                func = ida_funcs.get_func(call_target)
                                if func:
                                    functions_to_process.append((call_target, func.end_ea))
                                else:
                                    print(f"Warning: Could not create function at {hex(call_target)}")
                                    # still add it to the queue, we'll try to determine bounds later
                                    functions_to_process.append((call_target, None))
                            else:
                                print(f"Warning: Could not create function at {hex(call_target)}")
                                # still add it to the queue, we'll try to determine bounds later
                                functions_to_process.append((call_target, None))
            
            if insn_len > 0:
                curr_addr += insn_len
            else:
                curr_addr += 1
    
    return functions_to_process

reg_map = get_register_map()
for i in range(8):
    if i in reg_map:
        reg_map[i] = 'r' + reg_map[i]

main_start = 0x40B220
main_end = 0x40BC20

processed_functions = set()

function_queue = deque([(main_start, main_end)])

while function_queue:
    start_addr, end_addr = function_queue.popleft()
    
    if start_addr in processed_functions:
        continue
    
    # mark the function as processed if not alr processed 
    processed_functions.add(start_addr)
    
    new_functions = deobfuscate_function(start_addr, end_addr)
    
    # add new functions to the queue
    if new_functions:
        for func_start, func_end in new_functions:
            if func_start not in processed_functions:
                function_queue.append((func_start, func_end))

print(f"deobfuscation complete, processed {len(processed_functions)} functions")

This will fix up the entire binary, although you might have to manually fix up some function boundaries by yourself. You can do that by right clicking anywhere inside the function/subroutine already defined.

The script is kinda massive but I tried my best to make it as clean as possible so I can reuse parts of it later, hopefully someone shows me some goated IDA Python script snippets that are out there somewhere ๐Ÿ™

Thank you for reading this super-hurried post! If you have any questions or if I have made any errors, please reach out to me on Twitter/X ๐Ÿ˜„