Security Development

Flow of a Buffer Overflow Payload

Here is the flow of a buffer overflow payload with a NOP Sled.

  1. Jump to overwritten return address that (hopefully) points to somewhere in the NOPs (0×90)
  2. Consume the No Operations (NOPs)
  3. Execute the shell code


Trouble Spawning a Shell

I’ve been working on some of the examples out of the Hacking: The Art of Exploitation book which exploit a program with shellcode but I’m not having much luck. The exploit makes sense conceptually but I can’t get it to spawn the shellcode.

It uses the system() function to invoke the exploitable program from the shell with the payload as a commandline arg. I want to look at the stack in GDB of the command being invoked by the system() function … does anyone know how to do this or know if it’s even possible?

It’s possible for me to look at the stack when I pass the arguments myself from the commandline but the system() function is doing it instead cause it’s easier to construct the shellcode programmatically.

I also looked at the return value of the system() function and it’s zero meaning that the program exited normally. I’m curious if the somehow the command payload isn’t getting passed correctly so I was hoping to look at the stack…

My problems were related to trying to execute the exploit on a 64-bit system when the code was developed for 32-bit. After switching over to a 32-bit OS, I was able to spawn a shell no problem!

32 bit vs 64 bit exploitation

The exploit code in Hacking: The Art of Exploitation is targeted towards overwriting 32 bit return addresses (size 4). It uses code like this:

unsigned int ret;
ret = &var;
*((unsigned int *)(buffer+sizeof(unsigned int))) = ret;

I’m working on a 64 bit machine and thus my addresses are double in size. I’m guessing this means that the above code won’t work on my machine (I’m not sure?). I searched around and found an “unsigned integer pointer type” which is defined per architecture and it is indeed size 8 on my 64 bit machine. I changed the code to this:

uintptr_t ret;
ret = &var;
*((uintptr_t *)(buffer+sizeof(uintptr_t)) = ret;


Quickly Use BC to Convert Hex to Dec

This post is mainly for my own benefit because I’ll forget this if I don’t put it down somewhere. A lot of the times I need to convert hex to dec when I’m on the command line and it’s a pain to open up the calculator so here is the command line equivalent

echo "ibase=16; FFFF" | bc

Turning off buffer overflow protections in GCC

As I’m learning more and more about exploiting buffer overflows I’m realizing that it’s actually pretty hard to run the examples that teach you how to exploit buffer overflows. GCC (and other compilers) have built in support for mitigating the simple buffer overflows and it’s turned on by default.

With GCC you have to compile with the -fno-stack-protector option otherwise you get “***stack smashing detected***,” this is pretty well known and documented all over the net.

However, additionally you’ll need to disable the FORTIFY_SOURCE option otherwise you’ll get “Abort trap” if you try to do a buffer overflow that uses something like strcpy or memcpy.

To disable it, simply compile with the flag -D_FORTIFY_SOURCE=0 (e.g. gcc -g -fno-stack-protector -D_FORTIFY_SOURCE=0 -o overflow_example overflow_example.c)


Adobe PDF Return Oriented Exploit

The news just keeps getting worse for Adobe. After releasing an out-of-band patch for their other 0-day PDF exploit used to jailbreak the iPhone, the people at McAfee discovered malware in the wild exploiting another 0-day PDF exploit. Apparently this exploit takes advantage of your typical buffer overflow exploit (come on Adobe!) but the way it’s exploited is very interesting; (more…)

Finding the Return Address on the Stack

Chapter 3 of Hacking: The Art of Exploitation is all about exploitation. As mentioned in my post on stack-based buffer overflows, one of the key points to exploiting these types of overflows is gaining control of the return address from a function call.

I’ll be using the source code from my stack-based buffer overflows post. First, lets look at what main() looks like disassembled. (more…)

Void Function Pointers in C

Hacking: The Art of Exploitation has a really nice crash course on C. The book has a paragraph or two about function pointers.

Every time I look at function pointers, I find myself referring back to some documentation or getting confused (one too many *’s! or the parenthesis are wrong)

Here is a very simple example to demonstrate void function pointers, hopefully so I can remember in the future!.

#include <stdio.h>

void hello() {

void world() {

void space() {
	printf(" ");

void newline() {

void print(const void *f) {
	void (*f_ptr)() = f;

int main(int argc, char **argv) {
	return 0;

x86 64 Bit Stack Boundaries

While inspecting disassembled code on my Macbook Pro, I couldn’t figure out why the stack allocated so much space when I only requested a little. After some investigation into GCC I figured it out.

According to Apple’s developer documentation on GCC, “If -mpreferred-stack-boundary is not specified, the default is 4 (16 bytes or 128 bits).” That means that the stack will always allocate chunks in factors of 16 (2^4) unless you specify otherwise.

Types of Overflows: Stack-Based Overflows

A buffer is simply some fixed space in memory used to store data. In C, you create a buffer by declaring an array of some primitive type such as a ‘char array[SIZE]‘ or int ‘array[SIZE]‘. (more…)

Types of Overflows: Integer Overflows

Integer overflows occur when either a signed positive integer turns negative, a signed negative integer turns positive, or a large unsigned number “wraps around” to (or starts over at) zero. This can happen when you add, subtract, or multiply integers.

Some languages raise an exception when integer overflow occurs. C does not raise an exception; handling of integer overflows is left up to the programmer.

Integer overflows by themselves are mostly useless since they are likely to cause unexpected behavior or program failure but given the right circumstances, they can be used for buffer overflows (heap or stack based depending on the situation).

Here’s an example of an exploitable integer overflow. By inputting the correct integer, we can trick malloc into allocating zero bytes. Memcpy would then start copying over other elements in heap memory. Assuming sizeof(char) is 1, we just need to make size wrap around to zero. 4294967295 is the max unsigned integer; adding 1 to it will cause it to wrap around to 0!

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <limits.h>

void function(unsigned int size, char* input) {
	char *buf;
	buf = (char *) malloc(size * sizeof(char));
	memcpy(buf, input, size);

int main(int argc, char **argv) {
	unsigned int size = atoi(argv[1]);
	function(size, argv[2]);
	return 0;

Types of Overflows: Heap-Based Overflows

A heap-based buffer overflow is an overflow that occurs in dynamically allocated space called the heap. Data is stored in the heap when you call malloc(size). The concept is exactly the same as a stack-based overflow in the fact that you put more data in a buffer than was allocated but they are exploited in a much different way.

Exploitation of a heap based overflow varies greatly. Some heap overflows techniques overwrite data that another function may use. Other techniques involve overwriting heap data which contain pointers to control what those pointers point too. And still others overwrite heap metadata that will cause future allocation calls to allocate data in the wrong place, perhaps overwriting a function!

Suppose the following code had to be run to configure an application with a secret key. Since it writes to /etc you can only run this code as ‘root’. Suppose you really wanted to see the secret key but you don’t have root access. You can use a heap-based overflow when you provide the input filename to overwrite the heap space that contains the directory to write too (like your home directory!).

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main(int argc, char **argv) {
	char *file_name, *dir;
	file_name = (char *) malloc(8);
	dir = (char *) malloc(128);
	// Get the filename from the cmdline
	strcpy(dir, "/etc/conf/key/");
	strcpy(file_name, argv[1]);
	strcat(dir, file_name);
	FILE *fd = fopen(dir, "w");
	if (fd == NULL) {
		fprintf(stderr, "File open error.");
	fputs("zvsda34", fd);
	return 0;

Learning about rootkits

Rootkits are fascinating to me. When classes are over, I’m hoping I can find some time to read more about them. I have yet to find a computer with a rootkit on it until recently. My brother had a rootkit that would intercept web requests from any browser and redirect them to ad pages. It was done in a random manner so you didn’t recognize it at first; you think you clicked on the wrong link. Pretty sneaky! This book appears to be the best book out there to learn about rootkits on Windows (the more popular target)


“It’s imperative that everybody working in the field of cyber-security read this book to understand the growing threat of rootkits.”
–Mark Russinovich, editor, Windows IT Pro / Windows & .NET Magazine

“This material is not only up-to-date, it defines up-to-date. It is truly cutting-edge. As the only book on the subject, Rootkits will be of interest to any Windows security researcher or security programmer. It’s detailed, well researched and the technical information is excellent. The level of technical detail, research, and time invested in developing relevant examples is impressive. In one word: Outstanding.”
–Tony Bautts, Security Consultant; CEO, Xtivix, Inc.

“This book is an essential read for anyone responsible for Windows security. Security professionals, Windows system administrators, and programmers in general will want to understand the techniques used by rootkit authors. At a time when many IT and security professionals are still worrying about the latest e-mail virus or how to get all of this month’s security patches installed, Mr. Hoglund and Mr. Butler open your eyes to some of the most stealthy and significant threats to the Windows operating system. Only by understanding these offensive techniques can you properly defend the networks and systems for which you are responsible.”
–Jennifer Kolde, Security Consultant, Author, and Instructor

“What’s worse than being owned? Not knowing it. Find out what it means to be owned by reading Hoglund and Butler’s first-of-a-kind book on rootkits. At the apex the malicious hacker toolset–which includes decompilers, disassemblers, fault-injection engines, kernel debuggers, payload collections, coverage tools, and flow analysis tools–is the rootkit. Beginning where Exploiting Software left off, this book shows how attackers hide in plain sight.

“Rootkits are extremely powerful and are the next wave of attack technology. Like other types of malicious code, rootkits thrive on stealthiness. They hide away from standard system observers, employing hooks, trampolines, and patches to get their work done. Sophisticated rootkits run in such a way that other programs that usually monitor machine behavior can’t easily detect them. A rootkit thus provides insider access only to people who know that it is running and available to accept commands. Kernel rootkits can hide files and running processes to provide a backdoor into the target machine.

“Understanding the ultimate attacker’s tool provides an important motivator for those of us trying to defend systems. No authors are better suited to give you a detailed hands-on understanding of rootkits than Hoglund and Butler. Better to own this book than to be owned.”
–Gary McGraw, Ph.D., CTO, Cigital, coauthor of Exploiting Software (2004) and Building Secure Software (2002), both from Addison-Wesley

“Greg and Jamie are unquestionably the go-to experts when it comes to subverting the Windows API and creating rootkits. These two masters come together to pierce the veil of mystery surrounding rootkits, bringing this information out of the shadows. Anyone even remotely interested in security for Windows systems, including forensic analysis, should include this book very high on their must-read list.”
–Harlan Carvey, author of Windows Forensics and Incident Recovery (Addison-Wesley, 2005)

Rootkits are the ultimate backdoor, giving hackers ongoing and virtually undetectable access to the systems they exploit. Now, two of the world’s leading experts have written the first comprehensive guide to rootkits: what they are, how they work, how to build them, and how to detect them.’s Greg Hoglund and James Butler created and teach Black Hat’s legendary course in rootkits. In this book, they reveal never-before-told offensive aspects of rootkit technology–learn how attackers can get in and stay in for years, without detection.

Hoglund and Butler show exactly how to subvert the Windows XP and Windows 2000 kernels, teaching concepts that are easily applied to virtually any modern operating system, from Windows Server 2003 to Linux and UNIX. They teach rootkit programming techniques that can be used for a wide range of software, from white hat security tools to operating system drivers and debuggers.

After reading this book, readers will be able to

  • Understand the role of rootkits in remote command/control and software eavesdropping
  • Build kernel rootkits that can make processes, files, and directories invisible
  • Master key rootkit programming techniques, including hooking, runtime patching, and directly manipulating kernel objects
  • Work with layered drivers to implement keyboard sniffers and file filters
  • Detect rootkits and build host-based intrusion prevention software that resists rootkit attacks


Examining x86_64 Memory with GDB

The Art of Exploitation book has a nice section on using GDB to inspect memory. We start by disassembling the simple Hello World post.
Start by compiling hello_world.c with the ‘-g’ option.
-g Produce debugging information in the operating system's native format (stabs, COFF , XCOFF , or DWARF 2). GDB can work with this debugging information.
By default, when you disassemble code in GDB, it uses the AT&T format for the assembly instructions. The Art of Exploitation book prefers to use the Intel format. I’m not familiar with either but in case you want to follow along with the book, I’ll use the Intel syntax in my examples.
Since the book was written, GDB has changed the way you set the syntax format so you’ll want to modify it. The book says to do
set dis intel
You’ll want to do
set disassembly-flavor intel
You can create a file in your home directory called .gdbinit so you don’t have to specify this every time you start GDB. You can do all of this in one command
nobody@nobody:~$ echo "set disassembly-flavor intel" > ~/.gdbinit
Now lets fire up GDB. The book uses the ‘-q’ option which is optional
-q ``Quiet''.  Do not print the  introductory  and  copyright  messages.  These messages are also suppressed in batch mode.
nobody@nobody:$ gcc -g hello_world.c
nobody@nobody:$ gdb -q ./a.out 
Reading symbols from a.out...done.
Now that we have GDB up, lets use it to disassemble the x86_64 code. You can type the full version ‘disassemble main’ or the short version ‘disass main’
(gdb) disass main
Dump of assembler code for function main:
   0x0000000000400524 <+0>:	push   rbp
   0x0000000000400525 <+1>:	mov    rbp,rsp
   0x0000000000400528 <+4>:	mov    edi,0x40062c
   0x000000000040052d <+9>:	call   0x400418 <puts@plt>
   0x0000000000400532 <+14>:	mov    eax,0x0
   0x0000000000400537 <+19>:	leave  
   0x0000000000400538 <+20>:	ret    
End of assembler dump.
If you’re following along in the book, you’ll notice that the output is slightly different. The book uses the older x86 architecture, I’m using the x86_64 architecture. That means all of the 32-bit ‘E’ registers now have their equivalent 64-bit ‘R’ registers. For example, ‘esp’ is the 32-bit version of the “Extended Stack Pointer” and ‘rsp’ is the 64-bit version of the “Register Stack Pointer”.
I’ve highlighted the important part of the code, the body of main. Lets use GDB’s examine memory commands to see exactly what’s going on
GDB uses the ‘x’ command to look at the contents of memory. You can tell GDB what format you’d like to see the memory in and unit you’d like to see it in
o - octal
d - decimal
x - hexadecimal
u - unsigned integer
s - string
t - binary
b - byte
h - half
w - word
g - double word
You can also tell GDB how many you’d like to see by prefixing it’s type with a number. Since we know that “Hello World” is a string, we can use the x/s command to view the contents of memory where we think that string is stored.
(gdb) x/s 0x40062c
0x40062c:	 "Hello World!"
We can also look at the equivalent ASCII characters of “Hello World” in memory
(gdb) x/12db 0x40062c
0x40062c:	72	101	108	108	111	32	87	111
0x400634:	114	108	100	33
An interesting thing about GCC is that it automatically optimizes printf() with 1 argument to use puts(). This confused me at first since the code used printf() but the disassembly shows a call to puts()
   0x000000000040052d <+9>:	call   0x400418 <puts@plt>
The puts() function automatically adds a new line, that’s why when we examine memory in GDB, we don’t see the ‘n’ character that we added when we called printf() in the code
More to come; we’re just getting started!


I’m pretty comfortable with almost all operating systems. I like to use my MacBook Pro for school so this where I’ll be doing the work.

Hacking the Art of Exploitation comes with a LiveCD based on Ubuntu. All of the examples in the book are therefore targeted towards Linux. I opted to install the latest version of Ubuntu (10.04 at this time) using VirtualBox on Mac.

Go to Top