The exploit code in Hacking: The Art of Exploitation is targeted towards overwriting 32 bit return addresses (size 4). It uses code like this:

unsigned int ret;
ret = &var;
*((unsigned int *)(buffer+sizeof(unsigned int))) = ret;

I’m working on a 64 bit machine and thus my addresses are double in size. I’m guessing this means that the above code won’t work on my machine (I’m not sure?). I searched around and found an “unsigned integer pointer type” which is defined per architecture and it is indeed size 8 on my 64 bit machine. I changed the code to this:

uintptr_t ret;
ret = &var;
*((uintptr_t *)(buffer+sizeof(uintptr_t)) = ret;

I thought that this was the reason I was having trouble spawning a shell from shellcode but after changing the code it still doesn’t spawn a shell.

Update
Not only do I think the return addresses are going to be different between 64 bit and 32 bit but they’ve totally changed the syscall numbers in “unistd.h”. That means that this shellcode isn’t going to work at all on 64-bit because it assumes ‘execve’ is 11.

; execve(const char *filename, char *const argv [], char *const envp[])
  push BYTE 11      ; push 11 to the stack
  pop eax           ; pop dword of 11 into eax
  ....
  int 0x80          ; execve("/bin//sh", ["/bin//sh", NULL], [NULL])

On 64 bit, 11 is ‘munmap’

#define __NR_munmap                             11

Looks like I’ll definitely be installing a 32 bit version of Linux in order to follow along with the examples, 64 bit is going to be too challenging while I’m in the learning stage.