In-case you missed is, this is Part Two of our series ongoing from understanding the basics of exploitation to getting into real-world targets.

Click Here for Part One, an overview of our thoughts.

You can also check out the discussion video on Youtube where Specter and I discuss these things in a bit more depth.

Once again I want to preface this by saying its out opinion, and there definitely are some other thoughts on it. Follow what excites you.

Why Vulnerability Research?

It might seem a little odd to say that to get better at exploit development you should learn vulnerability research. During the last post I referred to gaining an intuitive understanding of vulnerabilities and primitives. Spending time to understand the discovery of vulnerabilities and the various ways they can be introduced feeds into giving you that deeper understanding and metal model of vulnerabilities that is essential for more complicated exploitation.

Technically, you’d could just skip vulnerability research and only do exploit development, there are some teams where you can specialize like that. I think at least understanding of what is going on will help you in the long run even if you don’t plan to remain on the research side. Most people end up doing this solo at least at first, it’s a skill that takes a long time to develop so get started sooner.

What is Vulnerability Research?

Vulnerability research is a huge, actively changing field. We can break it down into several subfields but in terms of a large divide, I’d break it down into Static analysis and Dynamic Analysis with Fuzzing being a sort of mix of the two. There are problems with the breakdown, but it serves our purpose.

Auditing (Static Analysis)

When someone mentions auditing, the first thing that may come to mind is someone with a checklist of questions to answer. That is not at all what I mean. There is a place for that type of auditing in security but all I’m referring to here is reviewing code (including disassembly) for vulnerabilities. This can further divided into manual and automated options and white/gray/black box styles.

It is the manual auditing that turns up some of the deeper, more subtle bugs that are harder for automated methods to notice. The more you understand about the architecture of your target and how everything fits together, the more you can spot the subtle issues that automated options wouldn’t discover.

Automated scanners, and the research going into them is useful, but not as a starting place for learning. In general, while learning to use various tools can give you results, it is the manual work that forces you to grapple with vulnerabilities and gain a better understanding. The process is important, not the quick results. We’ll dive more into the methodology later.

Dynamic Analysis and Fuzzing

The easiest and most familiar form of bug hunting is a dynamic analysis setup, where you open the application and start feeding it input. It's probably one of the first things you’d do with a CTF binary, connect/launch it and try it out. You are basically just observing software while its running to find vulnerabilities.

Fuzzing, at its core, is an automated form of dynamic analysis that's augmented by drawing on both manual and static analysis. Much of the movement in dynamic analysis research comes from fuzzing research and trying to improve fuzzing related workflows: either improving crash triaging, improving bug detection with sanitizers, providing more information to the fuzzer with instrumentation, or hooking.

Why not learn to fuzz? There is a lot of advice out there that recommends fuzzing as an easy entry point to vulnerability research. That is true, it is an easy option, but it doesn’t really require you to understand the vulnerabilities, you learn the tool, run it and then get your crash.

Once you have the crash, either you’ll hope for an easy bug or you’ll need to understand some degree of the auditing aspect to do triaging. It’s also important to keep the goal in mind here, the goal is to gain a deeper understanding of the vulnerabilities. You already understand the exploitation side, so having a program find the vulnerabilities for you so you can go right back to exploiting them isn’t really adding to that skill.

That said, fuzzing is insanely useful, and unreasonably effective at finding some types of bugs. It should be part of your toolkit.

First Step - Manual Auditing and Manual Testing

The purpose here is to learn about vulnerabilities. The most valuable approach is to dive in and explore the application and its source manually to find issues.

Choosing a First Project

• Interesting - Choose something interesting, a technology you’re interested in, a project you support, just something that is going to hold your interest.

• Open Source - Start off with something open-source. The black-box approach is just adding the extra step of reverse engineering onto the same process, focus on one skill at a time.

• Not Picked Over - Its probably tempting to just pick out a harder target and grind it out. If your motivation dictates that then it can be an option but getting some quick wins is nice for motivation also. Starting off with a target that hasn’t already been picked over gives you a chance to refine your own methodology and actually have some results. In something that’s picked over you might do things right and still not have many findings

Avoid Tips and Tricks

This ties into the reason why we suggest learning manual auditing in the first place. The tips and tricks have their place, but they provide you a shallow shortcut to finding some issues giving you the end-result without the understanding. They are great to add once you understand the core issues though and can save you time.

While you’re learning, I think it’s important to do things the hard way in a sense. You have the time available to make sure you really understand what’s going on and don’t have any time pressure to have results. It’ll pay off in the end.

Manual Methodology

While we are laying out one approach, Specter and I are both fans of The Laymen’s Guide to Zero-Day Engineering which introduces some more practical advice for getting up to speed on a hard-target. There is some valuable insight, but its the approach I’d only recommend once you have some foundation.

Step One - Environment Setup

Compile and run the code. First step is to be able to build and run your code-base. It’s not just because you need to run it to test the target, but you’ll also have to learn about the build system and any assumptions that are made when configuring. It will also be helpful for if you want to instrument anything in the future.

Once you can build, make sure you can build with debugging information, and run the program under a debugger. Unfortunately, running under a debugger isn’t always an option, but take the time to set up a debugging environment somehow even if its lighter-weight tracing or something. Try to solve some of the common problems upfront instead of when you need it.

Step Two - Exploration

Time spent in reconnaissance is seldom wasted

You might be excited to just go and find all the vulnerabilities, but hold off. Take the time to understand your target application, the big picture and the details. The goal at this stage is to start to intuitively understand how this software works. Don’t set-out to memorize the details, but play and explore the application until it becomes intuitive to you.

• Use the application

• Try to invoke new or non-obvious functionality

• Read available documentation (is it actually correct?)

• Understand the different components

• How data flows around components

Take some time to script up some interactions, programmatically doing things helps make sure you understand everything. I’ve sent many year as as a consultant and on an actual assessment I can spend around 20% of my total time just on this stage. It delays actual findings, but it pays off in the end and allows you to find more subtle bugs.

Step Two (b) - Targeting

This won’t be relevant for all targets, but when you have a larger application, you might need to break down your actual focus to a particular component or functionality and do your VR piece by piece. You can make this decision once you’ve actually understood the system.

As first this might be a bit of a shot in the dark, but over time you’ll gain a bit of an intuition and you’ll havea some hunches about what components might be bug farms or not.

Step Three - Auditing** ...for real this time.

Now you’re all setup to do some mixed analysis. Debugging and Source go hand-in-hand so use both if you have both. Ultimately, this is entirely application specific but there are a few skills you do need to learn and should practice.

• Going from source to in-app functionality. You’ll run into some questionable code, maybe it looks vulnerable, maybe its just odd, maybe you’re not sure what exactly it’s doing. Either way, you’ll need to figure out how to reach that code in the application. This isn’t always straightforward. Take some time if you run into some code where you’re not sure how to hit it, just try it. Set a breakpoint and try to hit it. As with most things here, its about getting practice, so create situation to do it.

• Going from in-app functionality to source. On the flip side, sometimes you’ll notice the application doesn’t perform quite how you expect and you’ll want to track down the source. Same deal as before, you need to get practice so find situations and do it. It forces you to understand the code-base.

• Tracing from source to sink. Practice doing this in the debugging and just by reading the code. Find some entry point and trace it through to a place it’s used.

Step Three (b) Auditing Strategy

Plenty of theory about how to approach this - too much to cover here. Instead I recommend the book Art of Software Security Assessment. Despite being from 2006 it is still very relevant, especially when it comes to code auditing strategies. It lays out the theory, different approaches to auditing and navigating and the pros/cons for the strategies. It also goes into how to prepare for an assessment, and covers some of the basic vulnerabilities in C. Though there is more to talk about that on that topic. I’ve never run into anything better and honestly, probably couldn’t do better myself. The book itself isn’t in-print anymore but the Kindle version is available.

Though I do have some things I’d want emphasis here.

Be systematic. Actually have an attempt at a plan, doesn’t really matter if you follow it, but the act of planning I think is essential. A test plan is also just a useful augment to your memory. There are a lot of weird random quirks you might not remember to test for, having something to reference will help remind you. Also being systematic in how you look for issues to make sure you’re not missing an area.

Do things your own way. There are different thoughts on the “big-picture” approach but ultimately you need to find what you’re comfortable with, and kinda create your own system. Everyone does things their own way, don’t be afraid to deviate from known methodologies, see what works for you. Just don’t cut corners.

Take notes. The book has an emphasis on this as well. Notes are important in part to be a reference when you forget but also to force you to put your thoughts into words. It is similar to the development concept of rubber duck debugging.

Take notes of your understanding of a system/component/flow whatever, if its wrong, take notes of the correction. Seeing how your own understanding grows can help you spot mistakes devs might have made. Once you have what you’d consider a full understanding, then go back and summarize for your future reference.

Note anything that is weird or non-standard. At first you might not have much other code to compare with but overtime you’ll get a feel for the common way to do certain things. So when code deviates from the expected solution you should look into why. You might also note when a particular area of code deviated from how the rest of the application does something. Take notes of this.

Really, just take notes of anything that stands out to you.

Fuzzing and Beyond

I ranted briefly against fuzzing early on, but it is insanely useful. You should take some time to learn to use some of the common fuzzers, and understand some of the theory behind them.

I’ll often run a fuzzer while doing the manual work. I’d get the fuzzer running early on when I get the debugging setup working. Then as I explore the application, I’m able to feed the fuzzer more specific targeting information (corpus, structure information, etc).

There are a lot of subfields that you could dive into if they interest you, but try not to fall into the academic traps. Tons of papers out there are suggesting novel approaches, and its easy to get sucked into trying the latest and greatest research. More often than not, they are only useful in a very specific use case. Research is just that, research, and it will often take more work before its practical.

And with that I wish you luck in finding some new vulnerabilities. We'll be back tomorrow with a post on developing your exploit dev skills. It will focus on the skills needed to approach exploitation in modern monolithic targets, and how it differs from CTF style work.