Can one of the experts commenting here let me know a good process to get started with learning to decompile/reverse engineer/modify Windows programs?
Is it worth doing any of those online ctf thingys? Any recommended resources or books I should check out?
Please don’t take this the wrong way, but googling is a very important skill to have. There’s quite a bit of information out there and I’ll basically just repeat what you can find online already. You’re additionally not very specific about your goals, which makes it hard to give specific advice.
Practical Malware Analysis is a really great book, even if you don’t intend to reverse malware. It has labs after every chapter, along with solutions. It teaches you about windows, how to approach reversing a program and what to focus on to not waste too much time.
Once you worked through that CTFs should be much more approachable.
Reverse Engineering: Secrets of Reverse Engineering and Practical Reverse Engineering are also often mentioned as good books, but I personally haven’t read them. Aside from that, the books from nostarch.com are great.
If you want to learn more about windows itself, there are the “windows internals” books.
It’s also a good idea to find a community with similar interests, learning together is often more fun. And when you ask for help try to properly state your problem and what approaches you already tried to solve it, along with their outcomes.
Good luck!
>There’s quite a bit of information out there and I’ll basically just repeat what you can find online already.
Yeah, too much information. Maybe I'm just not finding the precise search words to find what I want.
Your reply was very helpful so thanks for that.
>You’re additionally not very specific about your goals
I'd like to be able to do very basic audits of how programs work and precisely what they're doing, and also modify their behavior in simple ways (change hardcoded servers, change program execution flow, disable or enable certain functionality). Reverse engineering with basic modifications.
I'd also like to have a go at modifying firmware of embedded devices, eg a car head unit with a stupid splash screen on startup that I'd like to disable.
For example I have a camera with wifi functionality, and you can control it from a phone app. It would be cool if I could somehow control that from the command line, or create my own basic app for the PC to control it. On top of that it would be good if I could modify the firmware to make it automatically start up in that remote control mode.
Mostly just for adding to my box of tools and general understanding though.
The listed books should help with that! I think Practical Malware Analysis really is a good start, even for that. When looking at malware you also try to find important functionality and the book teaches you how to get there quickly.
> I'd also like to have a go at modifying firmware of embedded devices, eg a car head unit with a stupid splash screen on startup that I'd like to disable.
There are probably forums for that, where you can find some info. There's also a nostarch book on car hacking (https://nostarch.com/carhacking) and for embedded (https://nostarch.com/hardwarehacking). I haven't read either, but it might be worth to check out some reviews for it. Maybe they're good and can help you achieve your goals.
The difficulty probably depends on how new the car is. In case you need to actually find a vulnerability in the infotainment system to get proper access, something like Hacking: The Art of Exploitation (https://nostarch.com/hacking2.htm), which I heard many good things about, is probably a good read. But if it's too new it might be too time consuming (see e.g. https://www.youtube.com/watch?v=k_F4wHc4h6k)
> For example I have a camera with wifi functionality, and you can control it from a phone app. It would be cool if I could somehow control that from the command line, or create my own basic app for the PC to control it. On top of that it would be good if I could modify the firmware to make it automatically start up in that remote control mode.
I think getting at least a CLI client could be relatively easy. For that you probably don't even need to reverse a binary using IDA/Ghidra/Binary Ninja. You can try to get the .apk file of the app and decompile it using something like jadx (https://github.com/skylot/jadx/). You'll receive mostly readable Java code. It can try to deobfuscate names, if they're obfuscated. The code you're interested in is probably somewhere under "com.manufacturer...".
How easy it is to modify the firmware once again likely depends on how old or new the camera is. They could, for example, have some integrity checks that keep you from doing that. But I have absolutely zero experience here, so it might as well be really easy. I think there could be forums for this, too.
Also be warned that modifying the firmware of your car or camera can break (parts of) them if things go wrong. E.g. I accidentally (soft) bricked a device because I tried to flash it from within a VM. I don't know how big the risk in your cases is, maybe there isn't any. But it's a good idea to read lots before accidentally breaking something expensive!
Happy I could help you :)