I do this quite a lot myself.

Beside the games that do officially support it via modding APIs, or older games that are just naturally extensible like Unreal Engine games that ship with an editor; it is usually done by the way of code injection.

Think of code injection like a manually placed interrupt - you modify the flow of execution at some point to execute your own code.

For this, you need something known as an entrypoint - place where mod code begins executing. Usually, at the entrypoint, the mod code bootstraps itself further.

To obtain an entrypoint, you have 2 options:

1. On supported OSs, you can abuse dynamic library loading and make a library wrapper which masquerades as the original library. When its execution begins (on Windows this is usually very early in the importing stage, on Unix/Linux you need to swap functions), you can bootstrap your mod(s) further.
2. On other systems (game consoles, embedded, etc), you have to modify the game executable directly. Either via the way of memory injection of another tool or direct binary modifications. Depending on the platform, if it supports loading of libraries, you would either load your mod as a library or, if not, inject the mod elsewhere into memory (slack/unused space).

How do you figure all of this mumbo-jumbo out? Well, you use tools designed to disassemble the game binary into machine language. Most popular examples are IDA, Ghidra and Binary Ninja.

You usually won’t have any information on what’s what, but if you know a bit about code design of games, you come to realize that a lot of them follow similar principles/patterns (depending on the era and platform), so you can probably figure it out once you’ve seen a few games. Especially if the game is using the same game engines as others, then you can cross reference stuff.

However, if you’re lucky, you can obtain something known as debugging symbols which have all of the information about the generated binary. Ideally, it’d be for your executable that you’re working with directly, but if it’s not, you can usually just cross reference it by either binary pattern matching or simple deduction (same execution flows, same data, etc). This will give you a nice map of where’s what in your disassembler tool.

Thanks to standardization of binaries, ABIs (Application Binary Interfaces) usually allows you to match the types as they’re defined originally and have them work directly in your bespoke code. This means, for example, that if a function takes 3 arguments, those 3 arguments will be passed in the same way (depending on its calling convention), so you can expect to receive them in your own code the same way. (This is what we call code reflection - you can reflect the game’s types in your own code. If you’re using the same compiler as the original, even better!)

And, ideally, you would be working with code directly and not with machine language, but some lower level stuff will need it inevitably.

Usually, in your mod source code, you would work with a library that is specifically designed for code injection. This is effectively a JIT compiler which emits machine code instructions which are needed to redirect the flow of execution to your own modded functions. (Usually is capable of memory editing and making branch instructions)

There are 2 ways to perform a code modification:

1. Inlined code - this means you write your own code in the middle of another game function. Usually, this is done via lower level assembly and a branch instruction to put it in. Requires decent knowledge of the target CPU to use it (to not corrupt any stack or registers)
2. High-level function hooks - this usually means that you interrupt a function call/branch with your own equivalent. Your equivalent function will have to do the job of the original usually, but depending on the case, you may replace it outright or simply pass the call onto the original function. Thanks to ABIs, this is possible.

With all of this being said, you need to have the knowledge being able to read machine language of CPUs. This isn’t too difficult once you realize RISC CPUs are mostly very similar at a glance which helps out a lot.

But, in order to be able to read it, you have to first obtain it and, as we all know, DRMs, executable packers and anti-cheats/tampers protect this, so you may need to get your hands dirty a bit to bypass that (e.g. memory dumps). A lot of the same tricks that security researchers use in their rulebook apply to modding as well.

I hope I’ve explained enough. I’m aware I used a bunch of terms that may be foreign, so I hope you will get the gist at the very least.