The family runs a modded Minecraft server off the home server, a big modpack on an older version of Forge, and for a while it just worked, until people started getting kicked with out of memory errors on their own computers, not the server, the client. Sometimes it was ten minutes in, sometimes an hour or two, but it always ended the same way, a wall of red text about not being able to reserve memory.
My first instinct was that it was just too many mods, which is the usual answer for almost any modded Minecraft problem, so we went down the suspect list one at a time.
The mod that renders far away terrain got blamed first, since it is the hungriest thing running by design, but a live memory reading during a crash showed it barely moved. A minimap mod with its own image cache came next, and directly measuring it came back to a few megabytes, nothing close to the scale of the problem.
The usual suspects, cleared one by one
A compatibility layer that lets mods built for a different mod loader run on ours looked promising too, since the crash logs showed leaks attached to mods with nothing else in common, but reading its code cleared it, since it only touches the handshake when a player connects, never the part where packets get delivered.
A couple of mods that showed up constantly in the crash evidence looked like the obvious cause for a while, since both were unusually chatty over the network. Counted again though, they were not leaking memory, they just sent the most packets, so the real bug showed up on their packets more often than anyone else's.
Reading the evidence instead of guessing
Guessing stopped being useful, so we built a small tool to watch the part of memory where network buffers live, sampling it every few seconds during real play, and every join and leave cycle added a couple hundred megabytes that never came back down. Forcing a garbage collection freed only a small slice, which meant something was still holding onto that memory.
A full memory dump at the worst point, run through an analysis tool, found one real, separate bug, a mod that kept old player data around after logout, small and easy to fix but nowhere near big enough to explain the scale we were seeing. The bulk of the leaked memory was not even on the visible heap. Well, that is a big part of why it took this long to find, since most tools point you straight at the heap and stop there. Counting buffers directly showed over a thousand of the same object, the buffer Minecraft uses to hold an incoming network packet.
Reading Forge itself
At that point the only way forward was to stop looking at mods and read what Forge itself does with a packet after it arrives, so we turned on a debug flag that dumps the exact code Forge runs once every mod has modified it, and compared that against Forge's own published source.
Forge makes a defensive copy of every packet sent between a mod and the game, presumably so the original network buffer can be reused while a handler still reads the copy. That copy comes from a pooled area of memory, and pooled memory only comes back if something explicitly releases it, and as far as I can tell, nothing in Forge's delivery path ever did. We searched the whole section responsible and found exactly one release call, on the original buffer, never the copy, so every packet any mod sends over that path leaks its copy for good. An idle menu does not leak, since nothing is sending packets, but any real play session keeps feeding it. There was even an old issue on Forge's own tracker describing this same symptom on someone else's pack years earlier, closed without anyone tracing it to a missing release.
The fix, and shipping it
The fix itself was small, a single hook that runs right after Forge finishes handling a packet and releases the copy if nothing already has, and we built it as a standalone mod at home and measured it the same way we measured the problem.
Before the fix, that memory pool climbed by roughly seventy megabytes a minute, whether someone was doing anything or just standing still, since packets never stop moving during normal play, and after the fix the same measurement sat close to flat, drifting by less than a tenth of a megabyte a minute. The join and leave test that used to permanently add a couple hundred megabytes per cycle dropped to almost nothing, and then came an actual family evening with the fix installed, a few hours of normal play, and nobody got kicked once.
I posted the writeup to Forge's own issue tracker with the code citations and measurements attached, and one of Forge's founders replied within a day, unsure at first whether it was a bug or working as intended, with good follow up questions. I may be missing something he can see that I cannot, but it was a good way for it to end up.
We had to clear every mod off the list before we thought to look at Forge itself.