Ah, now this is a problem that I run into regularly. Here's the deal: it's not a bug and it's not a memory leak. Almost all communication in jACT-R is asynchronous. This allows the model to run instrumented with (relatively) little performance impact. The problem arises when the model is actually running too fast (and producing too much data) for the IDE to keep up with. When this happens, network buffers rapidly start filling up and memory is consumed at an alarming rate. The two process then start butting against the resource limits and performance crawls to a halt - potentially with out of memory errors.
This problem most often occurs when persistent execution is on and cycle skipping is off. Basically, your model is plowing through a lot of worthless cycles and generating a bunch of data that is worthless. The IDE chokes on the volume and kaboom.
Your best bet fix is to enable the SynchronizationManager instrument in the run configuration. This will force message synchronization at a periodic (and configurable) rate.