macOS Infinite Retain ... possibly?

[SecuritySteve]

Hello everyone, I have been stumped by a programming issue, and would really appreciate any input from a veteran.

Specifically, I believe I have a memory retention issue. My application should be able to run for days on end, but there is a memory build up that eventually causes the application to crash, potentially costing me the results. My program structure is the following:

View Controller, which acts as the main GUI controller and handles all interactions on the GUI from the user. It also possesses objects that control the flow of my program. Those objects have child objects that handle individual iterations of my program's business logic. For the sake of clarity:

ViewController shall be object A.

SubController shall be object B, child of A.

Worker shall be object C, child of B.

Each of these levels has interactions with a single item on the GUI, a console that updates the status of the process and reports any items of note found during the business logic's runtime. Objects C and B both update the console via asynchronous dispatch calls to the main thread. I'm not sure if this is relevant to the problem or not, but I wanted to be clear that all objects have a reference to this GUI item, the reference for which is passed as a parameter to the children.

My application's memory pressure looks like that. The spikey and gradually rising part is during the process where the workers are doing their task, and the flat part is after the iteration set is complete and the workers (and their sub controllers) are released. This is only after a few seconds of running, and over the span of days will eventually fill the system's memory. In my mind, the application should return back to it's original memory pressure when the Worker C is released after it's iteration is complete. However, obviously memory is being left over. When I transfer my session to Instruments, no memory leaks are detected. My application uses ARC to manage memory.

It should probably also be noted that my SubControllers B spawn multiple threads to run multiple Worker objects C at the same time. I'm not sure if multithreading might be interfering with ARC, but it was worth noting. I'll keep an eye on this thread to answer any follow up questions to the best of my ability.

 

[chown33]

1. What do you mean by "child" in "child of A" or "child of B"? Does it mean "creates the object" or "holds a reference to the object", or both? If it's holding a reference to the object (i.e. the object has an instance variable that holds an object reference (directly or indirectly)), then that reference will prevent ARC from reclaiming the object when the task is complete. This is significant because you mention threads. In general, if some object X creates an object Y that will be handed off to a thread T, then X should not hold any references to either the Y object or the T thread. As a design, I prefer to give a thread the responsibility for creating all objects it needs, rather than having some object X create objects that are then handed off to threads.

2. Exactly what causes "... Worker C is released after it's iteration is complete." If you're using ARC, you generally shouldn't be doing overt releases. If the "iteration" is a thread, then when the thread terminates all its objects should be automatically released. If the thread isn't terminating, then it probably should be (you should be able to inspect how many threads the process has, and whether that rises and falls in the way you expect). If your strategy is to keep threads around to perform more work (e.g. they queue up to await new work objects), then you probably need to coordinate that reuse with any object creation.

3. How did you generate the graph?

4. Written in Objective-C, right? OS version? Xcode version?

 

[SecuritySteve]

1. What do you mean by "child" in "child of A" or "child of B"? Does it mean "creates the object" or "holds a reference to the object", or both? If it's holding a reference to the object (i.e. the object has an instance variable that holds an object reference (directly or indirectly)), then that reference will prevent ARC from reclaiming the object when the task is complete. This is significant because you mention threads. In general, if some object X creates an object Y that will be handed off to a thread T, then X should not hold any references to either the Y object or the T thread. As a design, I prefer to give a thread the responsibility for creating all objects it needs, rather than having some object X create objects that are then handed off to threads.

2. Exactly what causes "... Worker C is released after it's iteration is complete." If you're using ARC, you generally shouldn't be doing overt releases. If the "iteration" is a thread, then when the thread terminates all its objects should be automatically released. If the thread isn't terminating, then it probably should be (you should be able to inspect how many threads the process has, and whether that rises and falls in the way you expect). If your strategy is to keep threads around to perform more work (e.g. they queue up to await new work objects), then you probably need to coordinate that reuse with any object creation.

3. How did you generate the graph?

4. Written in Objective-C, right? OS version? Xcode version?

I should also mention that I have a custom object that simply holds the GUI options that is passed as a parameter. Consider that object D.

1) Allow me to post some relevant code:

Within ViewController.swift:

@IBAction func Button_Pressed(_ sender: Any)
{
                  //Options parsed above, creating object to hold options
                  let param = Params(...)
                  let sub = SubController()
                  sub.SubKickOff(param: param)
}

Within SubController.swift:

@objc func SubKickOff(param: Params)
{

            let thread = Thread(target:self, selector:#selector(SubManager), object:param)
            thread.start()
}
@objc func SubManager(param: Params)
{
            let Console = param.Console // lets the Console window from my GUI be referenced
            autoreleasepool{
                  var i: IntegerLiteralType = 0
                  while i < param.iterationCount
                  {
                        let queue = DispatchQueue.global()
                        queue.sync(){
                              self.LaunchWorker(param:param,i:i)
                        }
                        i += 1
                  }
                  DispatchQueue.main.async() {
                        Console?.textStorage?.append(NSAttributedString(...))
                        Console?.scrollLineDown(nil)
                  }

            }

}
@objc func LaunchWorker(param: Params, i: IntegerLiteralType)
{
            autoreleasepool{
                  let Console = param.Console
                   // More stuff breaking down the parameters into Objective-C types
                  let myWorker = WorkerA()
                  let myWorker2 = WorkerB()//This guy is optional, and occasionally is created within WorkerA's task.
                  //Depending on the options, do some stuff with WorkerA() and potentially WorkerB()
                  DispatchQueue.main.async() {
                        Console?.textStorage?.append(NSAttributedString(...))
                        Console?.scrollLineDown(nil)
                  }
            }
}

2) The workers should be released by the autoreleasepool after LaunchWorker has finished and popped off the stack.

3) The graph is from Xcode, when you click on the sandwich looking icon on the left bar during runtime you can see the resource usage from your application.

4) The ViewController and SubController are written in Swift. The Workers on the other hand are written in Objective-C. The reason being that the top level stuff I wanted optimized with Apple's Swift API, but Objective-C is better at dealing with more primitive datatypes, and with file manipulation / creation and creating pipes to external applications, which is exactly what my Workers are doing. Xcode 9.3.1, MacOS 10.13.4

 

[chown33]

1. What debugger are you using?

2. I'd break the problem into two parts: the starting part and the working part, then I'd make stubs (do-nothing placeholders) for the working part. The starting part still does all the stuff you show now: passing Params around, making threads, putting things on dispatch queue, etc. The only difference is that instead of making real LaunchWorker objects, it makes and starts StubWorker objects. The StubWorker object should do nothing, as a minimum. They exist only so the starting part's funcs have an actual target object. When you run this stubbed-out version of the code, you should use the debugger to confirm that there are no threads left, no objects left, etc. at the end of a run. If you do have objects or threads left, then you know that the worker part's actions have no effect on the memory consumption problem. In other words, you've narrowed the problem down to the Swift code that creates, starts, dispatches, etc. the worker objects. At that point, you can dig into that code more deeply with the debugger, try additional tests, and so on.

3. Once the starting parts of the code are known working, you can replace the StubWorker with LaunchWorker and see if the memory problem reappears. If it does, then you use the debugger on the working part of the code, first breaking it down into at least 2 distinct sub-parts where you can again apply the "stub object" strategy (e.g. WorkerA and WorkerB could be stubs).


You may find that having a deinit func in either the StubWorker or LaunchWorker class gives you better visibility into their life-cycles. In particular, if deinit is never called, then you know the objects aren't being freed, so you can focus on tracking down who holds a reference that's preventing dealloc.

Two useful rules of thumb in debugging are Confirm Expectations, and Break It Down. To confirm expectations, one must have a clear idea of what is expected at each step or stage of execution. If you're not sure what to expect as a result, then the first step is to clarify that. In other words, after X happens, Y should exist, Z should not, and there should be Q bags of wool waiting on the master's doorstep down the lane.

 

[SecuritySteve]

1. What debugger are you using?

2. I'd break the problem into two parts: the starting part and the working part, then I'd make stubs (do-nothing placeholders) for the working part. The starting part still does all the stuff you show now: passing Params around, making threads, putting things on dispatch queue, etc. The only difference is that instead of making real LaunchWorker objects, it makes and starts StubWorker objects. The StubWorker object should do nothing, as a minimum. They exist only so the starting part's funcs have an actual target object. When you run this stubbed-out version of the code, you should use the debugger to confirm that there are no threads left, no objects left, etc. at the end of a run. If you do have objects or threads left, then you know that the worker part's actions have no effect on the memory consumption problem. In other words, you've narrowed the problem down to the Swift code that creates, starts, dispatches, etc. the worker objects. At that point, you can dig into that code more deeply with the debugger, try additional tests, and so on.

3. Once the starting parts of the code are known working, you can replace the StubWorker with LaunchWorker and see if the memory problem reappears. If it does, then you use the debugger on the working part of the code, first breaking it down into at least 2 distinct sub-parts where you can again apply the "stub object" strategy (e.g. WorkerA and WorkerB could be stubs).


You may find that having a deinit func in either the StubWorker or LaunchWorker class gives you better visibility into their life-cycles. In particular, if deinit is never called, then you know the objects aren't being freed, so you can focus on tracking down who holds a reference that's preventing dealloc.

Two useful rules of thumb in debugging are Confirm Expectations, and Break It Down. To confirm expectations, one must have a clear idea of what is expected at each step or stage of execution. If you're not sure what to expect as a result, then the first step is to clarify that. In other words, after X happens, Y should exist, Z should not, and there should be Q bags of wool waiting on the master's doorstep down the lane.

I will give this a shot. Thank you for your help. I'll update back when I've got some results.

Also I'm using Apple's default LLVM debugger.

 

[Krevnik]

Also consider running this through Instruments to see what is actually taking up the memory. It's hard to tell, but I wonder if your memory issues here are actually somewhere else. Especially if the workers are creating a bunch of text storage objects meant for use in the UI.

 

[SecuritySteve]

Also consider running this through Instruments to see what is actually taking up the memory. It's hard to tell, but I wonder if your memory issues here are actually somewhere else. Especially if the workers are creating a bunch of text storage objects meant for use in the UI.

When I have profiled in Instruments, the memory seems to be in all parts of my application. The Workers create pipes and file data, and I can clearly see those pipes in the memory even after it is done with all iterations. The only time the workers update the UI is if something interesting happens, like an external process crash or hang.

As a further update, after a set of iterations there are no left over threads. Just a big container of memory that doesn't seem to want to go down.

I created a base method stub that essentially just does the Console updating, and doesn't create any worker objects (in place of LaunchWorker). And to my surprise, the memory has the same build up issue. Clearly the memory that isn't being released is happening prior to the workers doing their bit.

Playing with this, I decided to run 1000 iterations with the base stub. The program starts at 27 MB of Memory used, and after 1000 iterations it ends up at around 28 MB. If I run the same number of iterations again, the memory does not grow to 29 MB, but remains at 28. If I run 100,000 iterations after, the memory consumption reaches 36.3 MB. Running 1000 iterations after the 100,000 increases the memory consumption to 37 MB. What the heck?

The only thing I can think of that isn't being released is the GUI itself? Perhaps the formatted strings in the text view are keeping objects alive?

 

[Krevnik]

The only thing I can think of that isn't being released is the GUI itself? Perhaps the formatted strings in the text view are keeping objects alive?

That's actually my thinking, and why I suggested Instruments. Obj-C types are all visible. Swift classes are also all visible (as Module.MyClass). Structs are messy because they are value types, so you will see stuff like _ContiguousArrayStorage<MyStruct> in the Allocation Summary of the Allocations instrument. I quickly wrote a small Swift app to double check all this (it just fires timers frequently, allocating memory and putting it in an array), and it holds up with XCode 9 at least.

The bit about appending text storage to the "Console" object, and scrolling a line down made me think that it may play a role. Without something to limit how many "lines" exist, you have unbounded memory requirements. And if that code is generating views, you will chew through memory a lot faster keeping those views around and in a scroll view or similar, than you will just keeping things like the attributed string around.

Table views and Collection views place upper bounds on how many cells they create for this reason. The backing data is usually small in comparison to the CGSurface backing a view/layer.

 

[SecuritySteve]

That's actually my thinking, and why I suggested Instruments. Obj-C types are all visible. Swift classes are also all visible (as Module.MyClass). Structs are messy because they are value types, so you will see stuff like _ContiguousArrayStorage<MyStruct> in the Allocation Summary of the Allocations instrument. I quickly wrote a small Swift app to double check all this (it just fires timers frequently, allocating memory and putting it in an array), and it holds up with XCode 9 at least.

The bit about appending text storage to the "Console" object, and scrolling a line down made me think that it may play a role. Without something to limit how many "lines" exist, you have unbounded memory requirements. And if that code is generating views, you will chew through memory a lot faster keeping those views around and in a scroll view or similar, than you will just keeping things like the attributed string around.

Table views and Collection views place upper bounds on how many cells they create for this reason. The backing data is usually small in comparison to the CGSurface backing a view/layer.

I think this could definitely be what I'm looking for. I'll think about what I can do, and post back here after I've tried a few ways of dumping the console contents.

 

[Senor Cuete]

The Xcode analyser is really good at finding problems like this. Have you run it on your project? Project -> Analyze.

 

[SecuritySteve]

The Xcode analyser is really good at finding problems like this. Have you run it on your project? Project -> Analyze.

Analyze found no issues, and only ran on my Objective-C files ... nice work Apple.

 

[Senor Cuete]

Analyze found no issues, and only ran on my Objective-C files ... nice work Apple.

A really big problem with Apple's invention of Swift is the need to update all of the development tools and documentation. Looking at threads about Xcode on the Apple developer site there are a lot of VERY unhappy developers who have huge problems developing their applications with Swift - particularly for iOS.

 

[SecuritySteve]

A really big problem with Apple's invention of Swift is the need to update all of the development tools and documentation. Looking at threads about Xcode on the Apple developer site there are a lot of VERY unhappy developers who have huge problems developing their applications with Swift - particularly for iOS.

Really? Swift has been out for years now. I remember hearing about it when I was in high school. If they haven't updated Xcode for Swift by now, they're incompetent.

 

[Senor Cuete]

Really? Swift has been out for years now. I remember hearing about it when I was in high school. If they haven't updated Xcode for Swift by now, they're incompetent.

They have worked to make it work with Xcode. They're working on it but it's a monumental task.

 

[Senor Cuete]

Did you ever figure this out?

I was reading about Swift on Wikipedia. In the memory management section it said this:

"One problem with ARC is the possibility of creating a strong reference cycle, where objects reference each other in a way that you can reach the object you started from by following references (e.g. A references B, B references A). This causes them to become leaked into memory as they are never released. Swift provides the keywords weak and unowned to prevent strong reference cycles. Typically a parent-child relationship would use a strong reference while a child-parent would use either weak reference, where parents and children can be unrelated, or unowned where a child always has a parent, but parent may not have a child. Weak references must be optional variables, since they can change and become nil."

Have you ever encountered this? Could something like this be the problem?

 

[Senor Cuete]

Another debugging strategy is to add lines of code that check what's happening. For example you could call retainCount and printf or NSlog the result .The Developer documentation says not to use this method and suggests other ways to check this.