Storing bundle files in arrays

Discussion in 'iOS Programming' started by xArtx, May 3, 2013.

  1. xArtx macrumors 6502a

    Joined:
    Mar 30, 2012
    #1
    Hi,
    I have started storing some files that were previously in the resources folder
    of the main bundle, in C arrays instead so they are embedded in the executable,
    and no longer plainly visible in the file system (if one goes looking in a jailbroken device).

    One reason is to make it harder for someone extracting data you didn't want
    them to without encryption
    (you'd have to at least know how to cut the file out of the executable with a hex editor),
    and another is to load UIImages from C arrays (particularly bitmaps),
    to perform operations on the C array and reload it to the UIImage object
    (say you just wanted to invert the green component of the image
    and invert it back at some stage).

    My question is with regard to any consequence with RAM consumption.
    If a UIImage is loaded from a C array, I think two instances of the same
    image data will exist in RAM. If the UIImage is loaded from a file in the
    main bundle, I don't know if the original png data (for example),
    would be freed from RAM once it's able to be represented as a bitmap on the screen.
    So I don't know if the usual way is more effective memory wise.

    Any thoughts on that?
     
  2. truehybridx macrumors member

    Joined:
    Dec 6, 2010
    #2
    I was thinking about how that would work a few days ago. How much pain was it to encode the files into a c string?

    I want to say if you use a UIImage, it will load the image data into each instance, but theres also some form of cacheing i think it does too

    So im thinking if you want one UIImage from the data you embedded, you will have atleast 2 copies of it in memory
     
  3. PhoneyDeveloper macrumors 68030

    PhoneyDeveloper

    Joined:
    Sep 2, 2008
    #3
    If you are compiling images into the global data of your app the data will always be in memory. I assume you're using NSData to copy the image data and then imageWithData (or some similar process) to create the UIImages. There are obviously multiple copies in memory at the same time.

    Seems like a lot of trouble. Whether the memory use is a problem depends on how many and big they are, I guess. UIImage is optimized and smart about memory usage but the way you're doing it is probably something that UIImage can't optimize.
     
  4. xArtx thread starter macrumors 6502a

    Joined:
    Mar 30, 2012
    #4
    There is an old Windows command line tool called bin2c that converts
    any file into a c source file.
    I started using it for iOS for a sample player that plays PCM data.
    In that case, it's playing sample data directly from the array,
    so no need to load that from a file just to get it into an array anyway,
    and I was able to strip out the wav file headers and footers with a
    hex editor before converting them to arrays, so it's just PCM data.

    Same with vector map data for my GPS program.

    The up side is the app launches faster if it had a bit of data to load from
    files at launch. The downside is it takes longer to compile.
    It might pay to have both ways set up, so you could switch to including
    the arrays for final testing only.

    I have loaded bitmaps from file to C arrays, and then to UIImages,
    so I know that can be done, but I don't have anything to hide away with
    images, so I'll leave them alone.
     
  5. Duncan C macrumors 6502a

    Duncan C

    Joined:
    Jan 21, 2008
    Location:
    Northern Virginia
    #5
    It seems to me that you are fighting against the compiler and the runtime memory management by putting resource data in your executable. You will almost certainly have multiple copies of the data in memory. The array declarations will exist in the executable code and bloat the size of the code module that loads the array. Then once you've run that code you will have the in-memory image object.

    Better to save your resources into files that you can load when needed and release when you are done with them. If you're worried about somebody stealing your assets, encrypt them, then decrypt them into memory at runtime. Even simple "swizzling" (running EOR, byte swapping, salting with some random constant, etc) will defeat all but the most determined thief. If you really care, use real encryption. However, even that can be defeated, since a determined hacker can trace through your code on a jailbroken device, find the point at which the decrypted asset exists in memory, then capture and save THAT. Without dedicated hardware support any software security system can be defeated.

    If you're determined to put media data in your code you should probably load it in a separate module that you load and call at startup, then release. That way the OS level memory management has a better chance of unloading that block of code once you are done with it. (I started to say "paging out", but iOS does not use virtual memory.) I honestly don't know how iOS manages loading and releasing blocks of code. I've never needed to know, or worry about it. By putting big media assets in code memory you make it your problem however.
     
  6. xArtx thread starter macrumors 6502a

    Joined:
    Mar 30, 2012
    #6
    Hi,
    I haven't embedded anything that will be duplicated (that I know of).
    Images I'm not too worried about leaving in the filesystem.

    Audio data, for example, is simply fed from the array to the input buffer
    frame by frame as usual.
    I wasn't trying to hide audio data either, it's just the way I've always done it
    on other platforms.

    And no, not actually fighting to keep anything in memory.
    That is what I was doing when I was using AVplayer.
    Now it would make no difference to me to load that same data from files.
    iOS has never discarded anything I've put into a C array myself, whether loaded from file,
    or explicitly declared including data.
    I think you'd get a low memory warning before iOS discarded your C arrays actually.

    The map data, I would like to hide, but didn't use any encryption
    (even simple xor encryption) because I've heard it's a hassle to get approval for that.
    Also been warned that any encryption is still encryption, and you need to
    tell them about it, and go through the process for using it.

    If it were free reign with encryption, I'd simply xor each byte of the map file
    with every byte of an audio file with the same index.
    If you really cared, and had the CPU time, you could make sure the decrypted
    file never exists in RAM.
     
  7. Duncan C macrumors 6502a

    Duncan C

    Joined:
    Jan 21, 2008
    Location:
    Northern Virginia
    #7
    True. If you're just passing C pointers to memory buffers it doesn't much matter if the data is in code space or the heap.

    That's not what I meant. I meant that the OS is set up to manage large data structures in the heap, and has support for loading content into the heap, then releasing it when you are done with it.

    iOS memory management is not designed around storing large media files in your executable. By storing large data structures in your executable you are going against the design of the iOS memory management system.

    iOS will never discard C arrays. It only manages NSObjects, and then only under ARC. Managing Malloc'ed memory is up to you. Managing Core Foundation objects is also up to you.

    That is my point. If you put large data structures in your executable you don't have any way to release them when you are done with them. They will get loaded into memory when the block of code that contains them is loaded into memory, and will tie up physical memory the entire time that block of code is in memory. You don't have insight into when the system unloads code blocks, so you can't free up the memory when you are done with it.
    I don't know. I would think that data obfuscation like you describe, which really can't be called encryption, does not fall under the export restrictions. Apple is concerned about not violating the US laws on exporting encryption technology outside of the US. XORing of your data is not really encryption.
    You could, I suppose, make sure that the ENTIRE decrypted file never exists in RAM, but in order for the OS to be able to use it, it needs to be decrypted. Not decrypting the whole thing at once just makes the hackers job harder, not impossible. (He would need to patch the OS to capture the byte stream you provide, assemble it into a whole, and save that.) Again, without hardware support there is no way around the fact that your program must decrypt the data and pass it to the OS in a decrypted form, which is vulnerable to interception. The goal is to make it hard enough that it's not worth the hacker's time and energy, so he/she steals somebody else's IP instead of yours. (as the saying goes, if you're being chased by a bear you don't have to run faster than the bear - just faster than the other people who are running from the bear.)
     
  8. xArtx thread starter macrumors 6502a

    Joined:
    Mar 30, 2012
    #8
    I'm never done with the particular data.
    Interface sounds, and map data that small enough to exist in RAM.
    ..But I see your point.

    That's right, the set of four paper maps is $44.
    I want the end user to by the maps from the Government dept.
    that is allowing me to distribute them digitally.
    It is an objective to make it harder than spending $44 to extract
    useful data from the app.

    I'm not sure about the encryption thing.
    I tend to agree with you, but would have to be sure before even using obfuscation.
    Apple's guidelines and laws are def not the same thing to me.
    As a guideline I don't date ladies that are smokers, but I am currently breaking my guideline based on other merrit ;) So I don't mind testing guidelines at all, I perceive that it meant they saw merrit.

    My next plan is different.
    The map data is very close to compatible with a Windows program that is
    capable of quickly converting it for use with a handheld GPS unit.
    I can aim to destroy that format, so that the end user would actually
    have to write a program to make the data useful at all.
    For example, the map data would not be pleasant to use if the digits of every
    coordinate pair were reversed.

    It would also be hell for the program used to convert the data if the
    longitude and latitude fields were swapped.
    The format has many rigid rules that I can easily break :)

    The app is out, and it's first update is submitted, but I have not promoted it
    in the intended way at all for this issue. It's only being tested by a few friends,
    but I can sort it out one way or another :)
     
  9. xArtx thread starter macrumors 6502a

    Joined:
    Mar 30, 2012
    #9
    Update was released today, and I'm confused.
    Estimated size was 17Mb, size in App Store is 6.6Mb.
    It appears that only the binary that is used by the device is downloaded by the device.

    I'm sticking with this.
     
  10. dejo Moderator

    dejo

    Staff Member

    Joined:
    Sep 2, 2004
    Location:
    The Centennial State
    #10
    I'm not following you here. Care to elaborate?
     
  11. xArtx thread starter macrumors 6502a

    Joined:
    Mar 30, 2012
    #11
    Well I left the project to compile for ARMv7 and ARMV7s
    for both the original version, and the update.
    My first dist version was 6Mb from iTunes, and that's what was estimated
    by Xcode in Organiser when I submitted it.

    Moving files from the file system to explicitly declared C arrays (with data)
    bloated the size of the archive because the binary is compiled for two architectures.
    At least that's what it looks like when the App increases from 6 to 17Mb in my archive.

    But the download size of the update is 6.6 Mb.
    The 0.6Mb sounds only like the size of the new data introduced for the update.

    It looks to me like they stripped the ARMv7s binary at their end.
    I don't see any other explanation.
     
  12. PhoneyDeveloper macrumors 68030

    PhoneyDeveloper

    Joined:
    Sep 2, 2008
    #12
    While it's possible they stripped the binary when the app is downloaded by iTunes it doesn't know what device it will be installed on. If you downloaded it onto a device directly from the app store it still gets synced to your computer and can be installed on another kind of device. It doesn't seem like there's a good point in that process to strip out the unused binary slice.
     
  13. xArtx, May 17, 2013
    Last edited: May 17, 2013

    xArtx thread starter macrumors 6502a

    Joined:
    Mar 30, 2012
    #13
    I don't have any other explanation.
    I didn't use the last two support tickets for my dev account,
    and would probably use one to find out.

    EDIT,,
    From what I understand, an ARMv7 compiled binary runs on all devices including iPhone 5,
    or the iPhone 5 wouldn't be able to run all of the old apps that were written before it.
    People out there have ditched the ARMv7s binary somehow, and submitted, and got approved.
    But I didn't do that at my end. It looks to me that they removed the ARMv7s binary then..
    .. given what you say about iTunes.

    The ARMv7s binary might have been deleted altogether, I just didn't do it.
     
  14. xArtx thread starter macrumors 6502a

    Joined:
    Mar 30, 2012
    #14
    What I did was shift the entire file so it is offset by one bit.
    So the file is still in the clear, and sequential in memory,
    but isn't directly accessible with C.
    You could read it directly from the start bit address with an old Amiga and floppy drive,
    as it is my understanding, you can start reading from any bit within a byte.

    As I shift the file back in the app, I count the live bits in the file,
    and an incorrect count is an error condition if you don't want the file replaced.
     

Share This Page