Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.

f54da

macrumors 6502a
Original poster
It should not actually be too difficult to get the magic trackpad 2 to work on 10.9 or lower, complete with native multitouch gestures (excluding force touch of course(. Basically, you will need 2 parts:

1) Kernel driver to communicate with MT2 and receive frames containing transducer (finger) locations and click status. This part is already well reverse-engineered thanks to Linux drivers for MT2, so they can directly re-used, with only a bit of effort needed to port the bluetooth (l2cap?) communication to an iokit driver.

2) Simulate a MT1 using the above inputs. A similar project already exists in the hackintosh community, VoodooInput, where arbitrary trackpad devices can map onto a simulated MT2 thereby gaining native multitouch support. IIUC the way that works is that they spoof the device code of the MT2, and then IOKit driver matching will automagically match the native multitouch HID driver to the nub exposed by the simulated hardware device. Then all you have to do is convert the actual device input to the magic trackpad frame format which is sent out from simulated device, and the osx multitouch stack takes care of the rest. While VoodooInput simulates a MT2, we want to simulate a MT1 so we cannot use the code exactly, but since the MT1 frame format is also decently well documented (since it has a linux driver), much of code can be reused here too.

Btw last I checked the MT1 drivers for linux were a bit incomplete in that they didn't reverse engineer some of the fields of the received frame. I think one of them was the finger id. It's actually quite easy to do this reverse engineering work from userspace since the private multitouch framework has a callback you can register to dump the raw frame and compare it against the parsed frame contents.

Here's some random code I threw together while playing around with this, in case it's useful to anyone.

So in principle it should only be a long-weekend project for someone who's decently comfortable with C. I post this here in hopes that it might inspire someone to go ahead and write the thing (it can also be a good learning experience).

Objective-C:
#include <math.h>
#include <unistd.h>
#include <CoreFoundation/CoreFoundation.h>
#import <Foundation/Foundation.h>

// IOKit Fundamentals: https://developer.apple.com/library/archive/documentation/DeviceDrivers/Conceptual/IOKitFundamentals/Introduction/Introduction.html

// https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/KernelProgramming/IOKit/IOKit.html#//apple_ref/doc/uid/TP30000905-CH213-TPXREF104

// https://encyclopediaofdaniel.com/blog/making-the-magic-trackpad-work/

/**

the IOKit structure is as follows: VoodooI2C publishes an IOHIDDevice (i.e VoodooI2CHIDDevice). IOHIDDevice creates an IOHIDInterface nub. AppleMultitouch(Trackpad|Mouse)HIDEventDriver (which is a subclass of IOHIDEventDriver) attaches to IOHIDInterface.
AppleMultitouch(Trackpad|Mouse)HIDEventDriver is a subclass of AppleMultitouchInputHIDEventDriver which is responsible for interfacing with the mutitouch library - it instanties an AppleMultitouchDevice object and passes the report calls back and forth from the multitouch library to the hid device
AppleMultitouch(Trackpad|Mouse)HIDEventDriver and AppleMultitouchInputHIDEventDriver can be found in AppleTopCaseHIDEventDriver
AppleMultitouchDevice can be found in AppleMultitouchDriver

http://mirror.informatimago.com/next/developer.apple.com/hardwaredrivers/customusbdrivers.html

**/

typedef struct { float x,y; } mtPoint;
typedef struct { mtPoint pos,vel; } mtReadout;

typedef struct {
    int frame;
    double timestamp;
    int identifier, state, fingerid, handid;
    mtReadout normalized;
    float size;
    int zero1;
    float angle, majorAxis, minorAxis; // ellipsoid
    mtReadout mm;
    int zero2[2];
    float density;
} Finger;

typedef void *MTDeviceRef;
typedef int (*MTContactCallbackFunction)(int,Finger*,int,double,int);
typedef void (*MTPathCallbackFunction)(int, long, long,Finger*);
typedef void (*MTFullFrameCallbackFunction)(int /*device*/, uint8_t* /*data*/, int /*size?*/);

MTDeviceRef MTDeviceCreateDefault();

CFMutableArrayRef MTDeviceCreateList(void);
CFStringRef mt_CreateSavedNameForDevice(MTDeviceRef);
void MTDeviceGetTransportMethod(MTDeviceRef, int *);
int MTDeviceGetParserType(MTDeviceRef);
int MTDeviceGetParserOptions(MTDeviceRef);
void MTRegisterContactFrameCallback(MTDeviceRef, MTContactCallbackFunction);
void MTRegisterPathCallback(MTDeviceRef, MTPathCallbackFunction);
void MTRegisterFullFrameCallback(MTDeviceRef, MTFullFrameCallbackFunction, int /*0x0 in practice*/, int /*unused?*/);
void MTDeviceStart(MTDeviceRef, int); // thanks comex
void mt_HandleMultitouchFrame(int, int);

// Sample frame data:
// |28| |68| |10 28 77| |5a 90| |b| |17 ellipse major 35 ellipse minor| |c5 = size in 1/8 increment| |82 45 = /*state + figner id*/|
void fullframeCallback(int device, uint8_t* data, int size) {
    if (size == 13) {
        printf("%d ", data[12] & 0xF);
    }
//    for(int i = 0; i < size; i++)
//        printf("%x ", data[i]);
//    printf("\n");
}



void pathCallback(int device, long pathID, long state, Finger* touch) {
    //printf("Inside callback\n");
    if (state == 3 || state == 4)
        printf("%d %d %d\n", touch->identifier, touch->fingerid, touch->handid);
}

int callback(int device, Finger *data, int nFingers, double timestamp, int frame) {
    for (int i=0; i<nFingers; i++) {
        Finger *f = &data[i];
        //if (f->state == 3 || f->state == 4)
            printf("%d\n", f->fingerid);
//        printf("Frame %7d: Angle %6.2f, ellipse %6.3f x%6.3f; "
//               "position (%6.3f,%6.3f) vel (%6.3f,%6.3f) "
//               "ID %d, state %d [finger id %d / hand id %d?] size %6.3f, density %6.3f?\n",
//       f->frame,
//       f->angle * 90 / atan2(1,0),
//       f->majorAxis,
//       f->minorAxis,
//       f->normalized.pos.x,
//       f->normalized.pos.y,
//       f->normalized.vel.x,
//       f->normalized.vel.y,
//       f->identifier, f->state, f->fingerid, f->handid,
//       f->size, f->density);
    }

    return 0;
}

int main() {
    printf("%p\n", (void *)(size_t) &mt_HandleMultitouchFrame);
    NSArray* devices = CFBridgingRelease(MTDeviceCreateList());
    for(int i = 0; i < devices.count; i++)
    {
        MTDeviceRef device = (__bridge MTDeviceRef)devices[i];
        int x;
        MTDeviceGetTransportMethod(device, &x);
        printf("%d ", x);
        int y = MTDeviceGetParserType(device);
        int z = MTDeviceGetParserOptions(device);
        printf("%d %d\n", y, z);
                if (x != 4) continue;
        //NSString *yourFriendlyNSString = (__bridge NSString *) mt_CreateSavedNameForDevice(device);
        //NSLog(@"%@",yourFriendlyNSString);
        MTRegisterContactFrameCallback(device, callback);
        MTRegisterFullFrameCallback(device, fullframeCallback, 0, 0);  
//        MTRegisterPathCallback(device, pathCallback);
        MTDeviceStart(device, 0);
    }
    printf("Ctrl-C to abort\n");
    sleep(-1);
    return 0;
}
 
Last edited:
  • Like
Reactions: Wowfunhappy
Was bored so I decided to just document a bit more. Here's the packet structure which I basically modified from https://github.com/acidanthera/Vood...InputSimulator/VoodooInputSimulatorDevice.hpp

And here's some rough code that just memcpys the frame into the struct so you can play around with it. Not too shabby.. I still don't know what Unk1 and Unk2 are though. Can you figure out what they might mean by playing around with it?

Note that the packet structure only works for the magic trackpad 1, the internal trackpad uses a different structure (but the contact frame callback should still work for that, since all data is already parsed).

Both Unk1 and Unk2 seem to be 7 for the top 1/3 of the trackpad. To me Unk2 seems somehow related coordinates as a rough granularity since it increases as you move your finger up and right, and decreases as you move your finger down and left. Unk1 shows a very similar pattern but is even more sensitive, like a more fine-grained version.

C++:
#include <math.h>
#include <unistd.h>
#include <CoreFoundation/CoreFoundation.h>
#import <Foundation/Foundation.h>


typedef struct { float x,y; } mtPoint;
typedef struct { mtPoint pos,vel; } mtReadout;

typedef struct {
    int frame;
    double timestamp;
    int identifier, state, fingerid, handid;
    mtReadout normalized;
    float size;
    int zero1;
    float angle, majorAxis, minorAxis; // ellipsoid
    mtReadout mm;
    int zero2[2];
    float density;
} Finger;

typedef void *MTDeviceRef;
typedef int (*MTContactCallbackFunction)(int,Finger*,int,double,int);
typedef void (*MTPathCallbackFunction)(int, long, long,Finger*);
typedef void (*MTFullFrameCallbackFunction)(int /*device*/, uint8_t* /*data*/, int /*size?*/);


MTDeviceRef MTDeviceCreateDefault();

extern "C" {
    CFMutableArrayRef MTDeviceCreateList(void);
    CFStringRef mt_CreateSavedNameForDevice(MTDeviceRef);
    void MTDeviceGetTransportMethod(MTDeviceRef, int *);
    int MTDeviceGetParserType(MTDeviceRef);
    int MTDeviceGetParserOptions(MTDeviceRef);
    void MTRegisterContactFrameCallback(MTDeviceRef, MTContactCallbackFunction);
    void MTRegisterPathCallback(MTDeviceRef, MTPathCallbackFunction);
    void MTRegisterFullFrameCallback(MTDeviceRef, MTFullFrameCallbackFunction, int /*0x0 in practice*/, int /*unused?*/);
    void MTDeviceStart(MTDeviceRef, int); // thanks comex
    void mt_HandleMultitouchFrame(int, int);
}

/**
| 77| |5a 90| |b| |17 ellipse major 35 ellipse minor| |c5 = size in 1/8 increment| |82 45 = (state + figner id)|
*/

struct __attribute__((__packed__)) MAGIC_TRACKPAD_INPUT_REPORT_FINGER {
    SInt16 X: 13;
    SInt16 Y: 13;
    UInt8 Unk1: 3;
    UInt8 Unk2: 3;
    UInt8 Touch_Major: 8;
    UInt8 Touch_Minor: 8;
    UInt8 Sz: 6;
    UInt8 TrackingId: 4;
    UInt8 Orientation: 6;
     UInt8 FingerId: 4;
    UInt8 HoverState: 4;
};

void parseTouch(int idx, uint8_t* tdata) {
//        x = (tdata[1] << 27 | tdata[0] << 19) >> 19;
//        y = -((tdata[3] << 30 | tdata[2] << 22 | tdata[1] << 14) >> 19);
//        touch_major = tdata[4];
//        touch_minor = tdata[5];
//        size = tdata[6] & 0x3f;
//        id = (tdata[7] << 2 | tdata[6] >> 6) & 0xf;
        int     orientation = (tdata[7] >> 2) - 32;
//        state = tdata[8] & TOUCH_STATE_MASK;
//        down = state != TOUCH_STATE_NONE;
    MAGIC_TRACKPAD_INPUT_REPORT_FINGER finger;
    memcpy(&finger, tdata, 9);
    printf("%d %d\n", finger.Unk1, finger.Unk2);    

}

// Sample frame data:
// |28 68 10 28| | 77| |5a 90| |b| |17 ellipse major 35 ellipse minor| |c5 = size in 1/8 increment| |82 45 = /*state + figner id*/|
// Ref: https://github.com/torvalds/linux/blob/master/drivers/hid/hid-magicmouse.c
void fullframeCallback(int device, uint8_t* data, int size) {
//    if (size == 13) {
//        printf("%d %d\n", data[4] & 0xF, data[6] & 0xF);
//    }


//    return;
    const int header_size = 4; // First 4 bytes are header
    const int finger_frame_size = 9; // 9 bytes
 
    /* Expect four bytes of prefix, and N*9 bytes of touch data. */
    if (size < header_size || ((size - header_size) % finger_frame_size) != 0)
        return;
    int nfingers = (size - header_size) / finger_frame_size;
    int trackpadclicked = data[1] & 1;
 
    for (int i = 0; i < nfingers; i++) {
        parseTouch(i, data + header_size + i * finger_frame_size);
    }
 
    /* The following brovide a device speits pcific timestamp. They
     * are unused here.
     *
     * ts = data[1] >> 6 | data[2] << 2 |
     */
 
    //printf("%d %d\n", nfingers, clicks & 1);
//    for(int i = 0; i < size; i++)
//        printf("%x ", data[i]);
}



void pathCallback(int device, long pathID, long state, Finger* touch) {
    //printf("Inside callback\n");
    if (state == 3 || state == 4)
        printf("%d %d %d\n", touch->identifier, touch->fingerid, touch->handid);
}

int callback(int device, Finger *data, int nFingers, double timestamp, int frame) {
    for (int i=0; i<nFingers; i++) {
        Finger *f = &data[i];
        //if (f->state == 3 || f->state == 4)
            printf("%d\n", f->fingerid);
//        printf("Frame %7d: Angle %6.2f, ellipse %6.3f x%6.3f; "
//               "position (%6.3f,%6.3f) vel (%6.3f,%6.3f) "
//               "ID %d, state %d [finger id %d / hand id %d?] size %6.3f, density %6.3f?\n",
//       f->frame,
//       f->angle * 90 / atan2(1,0),
//       f->majorAxis,
//       f->minorAxis,
//       f->normalized.pos.x,
//       f->normalized.pos.y,
//       f->normalized.vel.x,
//       f->normalized.vel.y,
//       f->identifier, f->state, f->fingerid, f->handid,
//       f->size, f->density);
    }

    return 0;
}

int main() {
    printf("%p\n", (void *)(size_t) &mt_HandleMultitouchFrame);
    NSArray* devices = CFBridgingRelease(MTDeviceCreateList());
    for(int i = 0; i < devices.count; i++)
    {
        MTDeviceRef device = (__bridge MTDeviceRef)devices[i];
        int x;
        MTDeviceGetTransportMethod(device, &x);
        printf("%d ", x);
        int y = MTDeviceGetParserType(device);
        int z = MTDeviceGetParserOptions(device);
        printf("%d %d\n", y, z);
                if (x != 4) continue;
        //NSString *yourFriendlyNSString = (__bridge NSString *) mt_CreateSavedNameForDevice(device);
        //NSLog(@"%@",yourFriendlyNSString);
    //    MTRegisterContactFrameCallback(device, callback);
        MTRegisterFullFrameCallback(device, fullframeCallback, 0, 0);
//        MTRegisterPathCallback(device, pathCallback);
        MTDeviceStart(device, 0);
    }
    printf("Ctrl-C to abort\n");
    sleep(-1);
    return 0;
}

Code:
/* Finger Packet
+---+---+---+---+---+---+---+---+---+
|   | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
+---+---+---+---+---+---+---+---+---+
| 0 |           x: SInt13           |
+---+-----------+                   +
| 1 |           |                   |
+---+           +-------------------+
| 2 |           y: SInt13           |
+---+-----------+-----------+       +
| 3 |   Unk1    |   Unk2    |       |
|   |   UInt3   |   UInt3   |       |
+---+-----------+-----------+-------+
| 4 |       touchMajor: UInt8       |
+---+-------------------------------+
| 5 |       touchMinor: UInt8       |
+---+-------------------------------+
| 6 |  TId   |         Sz           |
+---+-------+-----------------------+
| 7 |       angl            | FId   |
+---+-----------+---+---------------+
| 8 |   Hover     |      FingId     |
|   |   UInt4     |       UInt4     |
+---+-----------+---+---------------+
*/

I think the hardest part remaining is actually figuring out how you even write an IOKit driver. I can find 0 documentation on how to do this, and the few examples I can find are for non-bluetooth devices. I know there exists an IOBluetoothHIDDriver, and there's a minimal header file which seems to indicate that it exposes a get/set report function which should be all we need to communicate with the magic trackpad. Looking at the linux source, it seems to enable multitouch frames you've got to send the trackpad a sequence of magic bytes.

The other missing part is the hid descriptors. There's a fixed sequence of bytes we need to return for newReportDescriptor() and we need to handle getReport() appropriately. I'm not sure how we can find the existing values for these.
 
Last edited:
Forgot to include the header format

Code:
struct __attribute__((__packed__)) HEADER {
    UInt8 report_id: 8;
    UInt8 button_data: 6;
    UInt32 timestamp: 18;
};

Also you can use PacketLogger (included in xcode tools) to sniff the bluetooth traffic. We see the following set_feature_reports sent to the MT1 device {0xf1, 0xdb}, {0xf1, 0xdc}, {0xf0, 0x02}, etc. Basically the same ones found in this https://lkml.org/lkml/2010/8/31/324 – for the shim driver we just need to respond ok.

I also see some get feature_reports: {0x60}, {0x61}, etc. It seems that some of these are the sensor rows/cols, sensior region params, but others I'm not sure. I think we can just hard code the responses for these since they should be constant.

Finally here's a small bonus. Did you ever feel that magic trackpad palm recognition was terrible (especially when using tap to click)? Turns out that this is because it seems to always think the palm is a thumb. By combining the frame callback with an event tap, we can block these erroneous tap events. Enjoy

Code:
int lastFinger = -1;


CGEventRef eventOccurred(CGEventTapProxy proxy, CGEventType type, CGEventRef event, void* refcon) {
    if (type == kCGEventLeftMouseDown) {
        if (lastFinger == 1 || lastFinger == 7) {
            event = NULL;
            printf("Filtered click\n");
            lastFinger = -1;
        }
    }
    return event;
}


CFMachPortRef createEventTap() {
    CGEventMask eventMask = CGEventMaskBit(kCGEventLeftMouseDown);
   
    if (!AXAPIEnabled() && !AXIsProcessTrusted()) {
        printf("axapi not enabled\n");
    }
   
    return CGEventTapCreate(kCGHIDEventTap,
                            kCGHeadInsertEventTap,
                            kCGEventTapOptionDefault,
                            eventMask,
                            eventOccurred,
                            NULL);
}

int callback(int device, Finger *data, int nFingers, double timestamp, int frame) {
    if (nFingers == 1) {
        lastFinger = data[0].fingerid;
        //printf("%d\n", lastFinger);
    }
    else {
        lastFinger = -1;
    }

    return 0;
}


MTDeviceRef device = NULL;
void initMagicTrackpad() {
    printf("Init device\n");
    NSArray* devices = (__bridge_transfer NSArray *) MTDeviceCreateList();
        for(int i = 0; i < devices.count; i++)
        {
            MTDeviceRef deviceI = (__bridge MTDeviceRef) devices[i];
            int x;
            MTDeviceGetTransportMethod(deviceI, &x);
            printf("%d ", x);
            int y = MTDeviceGetParserType(deviceI);
            int z = MTDeviceGetParserOptions(deviceI);
            printf("%d %d\n", y, z);
            if (x != 4) {
                continue;
            }
            device = (__bridge_retained MTDeviceRef) devices[i];
            MTRegisterContactFrameCallback(device, callback);
            MTDeviceStart(device, 0);
            break;
        }
    if (device == NULL) {
        printf("No BT trackpad found\n");
    }
}

void stopMagicTrackpad() {
    if (device == NULL) return;
    printf("Stop magic trackpad\n");
    MTUnregisterContactFrameCallback(device, callback); // work
    MTDeviceStop(device);
    MTDeviceRelease(device);
}


static void magicTrackpadAdded(void* refCon, io_iterator_t iterator) {
    io_service_t device;
    printf("Magic trackpad added\n");
    while ((device = IOIteratorNext(iterator))) {
        IOObjectRelease(device);
    }
    dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 1 * NSEC_PER_SEC), dispatch_get_main_queue(), ^{
        initMagicTrackpad();
    });
}

static void magicTrackpadRemoved(void* refCon, io_iterator_t iterator) {
    io_service_t object;
        printf("Magic trackpad removed\n");
    while ((object = IOIteratorNext(iterator))) {
        IOObjectRelease(object);
    }
    stopMagicTrackpad();
}

void registerMTNotification() {
    IONotificationPortRef notificationObject = IONotificationPortCreate(kIOMasterPortDefault);
    CFRunLoopSourceRef notificationRunLoopSource = IONotificationPortGetRunLoopSource(notificationObject);
    CFRunLoopAddSource(CFRunLoopGetMain(), notificationRunLoopSource, kCFRunLoopDefaultMode);
       
    CFMutableDictionaryRef matchingDict = IOServiceNameMatching("BNBTrackpadDevice");
    matchingDict = (CFMutableDictionaryRef) CFRetain(matchingDict);

    //MagicTrackpad added notification
    io_iterator_t magicTrackpadAddedIterator;
    IOServiceAddMatchingNotification(notificationObject, kIOFirstMatchNotification, matchingDict, magicTrackpadAdded, NULL, &magicTrackpadAddedIterator);
    // Run out the iterator or notifications won't start (you can also use it to iterate the available devices).
    io_service_t d;
    while ((d = IOIteratorNext(magicTrackpadAddedIterator))) { IOObjectRelease(d); }

    //MagicTrackpad removed notification
    io_iterator_t magicTrackpadRemovedIterator;
    IOServiceAddMatchingNotification(notificationObject, kIOTerminatedNotification, matchingDict, magicTrackpadRemoved, NULL, &magicTrackpadRemovedIterator);
    while ((d = IOIteratorNext(magicTrackpadRemovedIterator))) { IOObjectRelease(d); }
   
}
   

int main() {
    initMagicTrackpad();
    registerMTNotification();
   
    CFMachPortRef tap = createEventTap();
    if (!tap) {
        return -1;
    }
    if (tap) {
        CFRunLoopSourceRef rl = CFMachPortCreateRunLoopSource(kCFAllocatorDefault, tap, 0);
        CFRunLoopAddSource(CFRunLoopGetMain(), rl, kCFRunLoopCommonModes);
        CGEventTapEnable(tap, true);
        CFRunLoopRun();
       
        printf("Tap created.\n");
    } else {
        printf("failed!\n");
        return -1;
    }
       
   
    printf("Ctrl-C to abort\n");
    sleep(-1);
    return 0;
}

Edit: For the parsed contact frame callback there's also a perhaps better described struct at https://gist.github.com/rmhsilva/61cc45587ed34707da34818a76476e11
 
Last edited:
Also I forgot to document this back when I discovered it, but in case you are one of the dozens of people on mavericks AND using a Magic Trackpad 1, you might have noticed that the trackpad is less responsive at the right edge, for maybe a ~2cm zone.

This exists on the magic trackpad 2 as well as documented in https://forums.macrumors.com/thread...ght-side.2289699/?post=30878184#post-30878184 but there's a bug in 10.9 where the dead zone exists even with notification center gesture disabled. If this bothers you as much as it bothered me, then you should patch the jump at 0xf8a4 in /System/Library/Extensions/AppleMultitouchDriver.kext/Contents/PlugIns/MultitouchHID.plugin/Contents/MacOS/MultitouchHID (this is assuming latest 10.9.5, you'll have to discover the exact offset if you're on an older OS). That corresponds to a conditional in `MTSlideGesture::isBlocked` that checks if the current gesture is horizontal at the right edge (and that basic block is itself only reached for 1 finger mouse pointevents under bluetooth transport).

I could ramble more about how gestures are internally represented but I don't think it's really useful to anyone. Let me know if anyone wants to hear more about it though, and I can polish up a draft of some exploration notes of the multitouch stack that I have.
 
Based on this idea, a lot of work by Claude, and some steering by me, I have a kext handling USB and Bluetooth with gestures. Probably a few days away from being polished enough to release.
 
  • Wow
Reactions: f54da
@schmonz That is amazing to hear! Tbh I had put off doing it myself, knowing back in 2022 that LLMs would probably get good enough that I wouldn't have to do the drudgery myself.
 
Yes, started in userspace for exactly those reasons, then promoted to kext. There’s still a userland helper to re-enumerate USB so that our kext in a non-standard location can win a fresh attachment contest against the built-in generic pointer device.

Could you share (as-is, no need for any additional effort on it) the draft of your multitouch notes? The task to become BNBDevice-equivalent is in front of me and I bet you figured out stuff that would help.
 
@schmonz Sure, here's the notes. The notes were actually figuring out how to fix the issue mentioned in #4, where magic trackpad is less responsive on the right side compared to the left side. This is a bug in the multitouch iokit drivers, specifically for magic trackpads.

I don't know if it will be helpful for your case since it's more an exploration/reverse engineering of how events are processed by the multitouch system rather than how to generate it.

I think for your needs it's best to just look at the OSS project in the hackintosh community, VoodooInput which already simulates a MT2 trackpad.


That said if you want to read about my RE'ing adventures...

A brief summary of IOKit to lay the necessary background: you basically have a hierarchy of kexts that increasingly abstract away events. So for instance, at the very root you might have some driver responsible for low-level PCI communication, and above that is a driver for a USB controller on the PCI bus, and then above that a driver that interprets the specific USB frames of the device. The interaction between each layer of this stack is mediated by what iokit calls "nubs", and basically a driver will expose a "nub" for each device that it wants to be handled by some child driver. IOKit will find the corresponding driver that can "attach" to the nub, load it, and the cycle continues until the leaf. Honestly it sounds a lot more complex than it is, and I think the easiest way to see how this works is to just open up IORegistryExplorer (included as part of xcode) where the tree driver structure is intuitive. Finally, there's one last piece: IOKit also has "user clients" which allow communication with userspace.

Now the multitouch stack on osx has 4 main parts: the multitouch driver, multitouch user client, multitouchhideventdriver, and the multitouch framework. The former 3 are in kernel space while the latter is just a library that can be accessed. Now the interesting part is that the pathway for bluetooth and internal trackpad is actually completely separate, down to the multitouch hid driver. That is, when you connect a magic mouse, the hierarchy is BNBTrackpad (just handles very low-level bluetooth property getting), AppleMultitouchDevice (note the lack of USB in the name), and AppleMultitouchHID. I remember reading that when the magic trackpad originally launched it required a software update which installed those kexts, so I guess the fact that the codepaths remains separate comes from that legacy.

The core of the "magic" is in the AppleMultitouchHID driver, which takes the raw touch events given to it from the higher layer and processes them, eventually dispatching IOHIDEvents for a swipe/gesture/etc. I think these IOHIDEvents are eventually propagated through the HID stack and end up being converted into CGEvents/NSEvents somewhere down the line. The multitouch HID driver has a lot of code related to path tracking, gesture recognition, smoothing, etc.

The overall codepath looks roughly like this: once the raw touch events are obtained from AppleMultitouchDevice, it's passed to the MultitouchHID driver, starting at MTSimpleHIDManager::handleContactFrameEntry and bubbling to MTTrackpadHIDManager::handleContactFrame. (A lot of the MultitouchHID methods have a "simple" version. I'm not quite sure what this is for, but I guess it's some sort of fallback pathway).

(Note that AppleMultitouch.kext also registers a user client which allows any userspace client to register to these raw touch events via MultitouchSupport.framework (communication via mach port. I'm actually not sure why MultitouchSupport.framework exists since from what I can tell it's only purpose is to allow userspace to listen on the raw events, all the actual processing is done within the kext itself. Maybe finding what things link against MultitouchSupport.framework might be insightful here. Maybe there might be some back and forth between the driver and AppleMultitouch when processing touch events, but I didn't find anything substantive here).

From there, MultitouchHID keeps track of a lot of things like hand statistics, the current path, any chords (which I assume are sequences of actions like tap drag), and after a lot of processing logic (which I didn't bother reversing) it eventually computes a gesture code. The list of possible gestures is seen in MTPListGestureConfig:😛arseGestureMotionCode, e.g. "tap, hold, lift, horizontal, rotate, etc." Now the way MultitouchHID is structured is that these gestures are then mapped to actions declaratively. That is, there are setup methods (e.g. createScrollZoomCombo) which has calls like


Code:
MTPListGestureConfig::addGestureToArray(r15, @"Fluid Notification", @"Horizontal", @"Edge Swipe", @"OnlyIfAllMoving", @"LockOnFirstUntilPause", rbx, r12);


The first string is the action to take, the second is the gesture, and the third/fourth/fifth are additional conditions that can be set. Now I'm not exactly sure what that "lock" term is, but I'm fairly certain it's not responsible for the delay I'm seeing since this gesture to action mapping is only registered if the notification gesture is indeed enabled. But since the lag exists with it disabled as well, the delay is likely introduced during the gesture recongition phase, not during the gesture to action mapping. (By the way, the matching to see if a given set of basic chord gestures matches the desired desired is done in MTPListGestureConfig:😛arseGesture). These actions (which are internally represented as MTAction) are then eventually converted into standard IOHIDEvent (which are appended to the base digitizer event created in handleContactFrame). These IOHIDEvents then somehow get picked up by MTTrackpadEventDispatcher::handleEvent where I assume they get sent to the system's usual HID stack. Interestingly, pointer and scroll gestures take a major indirection where instead of being forwarded directly, the kext calls into mt_DeviceDispatchRelativeMouseEvent from MultitouchSupport.framework which calls into AppleMultitouchDeviceUserClient:😛ostRelativeMouseEvent, which calls AppleMultitouchDevice::handlePointerEventFromMultitouch, which finally posts it to the system HID I presume. In case the absurdity of that wasn't clear, for some reason these pointer (and scroll) events seem to be passed around two extra indirections, and the call from MultitouchSupport into AppleMultitouchDeviceUserClient is done through IOConnectCallScalarMethod which is normally used when calling into drivers from userspace. I'm likely misunderstanding something there though, since I find it hard to believe something as basic as pointer events would be roundtripped through userspace.

Finally, with all this background laid we can get to the core issue. We want to determine the point at which the raw touch events are converted into the pointer event (formally known as relative mouse event). We'll work backwards: we know that all gestures eventually map to actions, and the action list in MTPListGestureConfig::eventTypeCFStringToCode just so has an entry named "Mouse Point". Even better, the configuration of gestures to actions is done with strings, so a simple search gives us this line in MTTrackpadHIDManager::createDefaultActionEventsDictionary

Code:
    MTPListGestureConfig::addActionEventToDictionary(rbx, @"Point", @"Mouse Point", 0x0, 0x0);

I have no idea what this does, but from the other lines in there

Code:
    MTPListGestureConfig::addActionEventToDictionary(rbx, @"Swipe Left", @"Swipe", 0x0, 0x0);
    MTPListGestureConfig::addActionEventToDictionary(rbx, @"Swipe Right", @"Swipe", 0x0, 0x0);
    MTPListGestureConfig::addActionEventToDictionary(rbx, @"Swipe Up", @"Swipe", 0x0, 0x0);

I get the feeling this is just mapping from one class of actions to another. So let's look for "Point" then. That brings us to MTTrackpadHIDManager::createDefaultGestureSetsDictionary which has

Code:
            MTPListGestureConfig::addGestureToArray(r13, @"Point", @"Translate", @"Tracking", @"Repetitive", 0x0, 0x0, 0x0);
            MTPListGestureConfig::addGestureToArray(r13, @"Click", @"Tap", 0x0, 0x0, 0x0, 0x0, 0x0);
            CFDictionaryAddValue(rbx, @"Gestures", r13);
            CFDictionaryAddValue(rbx, @"Transitions", @"ToMoreFingers FromMoreFingers FromMoreWithSlightIntegrationDelay AccelOnlyIfSomeResting");
            CFDictionaryAddValue(var_38, @"Point & Click", rbx);

Seems we're on the right track, "Point & Click" is presumably the name given to that enabled gesture set. Remember, the second string in the addGestureToArray call is the gesture that generates the event, so we're looking for where the "Translate" gesture is generated. Back to our handy-handy table in MTPListGestureConfig:😛arseGestureMotionCode which has the line

Code:
    rax = CFStringCompare(rbx, @"Translate", 0x1);
    LOWORD(rcx) = 0x0f;
    if (rax == 0x0) goto loc_10fcf;

Since in this case we can also know that rcx is returned, we know that 0x0f is the code that corresponds to Translate. Some other codes are given below, since it'll help us later on

Code:
    rax = CFStringCompare(rbx, @"Tap", 0x1);
    LOWORD(rcx) = 0x3000;
    if (rax == 0x0) goto loc_10fcf;
    rax = CFStringCompare(rbx, @"Scale", 0x1);
    LOWORD(rcx) = 0x30;
    rax = CFStringCompare(rbx, @"Translate+Scale", 0x1);
    LOWORD(rcx) = 0x3f;
    if (rax == 0x0) goto loc_10fcf;
    rax = CFStringCompare(rbx, @"Translate+Rotate", 0x1);
    LOWORD(rcx) = 0xcf;
    if (rax == 0x0) goto loc_10fcf;
    rax = CFStringCompare(rbx, @"Translate+Scale+Rotate", 0x1);
    LOWORD(rcx) = 0xff;
    if (rax == 0x0) goto loc_10fcf;
    rax = CFStringCompare(rbx, @"Lift", 0x1);
    LOWORD(rcx) = 0x2000;

Now at this point, I was a bit stuck since I had no idea what to do with this information, and just searching for "0x0f" in the entire binary was like looking for a needle in the haystack, and I wasn't even sure if I was on the right path. This is where those other values come into play. Note that Tap is 0x3000. That's a slightly bigger needle, and searching for this in the assembly we quickly come across the line

Code:
 mov        dword [ss:rsp], 0x3000

in MTChordCyclingTrackpad::handleChordTaps. That's a good sign we're on the right track, and the pseudocode shows that this value is used as a parameter in a call to

```
MTGesture::dispatchEvents(var_30, rax, r13, 0x0, 0x0, r12 + 0xa8, 0x3000);
```

Now `MTGesture::dispatchEvents` apparently a signature of

Code:
MTGesture::dispatchEvents(MTDragManagerEventQueue&, __IOHIDEvent*, MTGesturePhase, MTEventMickeys const*, MT2DPlaneMotion const&, MTMotionAxesSpecifier, MTChordSpecifier, double)

which doesn't match the call site exactly so Hopper probably borked the pseudocode a bit, and I have no idea what MTGesture::dispatchEvents actually does since all the function calls are indirect. But nonetheless, we don't really need to know what the function does, since what we care about is who might call it with the 0x0f (translate) gesture. Looking at the other callers of `MTGesture::dispatchEvents`, we see MTChordIntegrating::sendLiftSlideEvents which has the line

Code:
MTGesture::dispatchEvents(r15 - 0xffffffffffffff80, rsi, var_40, 0x10, 0x0, var_48, 0x2000);

Close, but no cigar. 0x2000 is "Lift" though, so at least we know that this is giving us sensible results. After going through all the other callers, we're left with MTSlideGesture::sendSlideKeys and MTSlideGesture::sendSlideMickeys which both do

Code:
MTGesture::dispatchEvents(rdi, rsi, rdx, 0x0, 0x0, r9, LODWORD(LODWORD(rax) & LODWORD(0xff)));

Now go back to the handy-dandy code table, and observe that the codes are actually a bitmask. That is, 0x0f is translate and 0x30 is scale, so scale | translate is 0x3f which is the code for "Translate+Scale". So it seems plausible that MTSlideGesture might handle all the gestures involving translate, which both the normal pointer gesture and 2-finger notification swipe are under. The only thing I'm not sure of, is how events like RotateLeft would be generated, since it doesn't make sense that it is handled by MTSlideGesture but at the same time I didn't find any other callers. I'm also not sure what the "Mickeys" in sendSlideMickeys means, but there's a lot of references to things like updatingMomentumMickeys. Maybe it's some sort of wordplay with Mickey "Mouse"?

Anyway, if we proceed on the assumption that MTSlideGesture is indeed responsible for producing translate events, then we see the following 3 interesting methods: "MTSlideGesture::isBlocked", " MTSlideGesture::isLockedOutByTriggeredSlide", and "MTSlideGesture::isActiveEdgeSlide". The "isBlocked" method makes some calls to "isActiveEdgeSlide", has some timing related things, and also calls the "isLockedOutByTriggeredSlide".

To me, this does indeed sound like something that could be responsible for the delay when at the right side of the trackpad. I guess one next step would be to just patch isActiveEdgeSlide to return false and see what happens.

First, a correction to my previous section: it turns out that modifying the MultitouchHID plugin within AppleMultitouchKext does in fact affect the internal trackpad as well. I have no idea why this is, and even IORegistryExplorer seems to show that the inbuilt (usb) multitouch has a separate multitouch HID nub, but it's worth noting just for correctness.

Now onto the actual reverse-engineering. It turned out that my initial guess of `isActiveEdgeSlide` was very close, but not quite correct. I didn't see any effect just patching that, but within the same MTSlideGesture class I also saw the methods "isLockedOutByTriggeredSlide", "isBlocked", and "canOverrideLockOn," Since each testing cycle required a reboot (it turns out that you can't hot-swap these kexts because they are retained by other kexts, so kextunload refuses to unload it – that's probably why no one in the hackintosh community does it that way), I just harcoded all of them (false, false, true respectively) to see if I was even somewhat on the right path.

This in fact worked – I was able to freely move the pointer along the right edge without that pesky lag from before. Unfortunately, I quickly found that it also had a few other side-effects (not entirely unexpected given the heavy handed patching). Swiping with two fingers *anywhere* on the trackpad (not just the edge) now triggered notification center, making two-finger scrolling impossible unless the NC gesture was explicitly disabled. By itself this wouldn't be an issue because I don't use it much, but multitouch gestures as a whole no longer felt as "stable": when performing three finger swipes, sometimes the pointer would move unexpectedly. Clearly those three methods play some role in ensuring that the slide-gestures (which includes pointer movement, if you recall) don't step on each other's toes.

So we have to take a more fine-grained approach. A bit of trial and error shows that patching "isBlocked" alone is sufficient to recreate the behavior where the pointer issue is fixed, but the weird glitches are introduced. (And in fact, isBlocked ends up calling "isActiveEdgeSlide" and "isLockedOutByTriggeredSlide", so it's an easy guess off the bat). Now at first glance this is one ugly function, and unfortunately Hopper's decompiler basically balks at this control flow and ends up giving you spaghetti code. (Maybe Ghidra or IDA would have done better?). Not only is the control-flow very branchy, but there aren't many function calls in there to infer what each branch does. It seems to just be pointer dereferences all the way down.

I have attached the CFG so you can take a look yourself. I'm going to discuss this part in quite a bit of detail because this is honestly the first time I've had to work through a CFG like this, and it really did feel like a puzzle that deserves a well-written solution. This kind of thing where the solution is just within reach is perfect nerd-sniping bait.

So our task is somewhat clear: we have a function whose return value determines if a gesture is "blocked." (I'm being a bit vague here about which gesture it corresponds to because at that time it wasn't clear to me either. I'll explain more later). Hardcoding it to false fixes the pointer issue, but introduces other glitches. The only two function calls it makes are to "isActiveEdgeSlide" and "isLockedOutByTriggeredSlide". But patching those two functions alone doesn't have any effect. So it's likely the case that somewhere in this maze of basic blocks there is a jump that is taken only when the finger is on the right-edge, causing the function to return true (is blocked). If the finger is not on the right-edge, we assume it will return false. (I didn't actually verify this because I feared that harcoding it to return true would effectively brick the trackpad, and then I'd need to dig out a usb mouse. But it's a reasonable assumption based on what we know at this point). There should also be some other codepath in there that is taken when two fingers move horizontally, and a jump that returns false (not blocked) if the fingers are near the edge, but true otherwise. If so, this would explain why harcoding the entire function to return false fixed the pointer issue but introduced the NC opening glitch.

Of course, the above might also be wishful thinking. Looking at that maze of jumps it wasn't clear at all if there were distinct codepaths like this. In principle, we could have tried to systematically gain info by hardcoding one basic-block at a time to figure out what each conditional was testing, but this would require a lot of reboots. We'll have to be a bit smarter about this. What we need is more info. So first let's try investigating the function that calls this one, MTChordIntegrating::continueChordIntegration so we can get some context as to what exactly this function does. It has this section


Code:
                            rax = MTSlideGesture::isBlocked(rdx + rax, r13, var_50, rbx);
                            rdi = r14;
                            if (!rax) {
                                    rcx = *(r12 + 0x188);
                                    MTSlideGesture::fireGesture(rdi, r13, var_50, rcx, var_90);
                            }

which seems to indicate that "MTSlideGesture::isBlocked", just as the name implies, returns a bool that indicates whether or not an action should be dispatched. At this point though I was still a bit unclear as to how the return value of "isBlocked" mapped onto the states of the different gestures that could be performed. Up until now I was thinking that MTSlideGesture was some sort of global controller that handled and interpreted all the slide gesture's states, and that the "isBlocked" would pause interpretation of all slide gestures. This seemed reasonable since under normal circumstances only one gesture is ever active at a time.

But then I thought back to the glitches induced when hardcoding the return value, where the pointer jumped at the same time a three-finger swipe as active... and in the midst of grumbling about Hopper's poor C++ disassembly the penny dropped that this was an IOKit driver where full-blown object-orientedness was the norm. And thus `MTSlideGesture` was not merely a global controller (singleton in OOP terms?) but in fact was meant to be constructed once for each slide gesture, and all these MTSlideGesture objects would independently handle their own thing. And looking at the code confirms that there is a vector<MTSlideGesture> driven by a loop (entity component system-esque? Idk I'm not good at these OOP patterns).

So this is good, we now have a solid idea of the terrain we're on. But note that this actually isn't OOP to the max since if it were we'd expect a derived class for every type of slide gesture an MTTranslateGesture, MTScrollGesture, etc. (And if IOKit were instead EnterpriseIOKit™ written in Java, maybe we'd have an `abstract MTTwoFingerGestureProviderModule`). But we don't have those, so the state identifying what type of gesture this corresponds to must be stored in some member field somewhere, probably as a bitmask since we've already seen use of those.

Armed with this knowledge we can start deciphering the function one basic block at a time. I've attached the annotated version with comments on each block for you to take a gander at. Overall, a pointer event at edge eventually lands on block 0xf8a4. There are two edges out of that: one returns from the function with the return value of true (isBlocked), the other seems to lead to a common pathway that a lot of other blocks end up pointing to, which ends up returning the value of isLockedOutByTriggeredSlide(). Since we know that patching isLockedOutByTriggeredSlide didn't work previously, we assume that whatever the check 0xf8a4 does is what causes it the function to return true. A simple few bytes of patching makes us unconditionally take the other path, and this ends up working perfectly! And with the change being extremely minimal and well-targeted, I'm reasonably confident this will not affect anything else.

With the entire CFG now understood we can describe its entire logic in high-level terms. It ends up being quite straightforward, really although it was a real pain to analyze.

* isBlocked returns false if this gesture is "live", and returns true if the gesture should be... blocked from being executed
* If the gesture config has "OnlyIfQuick", check if the time between the last few events is small enough
* If the gesture config has "OnlyFromEdge: If isActiveEdgeSlide() returns false, then return blocked, else go to default clause.
* If we're a mouse point gesture, and there is currently 1 finger active: First do some timing check to ignore accidental short brushes. Then if we're at the edge (and not enough time has passed???), return blocked, else go to default.
* If we're a scroll event (I think all 2/3 finger swipes count as scroll?): if we have 2 fingers, do some timing checks, and if that passes then go to default clause. If 3 fingers, go to default clause.
* Default: Do some more timing checks, and return isLockedOutByTriggeredSlide(). If there are no other slide gestures active, we thus return false (not blocked). This is where we end up most of the time.

Interesting that ultimately in the end there turned out the be a specific code path responsible for the edge-lag. I kind of expected it to be a bug, not an intentional feature (although it being a bug would have made it orders of magnitude harder to track down). I'm also curious how this logic changed in newer osx versions where this is supposedly fixed, although I don't want to stare at any CFGs for the next few months at least.

1781808515252.png

1781808538048.png

I still wasn't completely satisfied since I didn't really understand why what I did worked, so I ran it through Ghidra and got much better decompilation results, which made it a bit easier to understand how it worked. See the two below attached functions. The key point is that 0xf89a actually tests for bluetooth transport (i.e. magic trackpad) and 0xf8a0 actually tests the gesture code against horizontal motion (0b0011) which translate (0b1111) does satisfy (so that branch ends up being always taken for mouse point events). The mystery check in 0xf8a4 seems to be checking some sort of edge/zone bitmask, based on a very similar check in MTSlideGesture::isActiveEdgeSlide.

So put together, I learned the following additional things:
  • There's a separate path specifically for bluetooth multitouch devices here (i.e. magic trackpad), and it references a different edge bitmask than the internal one. Not sure why this is, and when I tried to figure out how the two differ they seemed identical except the former is updating by ORing the old and current bitmask, while the latter is updated by ANDing them. I have no idea if that's correct though, and what significance this has.
  • The check that I bypassed is indeed the correct one, and the fact that it works isn't just an accident but is the very check that is responsible for the observed behavior.
  • One very interesting thing though: note that in isBlocked we lookup handstats[0xd9], not 0xd8 like we do in isActiveEdgeSlide. This is probably not a mistake since these are compiler derived offsets so it's likely that whoever wrote the code intentionally did a right-shift of 8 bits which would be equivalent to reading one byte ahead. But why did they do this? If you change it to 0xd8 instead, then the "dead zone" at the right becomes smaller. It shrinks from two finger widths to one finger width. It really feels like the 1 finger width version is what it "should" have been, especially since other checks use 0xd8, but at the same time I'm not sure if it's actually a bug.
  • I'm still not completely sure what the format of the edge zone bitmask is: the middle 4 bits seem to be indicative of position somehow, but I'm not sure how that squares with the fact that the dead zone got bigger if 0x9 was used instead (effectively shifting the bits used by 8). I think I've reached the limits of my static analysis skills, and kernel debugging seems to require quite a bit of setup.
  • Thumb rejection is not done at this level. It happens when the raw events are processed into a path.
 
Last edited:
  • Like
Reactions: schmonz
Anything that seeds Claude’s inferences with some actual facts of what’s going on inside is helpful for sure! Thank you for this work from your actual brain. Mine has just been hand-waving some intuition, expressing preferences and desires, and making the needed local hand-meat fulfill Claude’s experimental requests. Repo is caught up to the latest now, BTW, including the README. Current task is figuring out what’s blocking tap-to-click.
 
>how to show up in the Trackpad prefpane.
This might be checking based on recognized vendor ids? If you disassemble the prefpane (or have claude do it), it should tell you how it's recognizing.
 
Also @schmonz I was skimming your code, it seems tap-to-click requires you to do state recognition before synthesizing frames? Is that true? It feels odd, I would have thought that the apple multitouch kext would handle everything and you'd only need to translate 1:1 between the MT2 and MT1 frame formats.

 
I’m hoping to end up with as little code as possible, driving as much of the existing multitouch stack as possible, but haven’t read the code much (deferring that until everything seems to work and we can refactor to my understanding). Just made a fresh push with a more minimal approach — maybe better on this front?
 
>maybe better on this front?
Idk, my eyes sort of glaze over when there's too much llm-generated code & docs, and doesn't seem the README was updated for the "more minimal approach" you mentioned.

Once it's all working it should be easy to pare it down though, so probably not worth prioritizing that right now. But the main point I was making was that philosophically you shouldn't need to do any complex state tracking yourself since I'd think that MT2 frames and MT1 frames can be mapped onto each other. The MT2 has a simulated click which might result in some complexity though, I guess for that you probably need to send something back to the device to tell it to trigger the click... maybe that's why the state tracking was required.
 
I don’t blame you one bit. My desired end state will be approximately 97% the interface that makes third-party multitouch drivers easy to write (and internally well-factored, including a few well-marked dragons) and 3% the MT2-specific invocation of said interface. That’s when I expect to finally be able to read the code.
😎
 
Register on MacRumors! This sidebar will go away, and you'll see fewer ads.