Register FAQ / Rules Forum Spy Search Today's Posts Mark Forums Read
Go Back   MacRumors Forums > Apple Systems and Services > Programming > Mac Programming

Reply
 
Thread Tools Search this Thread Display Modes
Old Mar 10, 2008, 12:38 AM   #1
jcl43
macrumors newbie
 
Join Date: Mar 2008
Getting frames into display link with CV

Hi all, I am working on an application that involves real time video editing with Core Framework in Cocoa. For testing we basically have a DVI cam hooked up to a Mac Book Pro and need to be able to edit the stream it produces. I successfully got the video into quick time using QTKit, but am having trouble aquiring the frames with a display link. The example in the Core Video guide shows it initializing the video source from file with this method:

* (id)initWithFilePath: (NSString*)theFilePath


, but I need to get the frames I can already access into the display link for OpenGL rendering and Core Image effects. If anyone has any ideas or advice I would greatly appreciate it. Thanks in advance!

Last edited by jcl43; Mar 10, 2008 at 12:55 AM.
jcl43 is offline   0 Reply With Quote
Old Mar 10, 2008, 02:36 AM   #2
Sayer
macrumors 6502a
 
Sayer's Avatar
 
Join Date: Jan 2002
Location: Austin, TX
In the Core Video Programming Guide, Core Video Tasks, at the very end of Obtaining Frames using Display Link it talks about getting frames from QuickTime. Basically you grab a frame of video as an OpenGL texture and pass it back to Core Video for manipulation somehow.

I think that is the direction you need to go (literally and figuratively).
__________________
Obama is a true statesman whose experience as a state senator, half-term US Senator & guest lecturer in a Constitutional Law class has fully prepared him to take control of our nuclear arsenal.-Me
Sayer is offline   0 Reply With Quote
Old Mar 10, 2008, 01:30 PM   #3
jcl43
Thread Starter
macrumors newbie
 
Join Date: Mar 2008
Thanks, im going to give it a try right now. When using this implementation would I set the delegate to be an NSOpenGLView? Right now i have it output via QTCaptureView. But if I want to do openGL rendering/Core Image effects on the frame does it need to be an NSOpenGLView?

Last edited by jcl43; Mar 10, 2008 at 01:46 PM.
jcl43 is offline   0 Reply With Quote
Old Mar 10, 2008, 02:24 PM   #4
jcl43
Thread Starter
macrumors newbie
 
Join Date: Mar 2008
maybe posting my code this far will help

Code:
//
//  VideoDisplayController.m
//  Prototype492
//
//  Created on 3/7/08.
//

#import "VideoDisplayController.h"

@implementation VideoDisplayController 

/*
    start the video capture session if it's not running
*/
- (IBAction)startVideoCapture:(id)sender
{

	if(![captureSession isRunning])
	{
	    [captureSession startRunning];
	}
}

/*
    stop the video capture session if it's running
*/
	
- (IBAction)stopVideoCapture:(id)sender
{
    if([captureSession isRunning])
	{
	    [captureSession stopRunning];
	}
	
}


/*
initialization of UI elements
*/
- (void)awakeFromNib
{
    NSLog(@"initializing video controller");

    if(!captureSession)	{

        //creating capture session

        captureSession = [[QTCaptureSession alloc] init];
	

	    //flags for connecting inputs and outputs
	    BOOL successConnect = NO;
	    NSError *error = nil;
	
		
	    //finding and creating the input device and adding it to the capture session
	    //if unsuccessful, send a message to error log


        //note: using QTMediaTypeMuxed for DV camera... otherwise use QTMediaTypeVideo
        videoCaptureDevice = [QTCaptureDevice defaultInputDeviceWithMediaType:QTMediaTypeMuxed];
    
	    //if the video capture device is found, attempt to open it and add it to the capture session
        if(videoCaptureDevice)
	    {
            successConnect = [videoCaptureDevice open:&error];

		
		    if(!successConnect)
		    {
            
		        NSLog(@"error opening video capture device");
                videoCaptureDevice = nil;

		        //other error handling code goes here
		    }

            captureVideoDeviceInput = [[QTCaptureDeviceInput alloc] initWithDevice:videoCaptureDevice];
     	
	
            //attempt to add capture device input to the capture session
		    successConnect = [captureSession addInput:captureVideoDeviceInput error:&error];

		    if(!successConnect)
		    {
		        NSLog(@"error adding video capture device input to capture session");
			
			    //other error handling code goes here
	        }
        
            //adding decompressed video output to capture session
			
			decompressedVideoOutput = [[QTCaptureDecompressedVideoOutput alloc] init];
			
			//assigning delegate to decompressed video output
			
			[decompressedVideoOutput setDelegate: self];
			
			successConnect = [captureSession addOutput:decompressedVideoOutput error:&error];

            if(!successConnect)
			{
			    NSLog(@"error adding decompressed video output to capture session");
				
				//other error handling code goes here
			
			}

	
            //linking capture session to the capture view
		    [captureView setCaptureSession:captureSession];
	    
		    //[captureSession startRunning];
	    }
        else
	    {
	        NSLog(@"couldn't find video capture device");
	    }
    }
}


/*
delegate method called when decompressedVideoOutput receives a frame
it is not called in the main thread, so a synhronized block is used
delegates of decompressedVideoOutput receive this message 
and then can used the provided video frame for further image processing

captureOutput - the QTCaptureDecompressedVideoOutput instance that sent the frame (decompressedVideoOutput)
videoFrame - Core Video image buffer containing the decompressed frame
sampleBuffer sample buffer containing additional info about the frame
connection - connnection from which vdieo was received
*/


- (void)captureOutput:(QTCaptureOutput *)captureOutput 
        didOutputVideoFrame:(CVImageBufferRef)videoFrame
        withSampleBuffer:(QTSampleBuffer *)sampleBuffer 
        fromConnection:(QTCaptureConnection *)connection
{
    CVImageBufferRef releasedImageBuffer;


    CVBufferRetain(videoFrame);
	

    @synchronized(self)
	{
        //basically, have frame to be released refer to the current frame
		//then update the reference to the current frame with the next frame in the "video stream"
	    releasedImageBuffer = currentImageBuffer;
        currentImageBuffer = videoFrame;
	}

    CVBufferRelease(releasedImageBuffer);

}





/*
event that's fired when window is closed
stop the session and close any opened devices
*/
- (void)windowWillClose:(NSNotification *)notification
{
    if([captureSession isRunning])
	{
	    [captureSession stopRunning];
    }
	
	if([videoCaptureDevice isOpen])
	{
	    [videoCaptureDevice close];
	}

}

/*
event that fires when memory is to be deallocated when object is killed
*/
- (void)dealloc
{

    [captureSession release];
	[captureVideoDeviceInput release];
    [decompressedVideoOutput release];
	
	[super dealloc];
}


@end


//
//  VideoDisplayController.h
//  Prototype492v1.1
//
//  Updated on 3/8/08.
//

#import <Cocoa/Cocoa.h>
#import <QTKit/QTkit.h>
#import "VideoView.h"

@interface VideoDisplayController : NSOpenGLView 
{
    QTCaptureSession *captureSession;
    QTCaptureDevice *videoCaptureDevice;
	QTCaptureDeviceInput *captureVideoDeviceInput;
    QTCaptureDecompressedVideoOutput *decompressedVideoOutput;
   
    //stores most recent frame grabbed from a CVImageBufferRef
	CVImageBufferRef currentImageBuffer;


    //this is the panel on which the video from the DV camera will be displayed
    IBOutlet QTCaptureView *captureView;


    //IBOutlet VideoView *openGLVideoView;

}



- (IBAction)startVideoCapture:(id)sender;
- (IBAction)stopVideoCapture:(id)sender;


@end
thanks again for any help

Last edited by jcl43; Mar 10, 2008 at 04:29 PM.
jcl43 is offline   0 Reply With Quote
Old Mar 12, 2008, 11:53 AM   #5
Sayer
macrumors 6502a
 
Sayer's Avatar
 
Join Date: Jan 2002
Location: Austin, TX
Yes, you use NSOpenGLView.

There is sample code called CIVideoDemoGL that does a lot of what you want to do and is easier to point to than copy/pasting it into a message board.
__________________
Obama is a true statesman whose experience as a state senator, half-term US Senator & guest lecturer in a Constitutional Law class has fully prepared him to take control of our nuclear arsenal.-Me
Sayer is offline   0 Reply With Quote
Old Mar 13, 2008, 12:18 AM   #6
jcl43
Thread Starter
macrumors newbie
 
Join Date: Mar 2008
Thanks for the info. That seriously helped me out a ton. I still need to figure out how to get the frames from the dvi came...in this example it opens from a file i believe.
jcl43 is offline   0 Reply With Quote
Old Mar 13, 2008, 04:27 PM   #7
Sayer
macrumors 6502a
 
Sayer's Avatar
 
Join Date: Jan 2002
Location: Austin, TX
QTKit now has video capture support in Leopard and its fairly easy to do.

Lemme look for the sample code I was studying earlier (for a big-time pro-level contract job with a rather large company).

Working...

The sample code is called "StillImage" but it does a lot of what you want e.g. capturing from a live video source and doing something with the frames. Here's some of it:

Code:
        mCaptureSession = [[QTCaptureSession alloc] init];
        
        // Find a video device
        QTCaptureDevice *device = [QTCaptureDevice defaultInputDeviceWithMediaType:QTMediaTypeVideo];
        success = [device open:&error];
        if (!success) {
            [[NSAlert alertWithError:error] runModal];
            return;
        }
        
        // Add a device input for that device to the capture session
        mCaptureDeviceInput = [[QTCaptureDeviceInput alloc] initWithDevice:device];
        success = [mCaptureSession addInput:mCaptureDeviceInput error:&error];
        if (!success) {
            [[NSAlert alertWithError:error] runModal];
            return;
        }
        
        // Add a decompressed video output that returns raw frames to the session
        mCaptureDecompressedVideoOutput = [[QTCaptureDecompressedVideoOutput alloc] init];
        [mCaptureDecompressedVideoOutput setDelegate:self];
        success = [mCaptureSession addOutput:mCaptureDecompressedVideoOutput error:&error];
        if (!success) {
            [[NSAlert alertWithError:error] runModal];
            return;
        }
        
        // Preview the video from the session in the document window
        [mCaptureView setCaptureSession:mCaptureSession];
        
        // Start the session
        [mCaptureSession startRunning];
Note that QTKit's video capture is Leopard only (for use in Cocoa apps anyway).

In this implementation you create a few delegate methods (callbacks) that hand off a frame of raw decompressed video that you can manipulate.

Code:
// This delegate method is called whenever the QTCaptureDecompressedVideoOutput receives a frame
- (void)captureOutput:(QTCaptureOutput *)captureOutput didOutputVideoFrame:(CVImageBufferRef)videoFrame withSampleBuffer:(QTSampleBuffer *)sampleBuffer fromConnection:(QTCaptureConnection *)connection
{
    // Store the latest frame
	// This must be done in a @synchronized block because this delegate method is not called on the main thread
    CVImageBufferRef imageBufferToRelease;
    
    CVBufferRetain(videoFrame);
    
    @synchronized (self) {
        imageBufferToRelease = mCurrentImageBuffer;
        mCurrentImageBuffer = videoFrame;
    }
    
    CVBufferRelease(imageBufferToRelease);
}
__________________
Obama is a true statesman whose experience as a state senator, half-term US Senator & guest lecturer in a Constitutional Law class has fully prepared him to take control of our nuclear arsenal.-Me
Sayer is offline   0 Reply With Quote
Old Mar 14, 2008, 08:04 AM   #8
fantastico
macrumors newbie
 
Join Date: Mar 2008
Sayer, from what I'm seeing in jcl43's code, he is already using QTKit to capture the frames from a DV Cam, and even storing the current frame in a CVBufferRef...very similar to what you just posted.

In fact, in the original post, he states:I SUCCESSFULLY GOT THE VIDEO INTO QUICK TIME USING QTKIT, but am having trouble aquiring the frames with a display link.

Last edited by fantastico; Mar 14, 2008 at 08:31 AM.
fantastico is offline   0 Reply With Quote

Reply
MacRumors Forums > Apple Systems and Services > Programming > Mac Programming

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
thread Thread Starter Forum Replies Last Post
Why do many people have problems with the min display port to dual link DVI adapter? Nostromo Mac Peripherals 3 Jul 7, 2011 10:40 AM
Right Clicking on Links in Safari doesn't display "Open Link in New Tab" Option iPud OS X 0 Jan 30, 2011 07:03 PM
Will John McCain get us into a war with Iran? MACDRIVE Politics, Religion, Social Issues 23 Feb 18, 2008 02:11 PM
help with frames into video p8ntballguy Digital Video 0 Jan 20, 2007 07:07 PM
How do I keep a site's header in the top frame when visiting links?? h0kie99 Web Design and Development (archive) 5 Jan 26, 2005 04:12 PM


All times are GMT -5. The time now is 12:35 PM.

Mac Rumors | Mac | iPhone | iPhone Game Reviews | iPhone Apps

Mobile Version | Fixed | Fluid | Fluid HD
Copyright 2002-2013, MacRumors.com, LLC