iOS Capture video simultaneously from both front and rear camera

knonk said:

Thanks for your information but did you contact with Apple and they did confirm you that ? Sorry for asking you again because I need to make sure and I have to have a solution for this problem.

If you are right, so the (1) problem is impossible, but I still face the second problem, that is save 2 video file to 1 with PIP effect.
Do you know how to do that in IPhone 4? Any framework help me do that ?

Click to expand...

knonk said:

Hi, Thanks for your advise, but I don't have premium account on App Dev Forum, I can't login there.

Click to expand...

firewood said:

You will need a paid Developer enrollment to develop for any non-jailbroken iPhone device. So you might as well get one.

Click to expand...

knonk said:

Hi, Thanks for your advise, but I don't have premium account on App Dev Forum, I can't login there. Could u repost that usefull thread here ?

Click to expand...

- (id) initWithViewForPreview:(UIImageView *) aView
{
     if ([super init])
     {
          self.previewView = aView;
          self.captureSession = [[AVCaptureSession alloc] init];
          self.captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
          NSError *error = nil;
          AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:self.captureDevice error:&error];
          if (input)
          {
               [self.captureSession addInput:input];
          }
          else
          {
               NSLog(@"Error creating video input device");
          }
          AVCaptureVideoDataOutput *outputData = [[[AVCaptureVideoDataOutput alloc] init] autorelease];
          [outputData setSampleBufferDelegate:self queue:dispatch_queue_create("renderqueue",NULL)];

          // Set the video output to store frame in BGRA (It is supposed to be faster)
          NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
          NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA];
          NSDictionary* videoSettings = [NSDictionary dictionaryWithObject:value forKey:key];
          [outputData setVideoSettings:videoSettings];              
          [self.captureSession addOutput:outputData];
          isRecording = NO;
     }
     return self;
}

- (void) toggleRecording
{
     if (isRecording)
     {
          NSLog(@"Stopping recording");
          [self.assetWriterInput markAsFinished];
          [self.assetWriter endSessionAtSourceTime:recordStartTime];
          [self.assetWriter finishWriting];
          NSLog(@"Export done");
     }
     else
     {
          NSLog(@"Starting to record");
          NSError *error = nil;
          NSURL *outputPath = [self tempFileURL];
          if (![outputPath isFileURL])
          {
               NSLog(@"Not file URL");
          }
          self.assetWriter = [AVAssetWriter assetWriterWithURL:outputPath fileType:AVFileTypeQuickTimeMovie  error:&error];
          if (error != nil)
          {
               NSLog(@"Creation of assetWriter resulting in a non-nil error");
               NSLog([error localizedDescription]);
               NSLog([error localizedFailureReason]); 
          }    
          NSMutableDictionary *d=[[NSMutableDictionary alloc] init];
          [d setValue: AVVideoCodecH264 forKey: AVVideoCodecKey];
          [d setValue:[NSNumber numberWithInt:1280] forKey:AVVideoWidthKey];
          [d setValue:[NSNumber numberWithInt:720] forKey:AVVideoHeightKey];
          self.assetWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:d];
          if (self.assetWriterInput == nil)
          {
               NSLog(@"assetWriterInput is nil");
          }
          //self.assetWriterInput.expectsMediaDataInRealTime = YES; // If you uncomment this you get an exception saying it's not implemented yet (this may well not be true anymore: this was written on a very early 4.1 beta
          [self.assetWriter addInput:self.assetWriterInput];
          [self.assetWriter startWriting];
          [self.assetWriter startSessionAtSourceTime:recordStartTime];
     }
     isRecording = !isRecording;
}

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
     if (!CMSampleBufferDataIsReady(sampleBuffer))
     {
          NSLog(@"sampleBuffer data is not ready");
     }

     CMTime timeNow = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
     CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer); 

     // Lock the image buffer
     CVPixelBufferLockBaseAddress(imageBuffer,0); 

     // Get information about the image
     uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer); 
     size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer); 
     size_t width = CVPixelBufferGetWidth(imageBuffer); 
     size_t height = CVPixelBufferGetHeight(imageBuffer); 

     // Create a CGImageRef from the CVImageBufferRef
     CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB(); 
     CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst); 

      // Temp: draw a black rect: replace the next 2 lines with the correct compositing that you want.
      CGContextSetFillColorWithColor(newContext, [[UIColor blackColor] CGColor]);
      CGContextFillRect(newContext, CGRectMake(0, 0, 400, 400));

     // We unlock the  image buffer
     CVPixelBufferUnlockBaseAddress(imageBuffer,0);

     // We release some components
     CGContextRelease(newContext); 
     CGColorSpaceRelease(colorSpace);
     if (isRecording)
     {
          if (![self.assetWriterInput isReadyForMoreMediaData])
          {
               NSLog(@"Not ready for data :(");
          }
          NSLog(@"Trying to append");
          if (![self.assetWriterInput appendSampleBuffer:sampleBuffer])
          {
               NSLog(@"Failed to append pixel buffer");
          }
          else 
          {
               NSLog(@"Append worked");
          }
     }
     recordStartTime = timeNow;
}

robbieduncan said:

I certainly can't repost the whole thread: I only have any sort of right to repost my own contributions. Fortunately for you they are the useful bit. The code below demonstrates how to get per-pixel data of the video stream, composite something onto it (in this case a black square) and save the video.

Note: you are expected to read, understand and if necessary research anything you don't understand yourself. I'm not answering any questions on this.

Before I post the code some setup: This is all in a class that manages the AV stuff. I have a button on screen to start/stop recording. When this is touched the toggleRecording method is called.

Code:

- (id) initWithViewForPreview:(UIImageView *) aView
{
     if ([super init])
     {
          self.previewView = aView;
          self.captureSession = [[AVCaptureSession alloc] init];
          self.captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
          NSError *error = nil;
          AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:self.captureDevice error:&error];
          if (input)
          {
               [self.captureSession addInput:input];
          }
          else
          {
               NSLog(@"Error creating video input device");
          }
          AVCaptureVideoDataOutput *outputData = [[[AVCaptureVideoDataOutput alloc] init] autorelease];
          [outputData setSampleBufferDelegate:self queue:dispatch_queue_create("renderqueue",NULL)];

          // Set the video output to store frame in BGRA (It is supposed to be faster)
          NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
          NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA];
          NSDictionary* videoSettings = [NSDictionary dictionaryWithObject:value forKey:key];
          [outputData setVideoSettings:videoSettings];             
          [self.captureSession addOutput:outputData];
          isRecording = NO;
     }
     return self;
}

The above has some issues, but works. This is the action for the button to start/stop recording:

Code:

- (void) toggleRecording
{
     if (isRecording)
     {
          NSLog(@"Stopping recording");
          [self.assetWriterInput markAsFinished];
          [self.assetWriter endSessionAtSourceTime:recordStartTime];
          [self.assetWriter finishWriting];
          NSLog(@"Export done");
     }
     else
     {
          NSLog(@"Starting to record");
          NSError *error = nil;
          NSURL *outputPath = [self tempFileURL];
          if (![outputPath isFileURL])
          {
               NSLog(@"Not file URL");
          }
          self.assetWriter = [AVAssetWriter assetWriterWithURL:outputPath fileType:AVFileTypeQuickTimeMovie  error:&error];
          if (error != nil)
          {
               NSLog(@"Creation of assetWriter resulting in a non-nil error");
               NSLog([error localizedDescription]);
               NSLog([error localizedFailureReason]);
          }   
          NSMutableDictionary *d=[[NSMutableDictionary alloc] init];
          [d setValue: AVVideoCodecH264 forKey: AVVideoCodecKey];
          [d setValue:[NSNumber numberWithInt:1280] forKey:AVVideoWidthKey];
          [d setValue:[NSNumber numberWithInt:720] forKey:AVVideoHeightKey];
          self.assetWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:d];
          if (self.assetWriterInput == nil)
          {
               NSLog(@"assetWriterInput is nil");
          }
          //self.assetWriterInput.expectsMediaDataInRealTime = YES; // If you uncomment this you get an exception saying it's not implemented yet (this may well not be true anymore: this was written on a very early 4.1 beta
          [self.assetWriter addInput:self.assetWriterInput];
          [self.assetWriter startWriting];
          [self.assetWriter startSessionAtSourceTime:recordStartTime];
     }
     isRecording = !isRecording;
}

Finally we have a callback that we can use to get each frame as it becomes available

Code:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
     if (!CMSampleBufferDataIsReady(sampleBuffer))
     {
          NSLog(@"sampleBuffer data is not ready");
     }

     CMTime timeNow = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
     CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

     // Lock the image buffer
     CVPixelBufferLockBaseAddress(imageBuffer,0);

     // Get information about the image
     uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
     size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
     size_t width = CVPixelBufferGetWidth(imageBuffer);
     size_t height = CVPixelBufferGetHeight(imageBuffer);

     // Create a CGImageRef from the CVImageBufferRef
     CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
     CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);

      // Temp: draw a black rect: replace the next 2 lines with the correct compositing that you want.
      CGContextSetFillColorWithColor(newContext, [[UIColor blackColor] CGColor]);
      CGContextFillRect(newContext, CGRectMake(0, 0, 400, 400));

     // We unlock the  image buffer
     CVPixelBufferUnlockBaseAddress(imageBuffer,0);

     // We release some components
     CGContextRelease(newContext);
     CGColorSpaceRelease(colorSpace);
     if (isRecording)
     {
          if (![self.assetWriterInput isReadyForMoreMediaData])
          {
               NSLog(@"Not ready for data :(");
          }
          NSLog(@"Trying to append");
          if (![self.assetWriterInput appendSampleBuffer:sampleBuffer])
          {
               NSLog(@"Failed to append pixel buffer");
          }
          else
          {
               NSLog(@"Append worked");
          }
     }
     recordStartTime = timeNow;
}

As I said: you either do the research to understand this code or you don't. This is the total extent of the help I am willing to give.

Click to expand...