VitalSnap Permissions

OCR View Controller

The VLDOCRViewController provides a simple interface to optically scan a peripheral and provide the resulting reading. This view controller presents a prepackaged view and is typically displayed modally. It presents a camera view with an overlay appropriate to the peripheral being scanned. Partial results are displayed in the view while recognition is in progress. When it converges on a value, the delegate method ocrViewController:didCompleteReading:image:metadata: is called with the results. An example application called "OCR example" is available to illustrate the VLDOCRViewController API. It is written in Swift and is provided within the "Example Apps/Swift OCR" directory of the library release package.

The app must have permission to access the camera. Permission will be requested on first launch of the VLDOCRViewController. If the library is unable to access the camera, the delegate method ocrViewControllerWasDeniedCameraAuthorization: is called.

The Info.plist file must also include a Camera Usage Description.

The presented view provides a button to cancel OCR. If the user cancels, the delegate method ocrViewControllerDidCancel: is called.

When any of the three delegate methods are invoked, the view controller should be dismissed.

The following example constructs a VLDOCRViewController for the designated peripheral and presents it.

let peripheral = VLDOCRPeripheral(forID: 3)
if let controller = VLDOCRViewController(ocrPeripheral: peripheral) {
   controller.delegate = self
   self.present(controller, animated: true, completion: nil)
}

Delegate callbacks dismiss the view controller and handle the returned record. The recognized value returned in the delegate call back should be displayed to the user so that they can verify the value and modify it if incorrect. The image that was recognized is also provided and can be displayed to the user to verify the value.

func ocrViewControllerDidCancel(_ viewController: VLDOCRViewController) {
  viewController.dismiss(animated: true, completion: nil);
}

func ocrViewControllerWasDeniedCameraAuthorization(_ viewController: VLDOCRViewController) {
    print("Unable to access camera")
    viewController.dismiss(animated: true, completion: nil);
}

func ocrViewController(_ viewController: VLDOCRViewController, didCompleteReading record: VLDInformRecord?, image: UIImage?, metadata: [AnyHashable : Any]?) {
    // Display the value and image, allow the user to correct
    viewController.dismiss(animated: true, completion: nil);
}

The record received from the OCR View Controller should be verified by the user and then submitted to the Validic server.

// After verification, queue the record and image to be uploaded to the server
VLDSession.sharedInstance().submitInformRecord(record, image: image)

Runtime unit selection

For any glucose meter in our lineup of supported meters, you can now specify mmol/L or mg/dL at runtime for a given reading. If no unit is provided, mg/dL is assumed.

For the Zewa 11110 thermometer, you can specify Celsius or Fahrenheit at runtime for a given reading. If no unit is provided, Fahrenheit is assumed.

An example using the VLDOCRViewController:

let peripheral = VLDOCRPeripheral(forID: 9)
ocrViewController = VLDOCRViewController(ocrPeripheral: peripheral, glucoseUnit: .MMOLL)

OCR - Custom view (optional)

If more control over the OCR view is needed, a custom view can be implemented to perform OCR using the VLDOCRController class. The simpler VLDOCRViewController is recommended unless customizing the view is required. An example application called "VitalSnap example" is available to illustrate the VLDOCRController API. It is written in Swift and is provided within the "Example apps/Swift VitalSnap" directory in the library release package.

Process Overview

An instance of the VLDOCRController class is used to manage the recognition process. A VLDOCRPeripheral represents the peripheral being scanned. The controller is initialized with a peripheral object.

The recognition process involves a video camera preview layer and an overlay image, both obtained from the controller. The overlay image must be precisely positioned over the preview layer for proper alignment and recognition using a frame provided by the controller.

The app creates a view hierarchy where a parent view contains two subviews (preview and overlay). The controller is configured with the size of the preview layer and it generates the appropriate frame for the overlay image view.

OCR processing begins as soon as the controller is initialized. A delegate is called with intermediate and final results. The camera capture session ends when the controller is deallocated.

View Structure

Two different approaches can be used to structure views for the preview and overlay, using either a UIImageView or CALayer for the overlay image.

In the preferred view-based approach, a parent view contains two subviews: one for the preview and the other for the overlay image. The preview layer is obtained from the VLDOCRController and added as a sublayer to the preview view. Once this layer is added, no subviews of the preview view will be visible. The overlay view, typically a UIImageView, and sibling of the overlay view, contains the overlay image obtained from the VLDOCRController, positioned to the frame specified by the controller. The origin of both views is assumed to be [0, 0] within their parent. The parent and and preview view are typically full screen.

In the alternate approach, a single view is used. The preview layer is added to this view and an additional CALayer is created, the overlay image is set on that CALayer, and it is added as another sublayer to the same view. The overlay layer's frame must be set to the frame given by the VLDOCRController.

Initializing VLDOCRController

The VLDOCRController requires a VLDOCRPeripheral for its initializers.

Once a peripheral is obtained, construct the VLDOCRController and assign its delegate.

// Maintain a reference to the controller
var controller: VLDOCRController?
...
// Get a peripheral
let peripheral = VLDOCRPeripheral(forID: 9)
// Initialize the controller
self.controller = VLDOCRController(ocrPeripheral: peripheral)
// Assign a delegate
self.controller.delegate = self

Preview

The camera preview layer is obtained from VLDOCRController and is added as a sublayer to a view within a view hierarchy as described under View Structure.

// Property or IBOutlet
@IBOutlet var previewView: UIView!

// Typically set up in ViewDidLoad
override func viewDidLoad() {
  if let previewLayer: AVCaptureVideoPreviewLayer = controller.previewLayer {
      previewView.layer.addSublayer(previewLayer)

Set the previewLayer's frame to match those of its containing preview view. Typically within viewDidLayoutSubviews:

override func viewDidLayoutSubviews() {
  self.controller.previewLayer?.frame = self.previewView.bounds

Overlay in a view

An overlay image is displayed over the preview layer using a specific frame calculated by VLDOCRController. The image is obtained from the VLDOCRPeripheral and used to initialize the VLDOCRController.

@IBOutlet var overlayView: UIImageView!

override func viewDidLoad() {
  // let peripheral: VLDOCRPeripheral = ...
  overlayView.image = peripheral.overlayImage()

Whenever the preview layer's frame changes, the controller needs to be informed by invoking configureForPreviewLayerSize:. After this call, the overlayView's frame should be set using the overlayFrame property of the VLDOCRController.

override func viewDidLayoutSubviews() {
  // Configure VLDOCRController with the current preview layer size
  self.controller.configureForPreviewLayerSize(previewLayer.bounds.size)
  self.overlayView.frame = self.controller.overlayFrame

Process Results

During OCR processing, methods on a delegate conforming to the VLDOCRControllerDelegate protocol are invoked.

ocrController:didProcessResult: is invoked for each camera frame captured and provides intermediate results. The VLDOCRResult object contains the current recognized string, an object describing possible glare, and the cropped image that was associated with this incomplete result. The result string can be displayed to the user as an indication of what portion of the display is not being recognized. Depending on the peripheral, the result string may contain linefeeds representing multiple lines being recognized.

func ocrController(_ ocrController: VLDOCRController, didProcessResult result: VLDOCRResult?) {
    if let result = result {
        NSLog("Partial result \(result.resultString)")
        let resultImage = result.image
    }
}

The ocrController:didCompleteReading:image:forPeripheral:metadata: delegate method is invoked when OCR processing has completed with reasonably high confidence.

func ocrController(ocrController: VLDOCRController!, didCompleteReading record: VLDInformRecord!, image: UIImage!, forPeripheral peripheral: VLDOCRPeripheral!, metadata: [NSObject : AnyObject]!) {
    // Obtain fields from the record to display to the user
    if peripheral.type == VLDPeripheralType.glucoseMeter {
        if let glucoseMetric = record.metric(withType: .bloodGlucose) {
            let bloodGlucose: NSNumber? = glucoseMetric.value

            // Obtain the captured image, display to the user for verification of reading
            var verificationImageView: UIImageView
            verificationImageView.image = image
        }
    }
}

The delegate is passed a VLDInformRecord subclass appropriate for the peripheral, the matching cropped preview image, and additional metadata. The recognized values returned in the record should be visually validated by the user. The cropped preview image can be displayed to the user to validate the recognized values before uploading to the server.

When the user approves of the values, the record can be submitted:

// After verification, queue the record and image to be uploaded to the server
VLDSession.sharedInstance().submitInformRecord(record, image: image)

OCR processing lifecycle

OCR processing commences when the VLDOCRController is instantiated. The camera preview session is stopped when the VLDOCRController is deallocated. OCR processing stops when the final result delegate method is invoked or when the controller is deallocated. To restart or to work with a different peripheral, construct a new VLDOCRController.

Alternate: Overlay in a CALayer

Instead of placing the overlay image within a sibling view in a view hierarchy, the overlay image can be displayed in a CALayer added to the same view containing the preview layer. A separate view for the overlay is unnecessary.

var overlayLayer:CALayer?
...
override func viewDidLoad() {
    self.overlayLayer = CALayer()
    self.overlayLayer.contents = peripheral.overlayImage()?.cgImage
    self.previewLayer.addSublayer(overlayLayer)

The overlayLayer's frame needs to be set when the containing view's frame changes:

override func viewDidLayoutSubviews() {
    self.controller.configure(forPreviewLayerSize:(self.controller.previewLayer.bounds.size)!)
    self.overlayLayer.frame = self.controller.overlayFrame