iOS uses a cellphone camera to measure heart rate
- 2020-12-07 04:27:40
- OfStack
The principle of
Simple introduction 1, you can find it on the Internet a lot about mobile phone heart rate measurement of this project, probably is: put the finger on the camera and flash light, through the fingers in pulse congestion in slight color change to determine the heartbeat, determine the wave trough, according to the time difference between two wave to determine the instantaneous heart rate.
Train of thought
First, we collected the video stream and converted it to the HSV color set according to the RGB color we got. In fact, we only used the H of HSV.
I will do some processing on the H, depending on people's preferences or specific conditions, mainly for the following line chart and calculation of instantaneous heart rate. If I have the ability, I can process 1 noise data, because a slight finger wobble may cause 1 unstable data.
According to the processed H, the line graph can be drawn. I have bound the processed H to the timestamp to calculate the heart rate later.
The crests and troughs were determined according to the treated H, and the heart rate was calculated by the time difference between the two troughs.
implementation
The general idea is such as above, let's look at the concrete implementation of 1 code below.
1. First, I initialized some data for later use
// equipment
@property (strong, nonatomic) AVCaptureDevice *device;
// Combined input and output
@property (strong, nonatomic) AVCaptureSession *session;
// Input devices
@property (strong, nonatomic) AVCaptureDeviceInput *input;
// Output devices
@property (strong, nonatomic) AVCaptureVideoDataOutput *output;
// All the points of the output
@property (strong, nonatomic) NSMutableArray *points;
// Record before floating point changes 1 The value of the time
static float lastH = 0;
// Used to determine if it is a control 1 A point value
static int count = 0;
// Initialize the
self.device = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
self.session = [[AVCaptureSession alloc]init];
self.input = [[AVCaptureDeviceInput alloc]initWithDevice:self.device error:nil];
self.output = [[AVCaptureVideoDataOutput alloc]init];
self.points = [[NSMutableArray alloc]init];
2. Set the video capture stream. In order to save memory, I did not output the video screen
// Flash on
if ([self.device isTorchModeSupported:AVCaptureTorchModeOn]) {
[self.device lockForConfiguration:nil];
// Flash on
self.device.torchMode=AVCaptureTorchModeOn;
// Turn down flash brightness (to reduce memory footprint and avoid burning hot for a long time)
[self.device setTorchModeOnWithLevel:0.01 error:nil];
[self.device unlockForConfiguration];
}
// Start the configuration input output
[self.session beginConfiguration];
// Set the pixel output format
NSNumber *BGRA32Format = [NSNumber numberWithInt:kCVPixelFormatType_32BGRA];
NSDictionary *setting =@{(id)kCVPixelBufferPixelFormatTypeKey:BGRA32Format};
[self.output setVideoSettings:setting];
// Discard delayed frames
[self.output setAlwaysDiscardsLateVideoFrames:YES];
// Turn on the child thread of camera capture image output
dispatch_queue_t outputQueue = dispatch_queue_create("VideoDataOutputQueue", DISPATCH_QUEUE_SERIAL);
// Sets the child thread to execute the proxy method
[self.output setSampleBufferDelegate:self queue:outputQueue];
// to session add
if ([self.session canAddInput:self.input]) [self.session addInput:self.input];
if ([self.session canAddOutput:self.output]) [self.session addOutput:self.output];
// Reduced resolution, reduced sampling rate (to reduce memory footprint)
self.session.sessionPreset = AVCaptureSessionPreset1280x720;
// Set the minimum video frame output interval
self.device.activeVideoMinFrameDuration = CMTimeMake(1, 10);
// With the current output Initialize the connection
AVCaptureConnection *connection =[self.output connectionWithMediaType:AVMediaTypeVideo];
[connection setVideoOrientation:AVCaptureVideoOrientationPortrait];
// The editing
[self.session commitConfiguration];
// Began to run
[self.session startRunning];
Here I've reduced the flash brightness, reduced the resolution, and reduced the number of frames per second output. The main purpose is to reduce the memory footprint. (I only have one 6 in my hand, so I don't test other equipment.)
3. Collect video stream in output's proxy method
// captureOutput-> The current output sampleBuffer-> The sample buffer connection-> Capture the connection
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
// Get layer buffering
CVPixelBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
uint8_t*buf = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
float r = 0, g = 0,b = 0;
float h,s,v;
// To calculate RGB
TORGB(buf, width, height, bytesPerRow, &r, &g, &b);
// RGB turn HSV
RGBtoHSV(r, g, b, &h, &s, &v);
// Gets the current timestamp (accurate to milliseconds)
double t = [[NSDate date] timeIntervalSince1970]*1000;
// Returns the processed floating point value
float p = HeartRate(h);
// Bind floating-point and timestamp
NSDictionary *point = @{[NSNumber numberWithDouble:t]:[NSNumber numberWithFloat:p]};
// Now you can calculate your heart rate or draw a heart rate graph based on your personal situation
}
Now that the data has been processed, we can draw a line graph based on the data or calculate the heart rate
Calculate RGB
void TORGB (uint8_t *buf, float ww, float hh, size_t pr, float *r, float *g, float *b) {
float wh = (float)(ww * hh );
for(int y = 0; y < hh; y++) {
for(int x = 0; x < ww * 4; x += 4) {
*b += buf[x];
*g += buf[x+1];
*r += buf[x+2];
}
buf += pr;
}
*r /= 255 * wh;
*g /= 255 * wh;
*b /= 255 * wh;
}
Turn RGB HSV
void RGBtoHSV( float r, float g, float b, float *h, float *s, float *v ) {
float min, max, delta;
min = MIN( r, MIN(g, b ));
max = MAX( r, MAX(g, b ));
*v = max;
delta = max - min;
if( max != 0 )
*s = delta / max;
else {
*s = 0;
*h = -1;
return;
}
if( r == max )
*h = ( g - b ) / delta;
else if( g == max )
*h = 2 + (b - r) / delta;
else
*h = 4 + (r - g) / delta;
*h *= 60;
if( *h < 0 )
*h += 360;
}
Handle floating-point as per h
float HeartRate (float h) {
float low = 0;
count++;
lastH = (count==1)?h:lastH;
low = (h-lastH);
lastH = h;
return low;
}
4. Analyze data and calculate heart rate
Here I tangled for a long time, tried several different methods, there is no more than an ideal result, the calculation is particularly inaccurate. Later saw http: / / ios jobbole. com / 88158 / this article, the optimization of one part of the pitch algorithm, is unknown, but sleep Shirley, expressed thanks very much. Woo hoo.
Principle: that is to say, draw a period of time, find a minimum peak in this period, and then determine a period, and then find a minimum peak in the interval of 0.5 period before this peak and the interval of 0.5 period after this peak. These values are then used to determine the instantaneous heart rate.
- (void)analysisPointsWith:(NSDictionary *)point {
[self.points addObject:point];
if (self.points.count<=30) return;
int count = (int)self.points.count;
if (self.points.count%10 == 0) {
int d_i_c = 0; // The location of the lowest peak Let's say it's in the middle c->center
int d_i_l = 0; // The lowest peak position on the left side of the lowest peak l->left
int d_i_r = 0; // The lowest peak to the right of the lowest peak r->right
float trough_c = 0; // A floating-point value with the lowest peak value
float trough_l = 0; // The lowest peak floating point value on the left side of the lowest peak
float trough_r = 0; // The lowest peak floating point value to the right of the lowest peak
// 1. Determine the lowest peak in the data first
for (int i = 0; i < count; i++) {
float trough = [[[self.points[i] allObjects] firstObject] floatValue];
if (trough < trough_c) {
trough_c = trough;
d_i_c = i;
}
}
// 2. After finding the lowest peak Center on the lowest peak Before the find 0.5-1.5 The lowest peak in a cycle And after 0.5-1.5 The lowest peak of the period
if (d_i_c >= 1.5*T) {
// a. If the lowest peak is in the center, That is, at least the distance before and after 1.5 A cycle
if (d_i_c <= count-1.5*T) {
// Lowest peak on the left
for (int j = d_i_c - 0.5*T; j > d_i_c - 1.5*T; j--) {
float trough = [[[self.points[j] allObjects] firstObject] floatValue];
if (trough < trough_l) {
trough_l = trough;
d_i_l = j;
}
}
// Lowest peak on the right
for (int k = d_i_c + 0.5*T; k < d_i_c + 1.5*T; k++) {
float trough = [[[self.points[k] allObjects] firstObject] floatValue];
if (trough < trough_r) {
trough_r = trough;
d_i_r = k;
}
}
}
// b. If the lowest peak is not enough to the right 1.5 A cycle There are two cases Not enough 0.5 Period and enough 0.5 A cycle
else {
// b.1 enough 0.5 A cycle
if (d_i_c <count-0.5*T) {
// Lowest peak on the left
for (int j = d_i_c - 0.5*T; j > d_i_c - 1.5*T; j--) {
float trough = [[[self.points[j] allObjects] firstObject] floatValue];
if (trough < trough_l) {
trough_l = trough;
d_i_l = j;
}
}
// Lowest peak on the right
for (int k = d_i_c + 0.5*T; k < count; k++) {
float trough = [[[self.points[k] allObjects] firstObject] floatValue];
if (trough < trough_r) {
trough_r = trough;
d_i_r = k;
}
}
}
// b.2 Not enough 0.5 A cycle
else {
// Lowest peak on the left
for (int j = d_i_c - 0.5*T; j > d_i_c - 1.5*T; j--) {
float trough = [[[self.points[j] allObjects] firstObject] floatValue];
if (trough < trough_l) {
trough_l = trough;
d_i_l = j;
}
}
}
}
}
// c. If the left side is not enough 1.5 A cycle 1 There are two cases enough 0.5 A cycle Not enough 0.5 A cycle
else {
// c.1 enough 0.5 A cycle
if (d_i_c>0.5*T) {
// Lowest peak on the left
for (int j = d_i_c - 0.5*T; j > 0; j--) {
float trough = [[[self.points[j] allObjects] firstObject] floatValue];
if (trough < trough_l) {
trough_l = trough;
d_i_l = j;
}
}
// Lowest peak on the right
for (int k = d_i_c + 0.5*T; k < d_i_c + 1.5*T; k++) {
float trough = [[[self.points[k] allObjects] firstObject] floatValue];
if (trough < trough_r) {
trough_r = trough;
d_i_r = k;
}
}
}
// c.2 Not enough 0.5 A cycle
else {
// Lowest peak on the right
for (int k = d_i_c + 0.5*T; k < d_i_c + 1.5*T; k++) {
float trough = [[[self.points[k] allObjects] firstObject] floatValue];
if (trough < trough_r) {
trough_r = trough;
d_i_r = k;
}
}
}
}
// 3. Determine which 1 Is closer to the lowest peak Use the closest one 1 The instantaneous heart rate was measured at the lowest peaks 60*1000 The time difference between the two peaks
if (trough_l-trough_c < trough_r-trough_c) {
NSDictionary *point_c = self.points[d_i_c];
NSDictionary *point_l = self.points[d_i_l];
double t_c = [[[point_c allKeys] firstObject] doubleValue];
double t_l = [[[point_l allKeys] firstObject] doubleValue];
NSInteger fre = (NSInteger)(60*1000)/(t_c - t_l);
if (self.frequency)
self.frequency(fre);
if ([self.delegate respondsToSelector:@selector(startHeartDelegateRateFrequency:)])
[self.delegate startHeartDelegateRateFrequency:fre];
} else {
NSDictionary *point_c = self.points[d_i_c];
NSDictionary *point_r = self.points[d_i_r];
double t_c = [[[point_c allKeys] firstObject] doubleValue];
double t_r = [[[point_r allKeys] firstObject] doubleValue];
NSInteger fre = (NSInteger)(60*1000)/(t_r - t_c);
if (self.frequency)
self.frequency(fre);
if ([self.delegate respondsToSelector:@selector(startHeartDelegateRateFrequency:)])
[self.delegate startHeartDelegateRateFrequency:fre];
}
// 4. Delete expired data
for (int i = 0; i< 10; i++) {
[self.points removeObjectAtIndex:0];
}
}
}
At present, I deal with it in this way. Later, I used the time difference between the two peaks before and after the peak and the peak with the lowest peak closest to the peak. I measured it several times and compared it with other app for 1 time. (In the case of relatively stable data, if there is a better method, please recommend it, thank you)
5. CoreGraphics is used here
PS: First, use this CoreGraphics inside of View and in View's drawRect: method, otherwise you won't get the canvas. I set up a separate UIView class to encapsulate it.
First of all, data. How do you draw without data
@property (strong, nonatomic) NSMutableArray *points;
// in init Initializes the array
self.points = [[NSMutableArray alloc]init];
// This can be translated, also in init In the
self.clearsContextBeforeDrawing = YES;
// External call method
- (void)drawRateWithPoint:(NSNumber *)point {
// Flashbacks insert arrays
[self.points insertObject:point atIndex:0];
// Delete overflow screen data
if (self.points.count > self.frame.size.width/6) {
[self.points removeLastObject];
}
dispatch_async(dispatch_get_main_queue(), ^{
// This method calls automatically drawRect: methods
[self setNeedsDisplay];
});
}
Before I called setNeedsDisplay,1 did not go drawRect: method, or just went straight once, and then went to Baidu means that setNeedsDisplay will execute drawRect: when the system is idle, and then I tried to go back to the main thread call, it was ok. The exact reason is not clear, but it could also be that View is being modified in the main thread.
b. The way to draw the line, how to adjust the specific mood of the individual.
CGFloat ww = self.frame.size.width;
CGFloat hh = self.frame.size.height;
CGFloat pos_x = ww;
CGFloat pos_y = hh/2;
// Get the current canvas
CGContextRef context = UIGraphicsGetCurrentContext();
// The line width
CGContextSetLineWidth(context, 1.0);
// anti-alias
//CGContextSetAllowsAntialiasing(context,false);
// Line color
CGContextSetStrokeColorWithColor(context, [UIColor redColor].CGColor);
CGContextMoveToPoint(context, pos_x, pos_y);
for (int i = 0; i < self.points.count; i++) {
float h = [self.points[i] floatValue];
pos_y = hh/2 + (h * hh/2) ;
CGContextAddLineToPoint(context, pos_x, pos_y);
pos_x -=6;
}
CGContextStrokePath(context);
c. To look good, I added the grid, which is also called in drawRect:
static CGFloat grid_w = 30.0f;
- (void)buildGrid {
CGFloat wight = self.frame.size.width;
CGFloat height = self.frame.size.height;
// Get the current canvas
CGContextRef context = UIGraphicsGetCurrentContext();
CGFloat pos_x = 0.0f;
CGFloat pos_y = 0.0f;
// in wight Draw vertical lines in the range
while (pos_x < wight) {
// Set the grid line width
CGContextSetLineWidth(context, 0.2);
// Set the grid line color
CGContextSetStrokeColorWithColor(context, [UIColor greenColor].CGColor);
// The starting point
CGContextMoveToPoint(context, pos_x, 1.0f);
// At the end of
CGContextAddLineToPoint(context, pos_x, height);
pos_x +=grid_w;
// Start line
CGContextStrokePath(context);
}
// in height Draw a line across the range
while (pos_y < height) {
CGContextSetLineWidth(context, 0.2);
CGContextSetStrokeColorWithColor(context, [UIColor greenColor].CGColor);
CGContextMoveToPoint(context, 1.0f, pos_y);
CGContextAddLineToPoint(context, wight, pos_y);
pos_y +=grid_w;
CGContextStrokePath(context);
}
pos_x = 0.0f; pos_y = 0.0f;
// in wight Draw vertical lines in the range
while (pos_x < wight) {
CGContextSetLineWidth(context, 0.1);
CGContextSetStrokeColorWithColor(context, [UIColor greenColor].CGColor);
CGContextMoveToPoint(context, pos_x, 1.0f);
CGContextAddLineToPoint(context, pos_x, height);
pos_x +=grid_w/5;
CGContextStrokePath(context);
}
// in height Draw a line across the range
while (pos_y < height) {
CGContextSetLineWidth(context, 0.1);
CGContextSetStrokeColorWithColor(context, [UIColor greenColor].CGColor);
CGContextMoveToPoint(context, 1.0f, pos_y);
CGContextAddLineToPoint(context, wight, pos_y);
pos_y +=grid_w/5;
CGContextStrokePath(context);
}
}
conclusion
When writing this function, I have a lot of thinking, also referred to a lot of other people's blogs, code and other people's graduation thesis, hehe, also asked a few medical students, the code is not difficult, data processing part may not be so easy to do, but the completion of writing or a sense of accomplishment.
There are still a lot of problems in the code, later I will slowly optimize, welcome correction.