Image interpretation by python

Brucele

Imageregistration.pdf

Home >Computer Science homework help >Image interpretation by python

Computer Vision

Image Registration

Interest Point Detector & Point Features

CIS/EEC 693

Almabrok Essa [email protected]

mailto:[email protected]

What is an interest point?

Keypoints are used for: Image alignment 3D reconstruction Motion tracking Robot navigation Indexing and database retrieval

Corners as Distinctive Interest Points

“Flat” region: no change in all directions

“Edge”: no change along the edge direction

“corner”: significant change in all directions

Harris Corner Detector

Given a grayscale image I, consider taking an image patch over the area (u, v) and shifting it by (x, y). The weighted sum of squared differences (SSD) between these two patches, denoted S, can be given by:

2 ( , ) ( , ) ( ( , ) ( , ))

u v

S x y w u v I u x v y I u v= + + −

IntensityShifted IntensityWindow Function

Window function is either a rectangular window or Gaussian window

which gives weights to pixels underneath

1 in window

0 outside Gaussian

Harris Corner Detector

• Let 𝐼𝑥 and 𝐼𝑦 be the partial derivatives of I then

• Taylor Series expansion of I

𝐼 𝑢 + 𝑥,𝑣 + 𝑦 = 𝐼 𝑢,𝑣 + 𝐼𝑥 𝑢,𝑣 𝑥 + 𝐼𝑦 𝑢,𝑣 𝑦 +

2! 𝐼𝑥𝑥 𝑢,𝑣 𝑥

2 + 𝐼𝑥𝑦 𝑢,𝑣 𝑥𝑦 + 𝐼𝑦𝑦 𝑢,𝑣 𝑦 2 +… + (higher order terms)

• For small shifts (x,y), the first order approx is good

𝐼 𝑢 + 𝑥,𝑣 + 𝑦 ≈ 𝐼 𝑢,𝑣 + 𝐼𝑥 𝑢,𝑣 𝑥 + 𝐼𝑦 𝑢,𝑣 𝑦

First partial derivatives

Second partial derivatives

Harris Corner Detector

Thus,

𝑆(𝑥,𝑦) ≈ σ𝑢 σ𝑣 𝑤(𝑢,𝑣)(𝐼𝑥 𝑢, 𝑣 𝑥 + 𝐼𝑦 𝑢,𝑣 𝑦) 2

This can be written in a matrix form as:

where A is the second-moment matrix

2 2 ( , )

x x yx x y

u v x y y x y y

I I II I I A w u v

I I I I I I

    = =       



Harris matrix, and angle brackets denote averaging

(i.e. summation over (u, v))

( , ) ( ) x

S x y x y A y

    

 

The interest point or a corner is characterized by a large variation of S in all directions of

the vector (x y). This characterization can be expressed by analyzing the eigenvalues of A

Harris Corner Detector

• For an interest point (corner), the magnitudes of the eigenvalues (λ1 and λ2) of A should be large, thus:

𝑅 = det 𝐴 − 𝑘(𝑡𝑟𝑎𝑐𝑒 𝐴 )2

det(A) = λ1 λ2

trace(A) = λ1 + λ2

• If λ1 ≈ 0 𝑎𝑛𝑑 λ2 ≈ 0 then 𝑅 is small and the region is flat

• If λ1 ≫ λ2 or vice versa, then 𝑅 < 0 and an edge is found

• If λ1 𝑎𝑛𝑑 λ2 have large positive values, then R is large and a corner is found

Corner Response Function

Harris Corner Detector --- Compute R

Harris Corner Detector --- R > Threshold

Harris Corner Detector --- Local Maxima of R

Harris Corner Detector

Harris corner (Python)

import cv2 import numpy as np

def viewAllPoints():

img = cv2.imread("image.jpg") grayScale = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) keyPoints = cv2.goodFeaturesToTrack(grayScale,0, 0.05, 15) keyPoints = np.int0(keyPoints) for j in keyPoints:

x, y = j.ravel() cv2.circle(img, (x, y), 3, (0,255,0), -1)

cv2.imshow('image',img) cv2.waitKey(0) cv2.destroyAllWindows()

viewAllPoints();

Scale Invariant Feature Transform (SIFT)

Idea of SIFT

Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters

Overall Procedure at a High Level

Difference of Gaussian for scale invariance

SIFT --- Key Point Localization

• Detect maxima and minima of difference-of-Gaussian in scale space

• Each pixel (X) is compared to its 8 neighbors in the current image and 9 neighbors each in the scales above and below

• A pixel (X) is compared with 26 pixels in current and adjacent scales (Green circles)

• Select a pixel (X) if it is larger/smaller than all 26 pixels

SIFT --- Orientation Normalization

Histogram of local gradient directions computed at selected scale

Assign principal orientation at peak of smoothed histogram

Each key specifies stable 2D coordinates (x, y, scale, orientation)

0 2 

SIFT --- Vector Formation

• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)

• Create array of orientation histograms for each cell

• 8 orientations x 4x4 histogram array = 128 dimensions (16 cells * 8 orientations)

• Threshold normalize the descriptor

Example 2x2 Histogram Array

SIFT --- Feature Extraction

• This is the stage where the interest points, which are called key-points in the SIFT framework, are detected. • The image is convolved with Gaussian filters at different scales, and then the

difference of successive Gaussian-blurred images are taken. • Key-points are then taken as maxima/minima of the Difference of Gaussians

(DoG) that occur at multiple scales. • This is done by comparing each pixel in the DoG images to its eight neighbors at the

same scale and nine corresponding neighboring pixels in each of the neighboring scales.

• If the pixel value is the maximum or minimum among all compared pixels, it is selected as a candidate key-point.

SIFT --- Feature Matching

Given a feature in I1,how to find the best match in I2? 1. Define distance function that compares two descriptors. 2. Test all the features in I2, find the one with min distance.

I2I1

Feature distance between two descriptors f1 and f2

f1 f2

Simple approaches:

1. Euclidian distance (d) between f1 and f2

2. SSD distance sum of square differences (SSD) between entries of the two descriptors

SIFT --- Feature Matching

Better approach is ratio distance = SSD(f1, f2) / SSD(f1, f2’) ▪ f2 is best SSD match to f1 in I2 ▪ f2’ is 2nd best SSD match to f1 in I2

SIFT --- Effect of threshold T

Image Registration

What is Image Registration?

• Image Registration is the process of aligning two or more images of the same scene with reference to a particular image.

• The images are captured from various sensors at different times and at multiple view-points.

• Image registration finds application in medical sciences, remote sensing and in computer vision.

Image Registration Applications

• Stereo imaging techniques

• Find out the point correspondence, or the point registration or as called

point matching

• Disparity for all the point in the images

• The depth or the distance of the different object points from the

camera

Image Registration Applications

• Remote sensing

• Where the images can be

• taken by different sensors

• working different bands

• even the sensors may be located in different locations

Image Registration Applications

• Automated navigation

• Finding a place in picture where it matches a given pattern

• Find out what is the location of a particular object with respect to a map

SIFT --- Based Registration Algorithm

• Extract SIFT features from images using a function from OpenCV library or any other

library

• Match features using naive nearest neighbor approach. Determine the distance

between each SIFT descriptor using squared Euclidean distance:

• Compute affine transformation using any built-in function.

• Transform Image2 to align with Image1 using any built-in function.

• Find registration error (absolute error or sum of squared differences).