Bio-Medial Engineering Institute of Fuzhou University, Fujian Fuzhou, 350002, China
Contrasted with other information carriers, such as speech and text, images contains larger amount of information, especially in sequential images, that is waiting to be exploited, in particular the dynamic information of correlation, difference, and temporal relationship between different frames. This dynamic information contributes a great deal in analysis of 4D images. This paper proposes a method for detecting dynamic information from sequential images, based on the rebuilding of their gray (position)~time function on direction lines, an approach that has been analyzed and studied extensively on the setting of various direction lines. This method is based on motion that is presented on sequential images. In particular, the method, Omni directional M-mode Echocardiography system, which we have studied extensively, will be described leading to a robust way of diagnosing heart diseases.
Keywords: Rebuilding gray (position) ~ time function, tracking moving object on any directional line, echocardiography, omnidirectional m-mode echocardiography system , motion information..
open-access license: This is an open access article licensed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted, non-commercial use, distribution and reproduction in any medium, provided the work is properly cited.
* Address correspondence to these authors at the Bio-Medial Engineering Institute of Fuzhou University, Fujian Fuzhou, 350002, China; Tel: 86-591-83727187; Fax: 86-591-87893589; E-mails: Chianglin@fzu.edu.cn, E-mails: Liwei@fzu.edu.cn
The Research of Sequential Images: Rebuilding of Gray (Position) ~ Time Function on Direction Lines and Their Applications
Sequential images, which composed of a stack of 2D static images in order arranged one by one, contain not only information that static images carry, but also information of motions hidden among frames, the latter appearing to be more important. To detect this information, optical flow detecting technique, which is a well-known method that applies computer vision theory to detection of dynamic information, analyzes the velocity field of each pixel in images from the viewpoint of statistics [1Liu X, Wende S, Xianhong S. Motion analysis of echocardiographic sequences: state of the art[J] Int J Biomed Eng 2003; 1: 6-11., 2Anna W, Silin X, Yue Y, et al. Medical Image Registration Method Based on Improved Optical Flow Model J Image Graphics 2010; 2: 153-8.]. According to our experience working on contour-based optical flow detecting [3Qiang Lin. A Detecting Method for contour-Based Optical Flow Field of Heart in Ultra- sound B-Scan Images Acta Electronics Sinica 1996; 24(4): 122-5.], it always suffers from many difficulties on mathematic analysis when it is used to solve the problem on deformable objects and non-continuous view angles. Furthermore, there exist many statistic, conjectural and uncertain problems as far as specific understanding of velocity is concerned. Therefore many new detecting methods on Echocardiography Sequences [4Sun Zheng. Estimation of Cardiac Motion Based on Non-rigid Motion Theory Opto-Electronic Eng 2008; 9: 113-7.-8Song M, Haralick RM, Sheehan FH. Ultrasound imaging simulation and echocardi ographic image synthesis. Proceedings [J] Int Conference Image Processing 2000; 3: 420-3.] have been developed. In this study, we have searched for a new method to detect dynamic information from sequential images, which can rebuild gray (position) ~time function (waveform) on direction lines to detect dynamic information on moving views, moving objects, and moving characteristic points in the images.
Rebuilding a gray (position)~time function (waveform) on an arbitrary direction line is to reposition the recognized gray level of views, objects or characteristic points on that direction line and further to confirm its position by capturing the white dots on the line. Because these direction lines are artificial, the positions of those white dots can be confirmed. On the other hand, views and objects in the same frame of sequential images are relatively still, leading the white dots can be confirmed uniquely, while altering with the shifting frames of sequential images. Accordingly, with the certainty of the time interval between two neighboring frames, these white dots, which move originally on the direction lines, are extended with the time. Consequently the gray (position) ~ time waveform (function) on direction lines are obtained. Moving equation on direction lines are obtained when these dots are confirmed as moving objects. From the point of kinematics, the most important moving basic information data can be obtained. Therefore we can mine data in depth and find correlative new dynamic information with the combination of kinematics and dynamics.
The movements of viewing angles, objects or characteristic points of practical images, however, are very complex. Therefore it is impossible that they simply move back and forth on a single direction line in most cases. As a result, the setting of direction lines of sequential images in a practical application should be a combination of many approaches so as to detect practical information and to engage the tracked target points with the combination of the settings of direction lines and certain algorithms. In this paper, two methods are developed to solve this problem. One is to catch the bright points in the ECG a form of single directional stepping translation with frames. Another is to obtain a non-functional motion model that is in turn applied on the direction line to capture functional moving points. Both methods are employed in practice [9Liqin H, Qiang L. Movement track extracting in omnidirectional M-mode echocardiographic image In: Proceedings of the First International Symposium on Test Automation and Instrumentation. SEP13-16; Beijing; 2006; pp. 538-42., 10Qiang L, Wenji W. An omnidirectional M-mode echocardiography system and its clinical application Computerized Medical Imaging Graphics 2006; 30: 333-8.]. According to the moving situations of image objects, more direction lines can be set, leading to more data and new accurate dynamic information being mined and detected.
In this paper, a method has been developed, which combines with certain algorithms, which focuses on the tracking of moving objects to set the direction lines, and is based on the movement of moving objects with the help of sequential images, composed of static images.
2. THE SETTING OF DIRECTION LINES TO REBUILD GRAY(POSITION)~TIME FUNCTION AND THE MINING OF THE DYNAMIC INFORMATION
As discussed above, direction lines are used to track moving objects, so the optimal direction line is just the trace of a moving object being tracked. However, finding a direction line from images is difficult if not possible. To this end, what we can do is to set artificially some controllable and positionable direction lines which can capture the predefined view, objects and characteristic points. And then we can combine with some related algorithms to set direction lines to obtain a moving trace of the engaged objects. That is gray (position) ~time function, which first-order differential is the velocity of movement, and second-order is its acceleration. Thus, we can mine various moving information.
The method used to set direction lines can be classified into two kinds. The first is the setting of fixed direction lines and the second is the setting of direction line variables within frames. The details are given below.
2.1. Setting of Fixed Direction Lines
In this way, the position coordinates of direction lines to be set can be fixed and are invariable of frame numbers in the sequential images. The detailed setting can be divided into following three classes:
Single arbitrary direction line
Arbitrary number arbitrary direction lines
Multigroup arbitrary direction lines
Multigroup arbitrary direction lines are a kind of arbitrary number arbitrary direction lines, which refers to that there are many groups of moving entities in a single image. With this setting, the settings of several direction lines are associated and constrained with each other, due to the fact that every moving entity in a group is coherent. For example, in a moving entity group of left ventricular short axis of cardio scans, the moving directions of every part tend to be centripetal. Thus, the direction of every direction line on every part of the group features commonly to be centripetal, which forms the association and restriction of each direction.
2.2. Setting of Direction Line Variables with Frames
With this setting, the position coordinates of a direction line are keeping altering and changing with the altering of sequential frames. It can be an arbitrary number on any arbitrary direction lines, as shown above. Therefore, this method is more complex than the setting of fixed direction lines. However, it may be a way to track more complex moving objects. As mensioned above, the major idea of the method is that all of the direction lines should be controllable and positionable, which requests the altering of this kind of direction lines should be regular and operated according to a certain algorithm so that they can be positioned in different frames of a set of sequential images. The detailed setting of altering direction lines can be divided into the following two subclasses:
Single directional stepping translation form with frames
Complex moving form with frames
The direction line of single directional stepping translation form keeps altering with frames. In this way, we can track the moving trace of an object, which moves very complexly within the range of R (the longest deviate distant to the center- line), as illustrated in Fig. (1). Where the length of the direction line is 2R, and the line is perpendicular to the centerline. The stepping translation of the direction line between two neighboring frames is d, and there is a direction line corresponding to each frame to track an object point. As a result, the moving trace of the object can be extracted from the gray (position) ~ time function (waveform), similar to sampling techniques.
With this direction line in practice, the moving trace of the ECG object on the lower part of echocardiography can be obtained, illustrated in Fig. (2). We can join it synchronized to Omnidirectional M-mode Echocardiography (which will be introduced in the subsequent introduction) as their time reference.
2.3. Nonfunctional Motion and Functional Motion
Normally, it is very complex to set a direction line of moving form within frames, but it may track more complicated moving traces, leading to a separation of number of important movements from the other movements, which is the main intension of this study. Using this method, we will evaluate Omnidirectional M-Mode Echocardiography functional motion separation, the important issue when it comes to detect every structural function of cardiac to a great extent of precision. As the equipment that is used to non-traumatically detect cardiac moving image is given a fixed position on an external body, the detected sequential motion images of one cardiac structure are only a general result, which consists of functional motion of this self structure (such as diastole and systole) and other nonfunctional motion (such as blood flow, breathe, the dragging between each structure.). Hence it can’t single out the functional motion that can reflect the inherent information of one structure, the most obvious disadvantage of these medical dynamic images as illustrated in Fig. (3a). If the first and the last coordinate of the fixed coordinate line(x, y) is (105, 30; 136, 112), the cardiac structure of being captured from No.1, 14, 34 frame in the sequential images is in three different position, they are aortic front wall, ventricular septum and aortic front wall. On the other hand, under these three pictures, the corresponding rebuilding echocardiography is illustrated. Thanks to the influence of non-functional motion, the setting of fixed direction lines can’t track point A of this structure nor capture the functional trace of this point at all times. In the current international standard, it is the momentous unfathomed problem to detect dynamic image information.
By statistic moving of nonfunctional motion character points which are close to point A, we obtain the nonfunctional motion of these points in the sequential images of Fig. (3a), and act it on the direction line according to the principle of relative motion, consequently make the direction line move complicated. From Fig. (3b) we can seethe first and the last coordinate of direction line in No.1,14,34 frame is (105, 30; 136, 112) , (125, 28; 156, 110), (93, 37; 124, 119). On the whole these points are all the positions of A of this structure, thus the rebuilding M-mode echocardiography is clearer, and synthesize the three lines near No.14 frame into one line.
3. THE SETTING MODEL OF FIXED ARBITRARY NUMBER DIRECTION LINES-OMNIDIRECTIONAL GRAY ~TIME ECHOCARDIOGRAPHY WAVEFORM
Omnidirectional gray ~ time echocardiography waveform, or called Omnidirectional M-mode Echocardiography, is the most typical example of the setting of fixed arbitrary number direction line, and is a fruitful method in our research. According to the rebuilding principle of Omnidirectional gray ~ time waveform, if the direction of a sample line selected is the same as the actual motion direction, the gray intensity data of the pixels on them (e.g. B) can generate gray ~ time waveform combined with corresponding temporal relationships. As shown in Fig. (4), the gray ~ time waveform of three lines is rebuilt from long and short axis echocardiography respectively and represents motion waveform of the wall of ventricle structure, which is because these gray values can stand for the boundary of cardiac structure in ultrasound images. Furthermore, all these resulted M-mode echocardiography are synchronised in time, which reflects the movements of different parts in different cardiac structures.
As for Omnidirectional gray ~ time waveform detected from echocardiography sequential images, we can select arbitrary direction of any parts in any structures to set sample line (direction line), which can be used by to follow the motion tracks of this part. Thus, sample lines are not confined by the direction of ultrasound wave-line any longer. Consequently a substantial improvement in detection accuracy is obtained. Moreover, we can compare the motion parameter of every echocardiography and their temporal-phase relationships, because any arbitrary number sample line whose waveform is synchronous in time can be selected. As a result, the idea of Omnidirectional M-mode Echocardiography Waveform System has been patented by the National Invent Patent of China (No. ZL 98 125713.5).
3.1. LEJ-1 Omnidirectional M-Mode Echocardiography System
Subsequently, we manufactured LEJ-1 Omnidirectional M-mode echocardiography system with the setting of fixed direction lines. The system facilitates two functions, namely automatic operation and manual operation. In the former, when confirmed a center, 12 oriented direction lines will give out every 30 angles, which looks like a clock panel. Then their gray ~ time waveform of specified duration will expend in screen or be printed on laser printer paper, as given in Fig. (5). This kind of operation is mainly suited for left ventricle short axis sequence images whose motion is mainly acentric and centripetal.
The second operation is manual operation. In accordance with any positions, any structures in echocardiography images, based on their motion direction, operator can select any direction sample line with mouse and set manually to build their Omnidirectional echocardiography gray ~ time waveform lasting for 5 seconds. In this operation, there will be a preview (echocardiography waveform) displayed on the monitor to give a choice on whether a sample line is selected.
Regardless which operation is carried out, a series of resulted M-mode echocardiography waveforms corresponding to a sampling line can be chosen to perform further detection for various parameters. Parameters such as amplitude, velocity, compressing peak velocity and the quality of thickness, which related to the motion of cardiac structure, are detected from appointed sample line in echocardiography waveform as demonstrated in Fig. (6). According to the description above, we rebuild ECG below echocardiography which is synchronized with all of Omnidirectional M-mode echocardiography waveform so as to compare Omnidirectional gray ~ time waveform with ECG (see Fig. 2).
3.2. Clinical Application
Having passed the physical test conducted by Fujian Province Central Test Institute in China and by several hospitals when being clinically practiced, the method detailed above is verified that resulting detection data are accurate, scientific, and trustworthy.
Consequently, we have developed the LEJ-1 Omnidirectional M-mode echocardiography system and applied it to clinical practice in the Fujian Provincial Research Institute for Cardiovascular Diseases [11Zhensheng Ye, Wei G, Shunqiong C, et al. Consistency of omnidirectional M-mode echocardiography and two-dimensional strain in assessment of left ventricular systolic asynchrony Chin J Med Imaging Technol 2010; 26(3): 496-9.-14Lihong L, Qiang L, Haoping L, et al. Clinical application of omnirange M-mode echocardiography—a new technique, LEJ-1 Chin J Ultrasound Med 2001; 17(8): 591-3.]. Some of clinical practice reports detected by Fujian Provincial Research Institute for Cardiovascular Diseases are given in Table 1.
In Table 1, the statistical values of t counter-sample, which are measured on Omnidirectional M-mode echocardiography system, are both smaller than the statistical values t with a degree of freedom of 29 and a probability of 0.05. On the point of statistics, the result is reasonable. Otherwise, between Omnidirectional M-mode echocardiography and original B-scan echocardiography, according to the comparative experimental standards practiced in clinic, the data at the same location are very similar. In fact, their difference is statistically insignificant.
The system has been applied into clinical practice in Chinese People’s Liberation Army (CPLA) General Hospital (301 Hospital, Beijing) for 3 years [15Yue L, Qing W, Chaoyang W. Observation of pulmonary artery motion curve of pulmonary hypertension with anatomical M-mode echocardiography Chin J Ultrasonogr 2003; 12(10): 593-.-18Qing W, Yue L, Chaoyang W, et al. The study of left ventricle regional systolic and diastolic function in normal subjects by LEJ-1 omnidirec- tional M-mode echocardiography system Chin J Med Imaging Technol 2003; 19(12): 1672-5.]. They summarize the data and the experiments in their own clinical practice according to the function of the system. The related research is mainly on the assessment of left ventricle regional systolic and diastolic functions, atria septal motion and atria load relation, observation of pulmonary artery motion curve of pulmonary hypertension, etc.
The ultrasound department of the General Hospital of CPLA has made a detailed research on regional motion function of heart with the system. Hundreds of cases with varying age, normal and abnormal cardiovascular functions are all diagnosed. The application of the Omnidirectional M-mode echocardiography in assessment of left ventricle regional diastolic function may be recommended as a diagnostic method of examining heart degradation earlier when the global functionality is correct while local functionality is lightly degrading.
4. THE DATA MINING OF DYNAMIC INFORMATION IN SEQUENTIAL IMAGES
The most extinctive difference between our Omnidirectional gray ~ time waveform system and Anatomy M-mode Echocardiography [19Yang Hu, Wei Guo. Clinical Application of M-mode Echocardiography Adv Cardiovascular Dis 2008; 29(5): 716-8.] is that we have obtained synchronous Omnidirectional gray ~ time waveform group on any parts, any direction and arbitrary columns (corresponding to arbitrary sample lines) based on dynamic information detected from sequential images with synchronous ECG, whereas. The anatomical M-mode echocardiography shows only two or three images simultaneously even when they come from a different cardiac period.
Compare with Tissue Doppler echocardiography [20Daowen Zhen, Jianping Bin, Shengxiong N, et al. Quantitative evaluation of left ventricular diastolic function by doppler tissue imaging China Journal of Modern Medicine 2008; 18(15): 2216-.], our system has some characteristics as specified in the follows [21Qiang Lin, Jianghong S. A dynamic information of echocardiography—omnidirectional m -mode echocardiography Chinese J Scientific Instrument 2005; 26(4): 437-0.]:
the accelerator comes from differential coefficient to athletic function directly not from frequency excursion indirectly.
the orientation can changes freely.
Groups of arbitrary direction lines in any part of any structure can be set and presented corresponding to Omnidirectional echocardiography images with synchronous ECG so as to be compatible with waveforms and temporal phases.
The dynamic analysis here we made in sequential images is similar to amplifier analysis made for several object units in static images. One example is that an image of streets in a static whole scene may only contain lines. However, when further amplified, these lines can display cars, shops, or other details in these streets. Under the similar pattern, the sequential images show the “whole scene” of every cardiac structure from its full view. It can be implemented by the way of Omnidirectional gray ~ time wave- form detecting method to acquire detailed knowledge of the motion status, namely dynamic details of any parts in any structures (including motion waveform, amplitude, velocity, and accelerate, etc. when the point’s movement being along with the time). The wave- form, which we obtained, is the motion trace or motion equation of the detecting part in cardiac structure along with time. On the base of it, one-order timing of differential of Omnidirectional gray ~ time wave- form is the velocity
of detecting movement of cardiac structure and two-order is acceleration
(). As follows Fig. (7).
In summary, it is the most important to obtain new dynamic information in the hemodynamics study. So we implement the data mining of new dynamic information from every cardiac structure from sequential images. We have applied the method to the analysis of cardiac B- scan/color scan (or echocardiography) sequential images. In fact, this motion analysis method on sequential images can also be employed to many other medical sequential images such as X-ray, coronary-contrast, etc. Consequently, this method for motion analysis can also be improved and developed further to fit the characteristics of each imaging modality.
5. CONCLUSION AND DISCUSSION
With the advanced development of science and technology, the method detailed in the paper and used to analyze dynamic information is becoming much profounder. The main methods currently used can be classified as a viewpoint-based approach, which is employed on sequential images, focusing on dynamic objects and on movement. While based on sequential images, the method proposed in this paper constitutes all its syntaxes, such as the setting of the direction lines and the algorithm combined, for moving objects principally. So it has some characteristics such as understanding motion details of each cardiac structure, confirming moving information data, and correcting detecting results. To this end, the prior knowledge bases are required in order to set direction lines easier to analyze the motion information.
At present, it is required that all pixels are relatively stationary in a single sequential image. The time interval between two neighboring frames is 1/25 second in the PAL system and1/30 second in the NTSC system, which is a very long time for usual objects. So, if the time of frame interval is reduced farther, the dynamic information detection of higher-speed movement can be obtained with this proposed method.
In addition, the direction lines used above are all straight lines. In the future, if they are modified into controllable, positionable, and non-linear direction lines, it will be easier to track more complex movement leading to the improvement of this approach further.
This research is supported by the EU Project of Teleimaging in Medicine (TIME): (CN/ASIA-IT&C/009 (91450)), and by the Science Research Foundation of the Ministry of Health & United Fujian Provincial Health and Education Project for Tacking the Key Research, P. R. China (No. WKJ2005-2-010). Thanks to Dr. Gao of Middlesex University for her kind of help while writing the paper.
CONFLICT OF INTEREST
Liu X, Wende S, Xianhong S. Motion analysis of echocardiographic sequences: state of the art[J] Int J Biomed Eng 2003; 1: 6-11.
Anna W, Silin X, Yue Y, et al. Medical Image Registration Method Based on Improved Optical Flow Model J Image Graphics 2010; 2: 153-8.
Qiang Lin. A Detecting Method for contour-Based Optical Flow Field of Heart in Ultra- sound B-Scan Images Acta Electronics Sinica 1996; 24(4): 122-5.
Sun Zheng. Estimation of Cardiac Motion Based on Non-rigid Motion Theory Opto-Electronic Eng 2008; 9: 113-7.
Jiayong Y, Tiange Z. Band Limitation Based Gradient Vector Flow Active Contour Model for Ultrasound Image Segmentation[J] J Shanghai Jiaotong University 2003; 2: 84-7.
Xu C, Pham DL, Prince JL. Medical image segmentation using deformable models Handbook of medical imaging-medical image processing and analysis. SPIE Press 2000; pp. 129-74.
Yanjun L, Ke Z. Vision Matching and Object Recognize Technology 2009.
Song M, Haralick RM, Sheehan FH. Ultrasound imaging simulation and echocardi ographic image synthesis. Proceedings [J] Int Conference Image Processing 2000; 3: 420-3.
Liqin H, Qiang L. Movement track extracting in omnidirectional M-mode echocardiographic image In: Proceedings of the First International Symposium on Test Automation and Instrumentation. SEP13-16; Beijing; 2006; pp. 538-42.
Qiang L, Wenji W. An omnidirectional M-mode echocardiography system and its clinical application Computerized Medical Imaging Graphics 2006; 30: 333-8.
Zhensheng Ye, Wei G, Shunqiong C, et al. Consistency of omnidirectional M-mode echocardiography and two-dimensional strain in assessment of left ventricular systolic asynchrony Chin J Med Imaging Technol 2010; 26(3): 496-9.
Wei Guo, Yang H, Chun W, et al. A study on Normal Short-axis Myocardial Velocity Gradient Assessed by Omni-directional M-mode Echocardiography Chinese J Med Imaging 2010; 18(2): 157-9.
Wei G, Lihong L, Chenbin . Relation between atrial septal motion and atrial load using omnidirectional M-mode echocardiography Chin J Ultrasonogr 2004; 13(9): 593-.
Lihong L, Qiang L, Haoping L, et al. Clinical application of omnirange M-mode echocardiography—a new technique, LEJ-1 Chin J Ultrasound Med 2001; 17(8): 591-3.
Yue L, Qing W, Chaoyang W. Observation of pulmonary artery motion curve of pulmonary hypertension with anatomical M-mode echocardiography Chin J Ultrasonogr 2003; 12(10): 593-.
Yue L, Qing W, Yanmi L, et al. Measuring left ventricular isovolumic relaxation time with anatomical M-mode echocardiography and pulsed doppler Chin J Ultrasonogr 2005; 14(4): 271-4.
Yue L, Qing W, Qiang L, et al. Assessment of left ventricle regional diastolic function by omnidirectional M-mode echocardiography—study about a kind of new method and index Chin J Med Imaging 2002; 10(3): 169-72.
Qing W, Yue L, Chaoyang W, et al. The study of left ventricle regional systolic and diastolic function in normal subjects by LEJ-1 omnidirec- tional M-mode echocardiography system Chin J Med Imaging Technol 2003; 19(12): 1672-5.
Yang Hu, Wei Guo. Clinical Application of M-mode Echocardiography Adv Cardiovascular Dis 2008; 29(5): 716-8.
Daowen Zhen, Jianping Bin, Shengxiong N, et al. Quantitative evaluation of left ventricular diastolic function by doppler tissue imaging China Journal of Modern Medicine 2008; 18(15): 2216-.
Qiang Lin, Jianghong S. A dynamic information of echocardiography—omnidirectional m -mode echocardiography Chinese J Scientific Instrument 2005; 26(4): 437-0.