[COMSCAN]: An Optical Music Recognition System Muhammad Sharif
Quratul-Ain Arshad
Mudassar Raza
Wazir Zada Khan
COMSATS Institute of Information Technology
COMSATS Institute of Information Technology
COMSATS Institute of Information Technology
COMSATS Institute of Information Technology
Wah Cantt- Pakistan +923005188998
Wah Cantt- Pakistan +92519272614
Wah Cantt- Pakistan +92519272614
Wah Cantt- Pakistan +92519272614
Muhammadsharifmalik@ yahoo.com
brightsuccess_12@hotmai l.com
[email protected]
[email protected]
ABSTRACT This paper presents the concept of optical music recognition (OMR), called “COMSCAN” which takes an electronically printed sheet of musical notes and processes it by using the four basic steps usually used by almost all OMR systems. The COMSCAN OMR system presented in this paper is fully based on our three proposed algorithms. First algorithm is for stave line detection which is a new form of horizontal projection. Second algorithm called “averaging” is for stave line removal which is done immediately after the detection. The third algorithm is used for musical notes’ symbol identification/recognition step. It is an enhanced form of template matching technique. After a detailed description of the working of COMSCAN and the proposed algorithms, results are reported to analyze the efficiency and accuracy of proposed system. Finally, the comparison is done with the system that uses an algorithm of adding more stave lines instead of removing them.
Keywords
writer to writer, symbols shinning from the reverse page and so on [7]. OMR works like OCR (Optical Character Recognition). Typical OCR techniques cannot be used in music score recognition due to some obvious reasons. In an OMR system, there is a complexity to read the symbols lying on stave containing five lines. Our proposed OMR system works in four steps i.e., stave line detection, musical object location, musical object recognition and finally interpretation of the given musical sheet. It then plays the given musical sheet accordingly.
2. WORKING OF OMR SYSTEMS OMR systems fall in the field of image processing techniques. Most OMR systems usually work in four steps for recognition of musical notes. The musical notes are basically symbols that can be identified by these systems using different image processing techniques. The stepwise working of OMR systems is as follows [14]:
OMR, Music, Recognition, Scan, Symbols.
1. INTRODUCTION The area of OMR is immature because it still do not provide a definitive method for identifying and locating the musical symbols on stave lines. The printed music notes are scanned using computer. These notes are then brought to some music editor with the help of OMR [1]. The OMR system then plays these notes and provides output to the listener. In the recent years, OMR systems are also in vogue. Some of them are either pure or part of a larger music editing system. Common to all these systems is that they are designed to recognize printed sheet of music scores in which the contrast between music symbols [9] and background is good, stave lines are straight and symbols are printed clearly. However, they fail in case of recognizing historic handwritten music scores because of tainted paper and ink, bent stave lines, notations that vary from
Figure 1. Steps of OMR System [14] Step 1: Detection of Stave line In this step, stave lines are found and pointed. Most of the time these lines are excluded and symbols are left behind. Step 2: Tracking of Musical object
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. FIT’09, December 16–18, 2009, CIIT, Abbottabad, Pakistan. Copyright 2009 ACM 978-1-60558-642-7/09/12....$10.
Detect the symbols that were on the area of stave lines. Step 3: Symbol identification In this step, the symbols are recognized and separated on the basis of classification.
Step 4: Semantics of music notation The symbols are further separated on the basis of their semantics [6]. These symbols are then replaced with the similar symbols which are readable by the music editor.
3. EXISTING SYSTEMS In 1993, Musitek Corporation released “music scanning and notation software” named as MIDISCAN which is known as the “world's first commercially available music-scanning software for Windows” [11]. In 1996, SmartScore (for Windows and Macintosh) was released. In 2003, Capella-scan was presented in Frankfurt Music Exhibition [12]. Meanwhile, Graham Jones developed SHARPEYE “Music Reader” which is used to convert printed sheet of music notes into a MIDI file [13].
4. PROPOSED SYSTEM The proposed OMR system COMSCAN also works like other OMR systems [5]. In COMSCAN, two new algorithms have been proposed for different steps of musical objects recognition. First algorithm is for stave line detection which is a new form of horizontal projection. Second one is an extension to template matching algorithm [8]. COMSCAN first reads music sheets as an image file and then reads it row by row for identifying the stave lines. After the identification of these lines, it then locates the musical objects (recognize their location) and applies object recognition algorithm for recognition of musical notes. Finally, it plays these recognized musical notes. The overall block diagram of our proposed OMR system is shown in Figure 2.
In most of the existing systems, the stave lines (after the detection) are either brushed aside or removed [3]. The removal of these stave lines makes the recognition process difficult because the musical notations are damaged and become fragmented. Many algorithms are being proposed for removals of these stave lines but they all result in the fragmentation of musical symbols. The proposed algorithm performs the stave lines’ removal through averaging, as a result of which marvelous results have been obtained because the symbols are not damaged and the recognition process has become easier. In the second stage of the proposed technique, the location of symbols is identified after which the location of an object is detected by using a pattern recognition algorithm to search different symbols on a musical score. In the last stage, musical symbols are played accordingly after detecting their locations and recognizing them.
4.2 Proposed Algorithm Step 1: Loading Musical Sheet Load musical notes file or musical sheet as an image file.
Figure 3. Loading Musical Sheet Step 2: Detecting Stave Lines • • • • •
Figure 2. Data Flow Diagram of COMSCAN
First horizontally squeeze the musical sheet. Go to the height and width of the sheet. Scan the sheet pixel by pixel in such a way that width is added after each increment in height. The color pixel values of each pixel are counted and then summed up. Decision about whether the pixel is black or white is taken by comparing the per pixel value with a threshold value (450) 9 If the value is less than 450, the pixel is black. 9 If the value is greater than 450, the pixel is white.
4.1 Working of COMSCAN COMSCAN comprises of four basic stages: 1) 2) 3) 4)
Stave Line Detection Musical Object Detection / Location Musical Symbol Identification / Recognition Interpretation of Recognized Musical Notation
In the first stage the five stave lines are detected which are the most fundamental part of a musical score because all the musical symbols are laid on these lines and some of the ledger lines. The musical symbols are superimposed on these five stave lines many times in such a way that the symbols overlap, making their recognition much more cumbersome than the recognition of other English characters.
Figure 4. Detecting Stave Lines •
After taking the sum of per pixel color values the sheet is converted into grey level image by taking the average. This average shows the percentage of pixels. 9 If the percentage (average of number of pixels of a row) is less than 50%, it means there is a stave
•
line. A line is drawn on the graphics sheet to show this detection. Repeating the same process will detect all the stave lines of a whole musical sheet. Keep the lines in a collection.
•
If the nearest two pixels above and below are white then set the current pixel as an average value. So the musical symbols are not fragmented by using this algorithm, making recognition process easier.
Step 3: Marking the Stave Boundary For marking the stave boundary, the difference of two lines is analyzed. It is required to find the last two gaps to detect which stave line is this. • • • •
•
If the number of lines detected is 1, then this line is saved because it is not known at this time that which line is this. If the number of lines detected is 2, continue detecting more lines. If the number of lines detected is 3 or greater than 3 then consider previous distance as p and last distance as d. Now if 9 p > d then this is the first line of next stave 9 d > p then this is the last line of current stave 9 d = p then continue in the same stave Find/ Mark the Boundary of Stave 9 Each starting line of stave is stepped back by three 9 Each last/end line of stave is increased by three 9 In this way, there are 11 positions (5 Stave lines + 6 Ledger lines) 9 Thus there is total = 11 + 10 spaces (gaps) = 21 positions
Figure 6. Removing the Stave Lines Step 5: Detecting or Locating Musical Objects •
Musical objects are located by traversing through stave bounds. Any existing object is checked by calculating 4% population of black pixels.
Figure 7. Detecting or Locating Musical Objects
Step 6: Recognizing or Identifying Musical Objects •
For the recognition of musical objects or symbols, go through each object and try each template to match. For saving the processing speed the most frequent symbols are placed and checked first.
Figure 5. Marking the Stave Lines Boundary Step 4: Stave Line Removal A new algorithm is proposed for stave line removal called “averaging”. Averaging of stave lines will not damage the symbols superimposed on the five stave lines. • •
As the entire stave lines are already calculated, so go through each stave line by picking the object of each line and guessing its position. Start from 0 and go through the whole width of a sheet. Pick up the four neighboring pixels of that current object/pixel. Take the two pixels above and two pixels below the current pixel.
Figure 8. Recognizing or Identifying Musical Objects
4.3 Results There exist many OMR systems, out of which some are commercial products. All the systems work in some limitations. In all implemented systems, the authors have used some specific collection of sheets and showed experimental results on them. The authors of a paper titled ”Defacing Music Scores For Improved
Recognition” [2] have proposed their idea for recognition of musical notes and have shown their results on a sample of 9 sheets. COMSCAN is tested using 11 test sheets. 100 percent results are achieved for stave line detection and 99.5 percent results for musical object recognition. The overall accuracy of COMSCAN is shown in Table 1 and Figure 9. Accuracy of ComScan
for detection and recognition of musical notes has shown 99.5% accuracy.
6. REFERENCES [1] David Bainbridge and Tim Bell. “An Extensible Optical Music Recognition System”, Volume 33 , Issue 2 (February 2003), John Wiley & Sons, Inc. New York, NY, USA, Pages: 173 - 200 [2]
101
100
[3] R. Randriamahefa, J.P. Cocquerez, C. Fluhr, F. PCpin, and S. Philipp. “Printed Music Recognition”, International Conference on Document Analysis and. Recognition, IEEE, Saint-Malo, France 1991
99 R e c ogn ition %
Scott Sheridan, and Susan E. George, “Defacing Music Scores For Improved Reognition.”, Proceedings of the Second Australian Undergraduate Students' Computing Conference, 2004, Published by the AUSCC
98
97
[4] Y. C. Chung, "Recognition of Printed Sheet Music Using Hough Transform And Morphology Operations," Master thesis, National Taiwan Normal University, Taiwan, 1995.
96
95
[5]
94 Treble clef Bass clef
Sharp
Flat
Natural
Whole note Half note
Quarter Note
Eighth Note
Sixteenth Note
Crochet Vertical line Rectangle rest rest
Musical Notes
Figure 9. Graph showing overall accuracy
Overall Accuracy Of ComScan Treble clef
100%
Bass clef
100%
Sharp
100%
Flat
100%
Natural
100%
Whole note
98%
Half note
97%
Quarter Note
100%
Eighth Note
99%
Sixteenth Note
100%
Crochet rest
100%
Vertical line
100%
Rectangle rest
100%
Table 1. Overall Accuracy of COMSCAN
5. CONCLUSION COMSCAN has been developed according to the proposed algorithms, as a result of which the system works efficiently. The algorithm proposed for stave line removal has becomes an aid for the next step of recognizing / identifying musical symbols because the musical objects are not damaged by using this algorithm and are thus recognized more accurately. The algorithm
K. Todd Reed and J. R. Parker, “Automatic Computer Recognition of Printed Music”, Proceedings of the 13th International Conference on m Pattern Recognition (IEEE), 1996, Volume 3, 25-29 Aug. 1996 Page(s):803 - 807 vol.3
[6] David Bainbridge, “A Complete Optical Music Recognition System:Looking to the future”, November 17, 1994, http://www.cs.waikato.ac.nz/~davidb/publications/complete [7] Roland Gocke. “Building a System for Writer Identification on Handwritten Music Scores” In M.H. Hamza (ed.), Proceedings of the IASTED International Conference on Signal Processing, Pattern Recognition, and Applications SSPRA 2003, pages 250-255, Rhodes, Greece, 30 June - 3 July 2003. Acta Press, Anaheim, USA [8] Walter O'Dell PhD, “Template Matching”, 2005, wodell at rochester.edu http://rsb.info.nih.gov/ij/plugins/template/matching.html [9] Elaine Ernst Schneider and Joanne Mikola, “Elements of Music Notation”, April 4, 2001 http://Lesson Tutor Introduction to Music Notation.htm [10] IchiroFujinaga, “Application of Optical Music Recognition technologies for the development of OCVE”, 2004/11/29 http://www.music.mcgill.ca/~ich/misc/OCVE_OMR/OCVE_ OMR.html [11] About Musitek, www.musitek.com/musitek.html [12] Fast conversion of printed scores to MIDI and capella format, http://www.softwarepartners.co.uk/index.php?option=com_content&task=view&i d=28&Itemid=71 [13] About SharpEye Music http://www.recordare.com/sharpeye/
Reader
2
[14] Horst Bunke, Patrick Shein-pei Wang, “Handbook of character recognition and document image analysis”, Published by World Scientific, 1997, ISBN 981022270X, 9789810222703
